Sending emails over Queue with AWS SES

One of most annoying things is when I have to wait for couple of seconds for form to submit data to the backend.

I guess developers usually hook many different services to register request like: create new user in the database, add user to Mailchimp list, send activation email etc.

Sending emails and contacting 3rd party APIs are time consuming operations and it is absolute must to move them to some queue.

Second important thing when signing up is: Do we require from user to click on activate link in email before he can use the service?

If answer on above question is Yes, than we must use system which sends emails without delays. Lets say if user wants to buy something on our site and he doesn't receives activation email on time, he will go on other site and buy that item there.

Many fancy systems like Mailchimp are having internal queues, shared among many accounts and we don't have access to them. It can happen that they send email in 2 seconds or 20 minutes, in some cases from my experience even in 1 hour. This could be fine for newsletter emails but for activation emails, password recovery, and other time critical emails we need to use faster systems.

The best email system on the market in Amazon's Simple Email Service or SES. It provides raw infrastructure for sending emails and it doesn't posses any analytics or internal queue. Because of this queue needs to be built into web application side or using 3rd party queue.

SES also posses hard limits of number of emails it can send per second. So if our application breaks this limit, SES will discard emails over that limit. This is not a problem when sending transactional emails, when sending campaigns it is important to keep track of sending speed.

Each month you can send 62.000 emails for free over AWS SES. There is no other service that provides that number of free emails per month.

AWS SES: Creating SMTP Credentials

When you login into AWS go to the list of Amazon web services, there you need to select SES.

Amazon Web Services List

Amazon Web Services List

After selecting SES you will be redirected to SES Dashboard. There you can see sending limits, number of sent emails and some metrics.

SES Dashboard

SES Dashboard

New user accounts on SES are in sandbox mode, and they can send emails only to verified addresses. By requesting a sending limit increase you will get better sending speed, and your app will be able to send emails to any address. For this tutorial I will work from sendbox mode, so I need to verify sending and receiving email address.

For this simple app I will use SMTP for sending emails, AWS SES also posses great API, which can be useful for huge email campaigns. To generate new set of SMTP keys I will visit SMTP Settings Tab.

SMTP Settings

SMTP Settings

By clicking on Create My SMTP Credentials button you will be redirected to IAM's page. IAM is AWS system for identity management ( simplified: generating keys for different AWS services ). That page will show text input field named IAM User Name, default name is fine and you can click create. After creating new set of keys, AWS will offer option to download that combination, that is always a good thing cause keys are visible only once, immediately after creating them.

Now we have all the info we need to send emails using AWS SES.

In production systems, AWS requests from senders to have web hooks which will catch bounce and comply requests from AWS. I will cover this in some of future tutorials.

Inserting SMTP details in .env file

From above AWS SMTP panel we can see host details and credentials are already downloaded.

MAIL_DRIVER=smtp
MAIL_HOST=email-smtp.us-west-2.amazonaws.com
MAIL_PORT=587
MAIL_USERNAME=AKIAJ3RS7BUIV7U2LCIQ
MAIL_PASSWORD=Av9/1W4ZfcinolyP+8PNvjyQk7owOxaIMUehg53/QhnH
MAIL_ENCRYPTION=tls

Queue Setup

For my Laravel applications I usually use Redis Server as Queue. For installation details please read my other tutorial dedicated to Redis, section Installing Redis.

During testing phase you can use sync driver for queue, that will mimic queue without actually using it. This helps you code everything like there is a queue, but without really configuring it.

In .env file just insert:

QUEUE_DRIVER=sync

Later you'll switch this with redis driver.

Because I am having 50 different services which use queue, I will create dedicated queue, if I go with default then emails from this application could wait longer. With this I'll create dedicated worker and system will work seamlessly.

In queue.php config file I will add new element in connections array.

// queue.php
// ...

        'social' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue'  => 'social',
            'expire' => 60,
        ],

// ...

From this new element you can see that I am using redis driver, connection settings are default and new queue is social

You can even create different queues for password resets and activation email, so they don't wait on each other.

Send Notification Emails to Queue

In this application most important email is Activation email, to be able to insert it in the queue first SendActivationEmail class should implement ShouldQueue interface.

// SendActivationEmail.php
// ...

class SendActivationEmail extends Notification implements ShouldQueue
{

// ...

Only with this modification Laravel will use queue specified in .env file for this email notification.

Because I am using dedicated queue for my emails I need to pass that queue name to this notification. We will do that by using Illuminate\Bus\Queueable trait and its method onQueue.

// SendActivationEmail.php
// ...

class SendActivationEmail extends Notification implements ShouldQueue
{
    use Queueable;

// ...

And passing queue name in constuctor of this notification is needed:

        $this->onQueue('social');

Queue Listener

Leaving project like this will not deliver any activation emails, they will be inserted into queue, but we didn't define any workers.

Workers are in charge of taking queue records one by one and processing them.

I can activate queue listener/worker manually with:

php artisan queue:listen --queue=social

Manually running queue listener is not something that I want to do, so I will automate this process.

Maintaining Queue Listener

As mentioned in Laravel documentation Supervisor is a process monitor tool for Linux operating systems. It can automatically restart given process and do a number of other tasks, for more visit Supervisor site.

Installation is simple and on Ubuntu I can install it with following command:

apt-get install supervisor

Supervisor configuration files are located in /etc/supervisor/conf.d directory. For each process I want to monitor I need to create new process configuration file.

So I will create email-over-queue.conf file.


[program:email-over-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /dev/projects/tutorials/laravel-social-email-authentication/artisan queue:work redis --queue=social --sleep=3 --tries=3
autostart=true
autorestart=true
user=forge
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/logs/tuts/laravel-social-email-authentication/storage/logs/supervisord.log

With this configuration listener will retry each job 3 times. Also Supervisor will restart listener if it fails or if system restarts.

Each process can have different log file this one is stored in custom folder.

Difference between queue:listen and queue:work is daemon mode, daemon mode never tries to re-boot the framework. This results in reduction of CPU usage, which is important for web servers.

You can find entire code used in this tutorial on Github.

Lessons of this Series

Trending

Newsletter

Subscribe to our newsletter for good news, sent out every month.

Tags