Configuring Queue Timeouts1:58
10 seconds to the queue functions timeout configuration, which is 60 seconds by default. The visibility timeout controls the time SQS waits for between your lambda receiving a job and SQS making it visible again for a retry. This value should always be greater than the job or the functions timeout. To control the default queue timeout, let's head over to the Vapor.yaml file and add a queue timeout attribute. Let's set it to 90 seconds instead of the default 60 seconds and with that the queue Vapor created will have a visibility timeout of 100 seconds. We can also change the maximum concurrency by adding a queue concurrency attribute. Let's set this one to 10. Now the SQS lambda integration will have 10 threads pulling from the SQS queue and invoking our queue function. You can think of those pulling threads as a regular Laravel queue worker. They monitor the queue, receive jobs, invoke the lambda, and go back to monitoring the
Default vs Custom Queues3:02
and invoking our queue function. You can think of those pulling threads as a regular Laravel queue worker. They monitor the queue, receive jobs, invoke the lambda, and go back to monitoring the queue again. Now the queue Vapor creates for every environment is used as the default queue, meaning that if you dispatch a job from your application without specifying any queue, this job will be dispatched to that Vapor created queue. If your application was spelled to dispatch jobs to multiple queues, you will need to inform Vapor about those custom queues in the Vapor.yaml file. However, there is a catch there. Let's say you have queue concurrency set to 10 and you have two custom queues that your app dispatches job to. The built-in SQS lambda integration will start 10 pulling threads for the first queue and another 10 threads for the second queue. That's total 20 threads pulling from the function. And since the queue lambda function has queue concurrency set
pulling threads for the first queue and another 10 threads for the second queue. That's total 20 threads pulling from the function. And since the queue lambda function has queue concurrency set to 10, it can only process 10 jobs at any given time. That means if the pulling threads were able to retrieve more than 10 jobs at any given time, only 10 will be processed and the other others will be throttled. And when a job is throttled, two things happen. First, they will be taken out of the queue and will only become available again after the visibility timeout of the queue passes, meaning that these jobs won't be available, like they will sit there idle, not being processed, and other threads won't be able to catch them. And second, the number of attempts will be incremented, even though the job wasn't processed at all. Let me show you how we can configure custom queues first and then we will talk about how to deal with that throttling issue. In our
Creating Custom Queues4:56
incremented, even though the job wasn't processed at all. Let me show you how we can configure custom queues first and then we will talk about how to deal with that throttling issue. In our vapor.yaml file, we will add a queues attribute. Then we will provide a list of queue names. I'll add default, laracasts-staging, and notifications-laracasts-staging. Now let's deploy. Notice that I'm giving my queues a unique name. That's because vapor will create the name of the queue just exactly as we provided it. And if we have multiple projects or multiple environments on the same AWS account that consume jobs from the same queue, we won't be able to predict which environment will process which job. So when using custom queues in vapor, you have to provide unique queue names. After the deployment is done, let's head over to the SQS console. And we can see that two new queues were created for us. Let's head back to the terminal and dispatch a job. We will
Dispatching and Verifying Jobs5:58
queue names. After the deployment is done, let's head over to the SQS console. And we can see that two new queues were created for us. Let's head back to the terminal and dispatch a job. We will run the vapor tinker command to open a tinker session. Then use sayHello, dispatch on queue default LaraCast staging. Then we run this code. And here we go. The job was dispatched. Let's verify that it ran. We will go to the vapor dashboard, then click on the staging environment, then logs, then we choose the queue function. Vapor opens the CloudWatch console for us to display the logs for the queue function. And here we go. We can see that the job ran and printed hello. If we check the job code, we can see that the handle function or the handle method just prints hello. So our job ran successfully. Let's take a look at how we dispatch the job. We had to provide the full queue name, which is not ideal as the same code base can be used.
Using SQS Queue Suffix7:04
prints hello. So our job ran successfully. Let's take a look at how we dispatch the job. We had to provide the full queue name, which is not ideal as the same code base can be used for multiple environments. We would like to be able to dispatch to a queue by specifying only its friendly name, not the full name. And to do that, let's copy the part we want to ignore, head over to the vapor dashboard, go back to the environment, click on environment variables, and then we add an SQS_SUFFIX environment variable. And we provide the portion we just copied. Finally, let's head back to the terminal and deploy. Once the deployment is done, let's open a php artisan tinker session and dispatch the job. This time, we will dispatch it to the default queue. Let's run the code now. And here we go. The Laravel application was able to dispatch this job to our queue. Now, as we mentioned, when using multiple queues with a lambda trigger,
Handling Throttling Limits8:09
Let's run the code now. And here we go. The Laravel application was able to dispatch this job to our queue. Now, as we mentioned, when using multiple queues with a lambda trigger, the SQS lambda integration isn't smart enough to respect the function concurrency limits. And to deal with this issue, Vapor handles the job attempts counter internally without relying on the reports coming from SQS itself. That way, even if the jobs are throttled, the number of attempts will be correct. However, since you don't control prioritization when using multiple queues anyway, it no longer makes sense. I recommend that you stick to using a single queue and dispatch all your jobs there. That way, you won't get any throttles. Any questions, ask below.
