Skip to content

Latest commit

 

History

History
186 lines (131 loc) · 5.62 KB

queue.md

File metadata and controls

186 lines (131 loc) · 5.62 KB

Queue

Some tasks are too long to be processed synchronously. Instead, they can be processed in the background via a job queue and worker.

The queue construct deploys a properly configured SQS queue with a worker running on AWS Lambda.

Quick start

service: my-app
provider:
    name: aws

constructs:
    my-queue:
        type: queue
        worker:
            handler: src/report-generator.handler

plugins:
    - serverless-lift

On serverless deploy, a my-queue queue will be created, and a Lambda function will be deployed to process jobs (aka "messages") from the queue.

How it works

The queue construct deploys the following resources:

  • An SQS queue: this is where jobs to process should be sent.
  • A "worker" Lambda function: this function will be processing each job sent to the SQS queue.
  • An SQS "Dead Letter Queue": this queue will contain all the jobs that failed to be processed.
  • Optionally, a CloudWatch alarm that sends an email if jobs land in the Dead Letter Queue.

To learn more about the architecture of this construct, read this article.

Variables

All queue constructs expose the following variables:

  • queueUrl: the URL of the deployed SQS queue
  • queueArn: the ARN of the deployed SQS queue

These can be used to reference the queue from other Lambda functions, for example:

constructs:
    my-queue:
        type: queue

functions:
    otherFunction:
        handler: src/publisher.handler
        environment:
            QUEUE_URL: ${construct:my-queue.queueUrl}

How it works: the ${construct:my-queue.queueUrl} variable will automatically be replaced with a CloudFormation reference to the SQS queue.

Permissions

By default, all the Lambda functions deployed in the same serverless.yml file will be allowed to push messages into the queue.

In the example below, there are no IAM permissions to set up: myFunction will be allowed to send messages into my-queue.

constructs:
    my-queue:
        type: queue
        # ...

functions:
    myFunction:
        handler: src/publisher.handler
        environment:
            QUEUE_URL: ${construct:my-queue.queueUrl}

Configuration reference

Worker

constructs:
    my-queue:
        type: queue
        worker:
            # The Lambda function is configured here
            handler: src/report-generator.handler

Note: the Lambda "worker" function is configured in the queue construct, instead of being defined in the functions section.

The only required setting is the handler: this should point to the code that handles SQS messages. The handler should be written to handle SQS events, for example in JavaScript:

exports.handler = async function (event, context) {
    event.Records.forEach(record => {
        // `record` contains the job that was pushed to SQS
    });
}

All settings allowed for functions can be used under the worker key. For example:

constructs:
    my-queue:
        # ...
        worker:
            handler: src/report-generator.handler
            memorySize: 512
            timeout: 10

Note: Lift will automatically configure the function to be triggered by SQS. It is not necessary to define events on the function.

Alarm

constructs:
    my-queue:
        # ...
        alarm: [email protected]

It is possible to configure email alerts in case jobs end up in the dead letter queue.

After the first deployment, an email will be sent to the email address to confirm the subscription.

Retries

constructs:
    my-queue:
        # ...
        maxRetries: 5

Default: 3 retries.

The maxRetries option configures how many times each job will be retried when failing.

If the job still fails after reaching the max retry count, it will be moved to the dead letter queue for storage.

Retry delay

When Lambda fails processing a SQS job (i.e. the code throws an error), the job will be retried after a delay. That delay is also called "Visibility Timeout" in SQS.

By default, Lift configures the retry delay to 6 times the worker functions timeout, per AWS' recommendation. Since Serverless deploy functions with a timeout of 6 seconds by default, that means that jobs will be retried every 36 seconds.

When the function's timeout is changed, the retry delay is configured accordingly:

constructs:
    my-queue:
        # ...
        worker:
            handler: src/report-generator.handler
            # We change the timeout to 10 seconds
            timeout: 10
            # The retry delay on the queue will be 10*6 => 60 seconds

Batch size

constructs:
    my-queue:
        # ...
        batchSize: 5 # Lambda will receive 5 messages at a time

Default: 1

When the SQS queue contains more than 1 job to process, it can invoke Lambda with a batch of multiple messages at once.

By default, Lambda will be invoked 1 messages at a time. The reason is to simplify error handling: in a batch, any failed message will fail the whole batch.

It is possible to change the batch size between 1 and 10.

More options

Looking for more options in the construct configuration? Open a GitHub issue.