Steven.Lee

Serverless AWS SQS (Part 1)

Setup from scratch to work in offline/local mode

Steven Lee's photo
Steven Lee
·

6 min read

What I will share in this post:

To get started

Install Serverless

npm install -g serverless

Create serverless template to get started

For my case, I'm using NodeJS in this project.

sls create --template aws-nodejs --path aws-sqs-offline

The command above will create aws-sqs-offline directory with serverless.yml and handler.js in it.

serverless.yml is the serverless template file that you can define your AWS stack.

handler.js is the source code of the lambda function defined in serverless.yml.

Install serverlesss-offline

// Go into your project directory.
$ cd aws-sqs-offline
// Initialized npm project
$ npm init -y
$ npm install -D serverless-offline

Once that are done, add the plugins section, like the code below in the serverless.yml to enable serverless offline plugin.

plugins:
  - serverless-offline

Add npm command script to start dev in offline mode.

  "scripts": {
    "offline": "OFFLINE=TRUE sls offline start"
  },

In you terminal, execute npm run offline, you should be able to see the lambda function (hello) is created, together with your local port.

offline: Starting Offline: dev/us-east-1.
offline: Offline [http for lambda] listening on http://localhost:3002
offline: Function names exposed for local invocation by aws-sdk:
           * hello: aws-sqs-offline-dev-hello

At this point, only the hello lambda is created, you can use the method below to test hello function.

  1. Define API Gateway and bind to the lambda function.
  2. Invoke the lambda function using serverless CLI.

In this post, i'm gonna to use serverless CLI to invoke the lambda function.

sls invoke local -f hello

You should be able to see the output below:

{
    "statusCode": 200,
    "body": "{\n  \"message\": \"Go Serverless v1.0! Your function executed successfully!\",\n  \"input\": \"\"\n}"
}

Install Serverless Offline SQS

Before you install serverless-offline-sqs, you will need to install ElasticMQ.

To install ElasticMQ

mkdir -p .elasticmq && cd .elasticmq && wget https://s3-eu-west-1.amazonaws.com/softwaremill-public/elasticmq-server-0.15.7.jar

To configure ElasticMQ

touch elasticmq.conf

Add below configuration in elasticmq.conf

include classpath("application.conf")

node-address {
    protocol = http
    host = localhost
    port = 9324
    context-path = ""
}

rest-sqs {
    enabled = true
    bind-port = 9324
    bind-hostname = "127.0.0.1"
    sqs-limits = strict
}

generate-node-address = false

queues {
    "myFirstQueue" {
        defaultVisibilityTimeout = 10 seconds
        delay = 0 seconds
        receiveMessageWait = 0 seconds
        deadLettersQueue {
            name = "myFirstQueue-deadletter-queue"
            maxReceiveCount = 3
        }
        fifo = false
        contentBasedDeduplication = false
    }
    myFirstQueue-deadletter-queue {
        fifo = false
    }
}

aws {
    region = us-east-1
    accountId = 000000000000
}

Add npm command script to start the queue system.

  "scripts": {
    "offline": "OFFLINE=TRUE sls offline start",
    "queue-start": "java -Dconfig.file=elasticmq.conf -jar .elasticmq/elasticmq-server-0.15.7.jar"
  },

Now test the queue command.

npm run queue-start

The queue should be working now.

> java -Dconfig.file=elasticmq.conf -jar .elasticmq/elasticmq-server-0.15.7.jar

14:49:33.983 [main] INFO  org.elasticmq.server.Main$ - Starting ElasticMQ server (0.15.7) ...
14:49:34.301 [elasticmq-akka.actor.default-dispatcher-5] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
14:49:34.950 [elasticmq-akka.actor.default-dispatcher-6] INFO  o.e.rest.sqs.TheSQSRestServerBuilder - Started SQS rest server, bind address 127.0.0.1:9324, visible server address http://localhost:9324
14:49:34.978 [elasticmq-akka.actor.default-dispatcher-6] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue QueueData(myFirstQueue-deadletter-queue,MillisVisibilityTimeout(30000),PT0S,PT0S,2021-07-18T14:49:34.955+08:00,2021-07-18T14:49:34.955+08:00,None,false,false,None,None,Map())
14:49:34.991 [elasticmq-akka.actor.default-dispatcher-6] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue QueueData(myFirstQueue,MillisVisibilityTimeout(10000),PT0S,PT0S,2021-07-18T14:49:34.991+08:00,2021-07-18T14:49:34.991+08:00,Some(DeadLettersQueueData(myFirstQueue-deadletter-queue,3)),false,false,None,None,Map())
14:49:34.993 [main] INFO  org.elasticmq.server.Main$ - === ElasticMQ server (0.15.7) started in 1309 ms ===

Once ElasticMQ is in place, you can install serverless-offline-sqs.

npm install -D serverless-offline-sqs

Add the serverless-offline-sqs plugin in serverless.yml

plugins:
  - serverless-offline-sqs
  - serverless-offline

Now, we need to configure the integration of serverless-offline-sqs with ElasticMQ. Add below configuration into serverless.yml:

custom:
  serverless-offline:
    host: 0.0.0.0
    httpPort: 3000
    lambdaPort: 3001
    useChildProcesses: false
  serverless-offline-sqs:
    sqsHost: 127.0.0.1
    sqsPort: 9324
    autoCreate: false
    apiVersion: "latest"
    endpoint: http://127.0.0.1:9324
    region: us-east-1
    accessKeyId: local
    secretAccessKey: local
    skipCacheInvalidation: false

Define AWS SQS & AWS Lambda

At this point, we have successfully setup serverless-offline, ElasticMQ and serverless-offline-sqs.

In order to get all these work together, we have to define AWS SQS, what I plan to do here is to define AWS SQS Event that trigger will trigger Lambda function, and print the message output in the Lambda.

To do that, we need to define 1 SQS queue resource, 2 Lambda functions.

  • SQS Queue resource as the source of Lambda event.
  • Lambda function to send SQS message.
  • Lambda function to receive SQS message.

Note MAKE SURE you start ElasticMQ before you run npm run offline.

In order to test a local Lambda function, you should have this running in sequence:

  1. $ npm run queue-start
  2. $ npm run offline
  3. $ sls invoke local -f hello

This also means that you need to have at least 3 terminal opened to run these 3 commands.

To define AWS SQS resource in serverless.yml

resources:
  Resources:
    myFirstQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: myFirstQueue
        MessageRetentionPeriod: 1209600
        RedrivePolicy:
          deadLetterTargetArn:
            Fn::GetAtt:
              - myFirstQueue
              - Arn
          maxReceiveCount: 3
        VisibilityTimeout: 10

To define AWS Lambda function in serverless.yml

functions:
  hello:
    handler: handler.hello

  QueueSendMessage:
    handler: handler.sendMessage
    name: Queue-SendMessage-Lambda
    description: to send sqs message

  QueueReceiveMessage:
    handler: handler.receiveMessage
    name: Queue-ReceiveMessage-Lambda
    description: to receive sqs message
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - myFirstQueue
              - Arn
          batchSize: 10

Now we had defined the Lambda functions, but the actual source code is not being created yet. To create the Lambda functions that we just defined, in handler.js, we need to add 2 more functions, the function name must be the same that we have just defined in the serverless.yml, the handler under the functions section.

module.exports.sendMessage = async (event) => {
  return {
    message: 'sendMessage'
  }
};
module.exports.receiveMessage = async (event) => {
  return {
    message: 'receiveMessage'
  }
};

You can test these Lambda functions through serverless CLI

$ sls invoke local -f QueueSendMessage
// Expected output:
// {
//    "message": "sendMessage"
// }
$ sls invoke local -f QueueReceiveMessage
// Expected output:
// {
//    "message": "receiveMessage"
// }

Send SQS Message through Lambda function

Before we jump into the code, we need install aws-sdk to send SQS message.

npm install aws-sdk

Now, modify the function module.exports.sendMessage in handler.js

module.exports.sendMessage = async (event) => {
  const AWS = require("aws-sdk");
  const SQS = new AWS.SQS({
    accessKeyId: "local",
    secretAccessKey: "local",
    endpoint: "127.0.0.1:9324"
  });

  try {

    const queueParams = {
      Entries: [
        {
          Id: "1",
          MessageBody: "this is a message body",
        }
      ],
      QueueUrl: 'http://127.0.0.1:9324/queue/myFirstQueue'
    }

    const result = await SQS.sendMessageBatch(queueParams).promise();
    console.log(JSON.stringify(result, null, 2));
  } catch (e) {
    console.error(e);
  }
};

Like before, you can trigger send message function by using serverless CLI.

$ sls invoke local -f QueueSendMessage
// Expected output:
{
  "ResponseMetadata": {
    "RequestId": "00000000-0000-0000-0000-000000000000"
  },
  "Successful": [
    {
      "Id": "1",
      "MessageId": "dab5444c-e401-46a8-b0de-abd600169872",
      "MD5OfMessageBody": "592cc4f6866f8c9fc50f77324e2b8c6f"
    }
  ],
  "Failed": []
}

Invoke Lambda through SQS Message event (Receive SQS Message)

The function definition in serverless.yml actually already bind SQS message and Lambda function together. Whenever a message is received, QueueReceiveMessage will be triggered/invoked.

Lambda event definition:

events:
  - sqs:
      arn:
        Fn::GetAtt:
          - myFirstQueue
          - Arn
      batchSize: 10

Now, let's print the received message in the Lambda function. Modify receiveMessage function in handler.j

module.exports.receiveMessage = async (event) => {
  console.log(JSON.stringify(event.Records, null, 2));
};

You should be able to see the result below by invoke the sendMessage function:

$ sls invoke local -f QueueSendMessage

[
  {
    "messageId": "cf79a371-41d0-4abc-82a6-2b9b8d1571ad",
    "receiptHandle": "cf79a371-41d0-4abc-82a6-2b9b8d1571ad#27c0a3ef-ba74-4eaa-acb9-3bcdff46115a",
    "body": "this is a message body",
    "attributes": {
      "SentTimestamp": "1626594893106",
      "ApproximateReceiveCount": "1",
      "ApproximateFirstReceiveTimestamp": "1626594893106",
      "SenderId": "127.0.0.1",
      "MessageDeduplicationId": "",
      "MessageGroupId": ""
    },
    "messageAttributes": {},
    "md5OfBody": "592cc4f6866f8c9fc50f77324e2b8c6f",
    "eventSource": "aws:sqs",
    "eventSourceARN": "arn:aws:sqs:us-east-1:000000000000:myFirstQueue"
  }
]

Full source code can be found here :

What's next?

The next post I will be sharing:

  • Update project structure.
  • Convert to Typescript.
  • Production deployment.
AWSaws lambdaserverless