Realtime with serverless using Websocket in AWS

Hello everyone, So today we are going to discuss about implementing websocket with serverless by using Node.js with Typescript for realtime communication.

If you try to search it on internet, you will find pieces of implementation details but nothing concrete is available. So, here I am building a full fledged websocket ecosystem with all the code & configuration details.

For development, you need to have serverless, npm and/or yarn globally installed. See how to install YARN and NPM. I also prefer using NVM to manage my Node versions.

Development

Step 1. Configure AWS

Setup AWS CLI if you have not already, there is a very good article here on create and configure AWS credentials. Next, we need to set up our application.

Step 2. Setup Project

Create and set up a serverless project with typescript:

$ sls create --template aws-nodejs-typescript --path <PROJECT-NAME>

Where <PROJECT-NAME> is the name of your project.

This generates a serverless boilerplate for our application. Next, we need to navigate to our new project cd <PROJECT-NAME>and install our dependencies by running yarn install.

The most important files we will be updating as we develop our application are the handler.ts and serverless.ts . The handler.ts file handles our lambda functions or references to our lambda functions. It should look like this:

import { APIGatewayProxyHandler } from 'aws-lambda';
import 'source-map-support/register';

export const hello: APIGatewayProxyHandler = async (event, _context) => {
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Go Serverless Webpack (Typescript) v1.0! Your function executed successfully!',
      input: event,
    }, null, 2),
  };
}

The serverless.ts file provides us with a template to model and provision our application resources for AWS CloudFormation by treating infrastructure as code. This helps us create, update and even delete resources easily. It should look like this:

import type { Serverless } from 'serverless/aws';

const serverlessConfiguration: Serverless = {
  service: {
    name: 'serverless-websocket-ts',
    // app and org for use with dashboard.serverless.com
    // app: your-app-name,
    // org: your-org-name,
  },
  frameworkVersion: '>=1.72.0',
  custom: {
    webpack: {
      webpackConfig: './webpack.config.js',
      includeModules: true
    }
  },
  // Add the serverless-webpack plugin
  plugins: ['serverless-webpack'],
  provider: {
    name: 'aws',
    runtime: 'nodejs12.x',
    apiGateway: {
      minimumCompressionSize: 1024,
    },
    environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
    },
  },
  functions: {
    hello: {
      handler: 'handler.hello',
      events: [
        {
          http: {
            method: 'get',
            path: 'hello',
          }
        }
      ]
    }
  }
}

module.exports = serverlessConfiguration;

Also, due to deprecations you will need to set the service property directly to service name:

{ 
  service: 'serverless-websocket-ts',
  ...
}

And update the provider.apiGateway object as follows:

provider: {
  ...
  apiGateway: {
    shouldStartNameWithService: true,
    ...
  },
  ...
},

We can run our function in our project root $ serverless invoke local --function hello and you should see the response:

{
    "statusCode": 200,
    "body": "{n  "message": "Go Serverless Webpack (Typescript) v1.0! Your function executed           successfully!",n  "input": "n}"
}

The serverless framework locally invokes the hello function and runs the exported hello method in the handler.ts file. The serverless invoke local command allows us to run our Lambda functions locally before they are deployed.

Next, we need to install a few dependencies for our application:

$ yarn add -D serverless-offline serverless-dotenv-plugin serverless-bundle
$ yarn add aws-sdk uuid @types/uuid

• aws-sdk — allows us to communicate with AWS services
• uuid — generates unique ids for our database entries
• serverless-offline— this plugin enables us to run our application and Lambda functions locally
• serverless-dotenv-plugin — to enable us load .env variables into our Lambda environment
• serverless-bundle — this plugin optimally packages our Typescript functions and ensures that we don’t need to worry about installing Babel, Typescript, Webpack, ESLint and a host of other packages. This means that we don’t need to maintain our own webpack configs. So we can go ahead and delete the webpack.config.js file and the reference to it in serverless.ts custom property:

// FROM
custom: {
  webpack: {
    webpackConfig: './webpack.config.js',
    includeModules: true
  }
}// TO
custom: {
  
}

Step 3. Setup Resources

Now we need to set region, stage, table_throughputs, table_throughput and connections_table variables. It also configures dynamodb and serverless-offline for local development.

import type { Serverless } from 'serverless/aws';

const serverlessConfiguration: Serverless = {
  service: 'serverless-todo',
  frameworkVersion: '>=1.72.0',
  custom: {
    region: '${opt:region, self:provider.region}',
    stage: '${opt:stage, self:provider.stage}',
    connections_table: '${self:service}-connections-table-${opt:stage, self:provider.stage}',
    table_throughputs: {
      prod: 5,
      default: 1,
    },
    table_throughput: '${self:custom.TABLE_THROUGHPUTS.${self:custom.stage}, self:custom.table_throughputs.default}',
    dynamodb: {
      stages: ['dev'],
      start: {
        port: 8008,
        inMemory: true,
        heapInitial: '200m',
        heapMax: '1g',
        migrate: true,
        seed: true,
        convertEmptyValues: true,
        // Uncomment only if you already have a DynamoDB running locally
        // noStart: true
      }
    },
    ['serverless-offline']: {
      httpPort: 3000,
      babelOptions: {
        presets: ["env"]
      }
    }
  },
  plugins: [
      'serverless-bundle',
      'serverless-offline',
      'serverless-dotenv-plugin',
  ],
  package: {
    individually: true,
  },
  provider: {
    name: 'aws',
    runtime: 'nodejs12.x',
    stage: 'dev',
    region: 'eu-west-1',
    apiGateway: {
      shouldStartNameWithService: true,
      minimumCompressionSize: 1024,
    },
    environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
    },
  },
  functions: {
    hello: {
      handler: 'handler.hello',
      events: [
        {
          http: {
            method: 'get',
            path: 'hello',
          }
        }
      ]
    }
  },
  
}

module.exports = serverlessConfiguration;

Next, we need to update the provider.environment property in theserverless.ts file as follows:

// serverless.ts
provider: {
  ...
  environment: {
    ...
    REGION: '${self:custom.region}',
    STAGE: '${self:custom.stage}',
    CONNECTIONS_TABLE: '${self:custom.connections_table}',
  },
  ...
},

Next, we need to add resources which are added into your CloudFormation stack when our application is deployed. We need to define AWS resources in a property titled resources.

You can read more about working with tables and data in DynamoDB.

Using the power of Javascript, we can avoid large service definition files by splitting into files and using dynamic imports. This is important because as our application grows, separate definition files makes it easier to maintain.

For this project, we will be organizing our resources separately and importing into serverless.ts. To do this, we need to first create a resources directory in our root directory and then create a dynamodb-tables.ts file for our DynamoDB tables:

// At project root
$ touch resources/dynamodb-tables.ts 

Next, we update the dynamodb-tables.ts file as follows:

export default {
    ConnectionsTable: {
        Type: 'AWS::DynamoDB::Table',
        Properties: {
            TableName: '${self:provider.environment.CONNECTIONS_TABLE}',
            AttributeDefinitions: [
                { AttributeName: 'connectionId', AttributeType: 'S' }
            ],
            KeySchema: [
                { AttributeName: 'connectionId', KeyType: 'HASH' }
            ],
            ProvisionedThroughput: {
                ReadCapacityUnits: '${self:custom.table_throughput}',
                WriteCapacityUnits: '${self:custom.table_throughput}'
            },
            SSESpecification: {
                SSEEnabled: true
            },
            TimeToLiveSpecification: {
                AttributeName: 'ttl',
                Enabled: true
            }
        }
    },
}

And import into serverless.ts — and set it in resources properties as given below,

// DynamoDB
import dynamoDbTables from './resources/dynamodb-tables';

const serverlessConfiguration: Serverless = {
  service: 'serverless-todo',
  ...
  resources: {
    Resources: dynamoDbTables,
  }
  ...
}

module.exports = serverlessConfiguration;

To run DynamoDB locally, we need to first install the serverless-dynamodb-local plugin:

$ yarn add -D serverless-dynamodb-local 

Next, we need to update serverless.ts plugins array:

plugins: [
    'serverless-bundle',
    'serverless-dynamodb-local',
    'serverless-offline',
    'serverless-dotenv-plugin',
],

To use the plugin, we need to install DynamoDB Local by running sls dynamodb install at the project root. Runningsls dynamodb start will start it locally:

Step 4. Websocket Integration

In this step, we are going to create web socket handler, to connect with websocket events.

You can check here in details about websocket events.

// create a websocket folder
mkdir websocket

cd websocket
touch handler.ts
touch index.ts
touch schemas.ts
touch broadcast.ts

handler.ts file will be like given below:

// handler.ts

import type { ValidatedEventAPIGatewayProxyEvent } from '@libs/apiGateway';
import { formatJSONResponse } from '@libs/apiGateway';
import { middyfy } from '@libs/lambda';
import * as AWS from 'aws-sdk';
import { getAllConnections, sendMessage } from './broadcast';
import schema from './schema';

const config: any = { region: "us-east-1" };
if (process.env.STAGE === process.env.DYNAMODB_LOCAL_STAGE) {
  config.accessKeyId = process.env.DYNAMODB_LOCAL_ACCESS_KEY_ID;
  config.secretAccessKey = process.env.DYNAMODB_LOCAL_SECRET_ACCESS_KEY;
  config.endpoint = process.env.DYNAMODB_LOCAL_ENDPOINT;
}

AWS.config.update(config);

const dynamodb = new AWS.DynamoDB.DocumentClient();

const connectionTable = process.env.CONNECTIONS_TABLE;

const websocketHandler: ValidatedEventAPIGatewayProxyEvent<typeof schema> = async (event) => {
  console.log(event);
  const { body, requestContext: { connectionId, routeKey }} = event;
  console.log(body, routeKey, connectionId);
  
  switch (routeKey) {
    case '$connect':
      await dynamodb.put({
        TableName: connectionTable,
        Item: {
          connectionId,
          // Expire the connection an hour later. This is optional, but recommended.
          // You will have to decide how often to time out and/or refresh the ttl.
          ttl: parseInt((Date.now() / 1000).toString() + 3600)
        }
      }).promise();
      break;

    case '$disconnect':
      await dynamodb.delete({
        TableName: connectionTable,
        Key: { connectionId }
      }).promise();
      break;

    case '$default':
    default:
      const connections = await getAllConnections();
      await Promise.all(
        connections.map(connectionId => sendMessage(connectionId, body))
      );
      break;
  }

  return formatJSONResponse({ statusCode: 200 });
}

export const wsHandler = middyfy(websocketHandler);

The API-Gateway provides 4 types of routes which relate to the lifecycle of a ws-client:

Based on that, we have created a switch case, where every case will do appropriate action related to that event.

schema.ts will be responsible for validating the schema.

// schema.ts

export default {
  type: "object",
  properties: {
    name: { type: 'string' }
  },
  required: ['name']
} as const;

index.ts will map the websocket events with handler:

// index.ts

import { handlerPath } from '@libs/handlerResolver';

export const wsHandler = {
  handler: `${handlerPath(__dirname)}/handler.wsHandler`,
  events: [
    {
      websocket: '$connect'
    },
    {
      websocket: '$disconnect',
    },
    {
      websocket: '$default'
    }
  ]
};

We need to register wsHandler export in src/functions/index.ts

// src/functions/index.ts

export { default as hello } from './hello';
export { wsHandler } from './websocket';

broadcast.ts contains 2 important functionsas given below

import * as AWS from 'aws-sdk';

const config: any = { region: "us-east-1" };
if (process.env.STAGE === process.env.DYNAMODB_LOCAL_STAGE) {
    config.accessKeyId = process.env.DYNAMODB_LOCAL_ACCESS_KEY_ID;
    config.secretAccessKey = process.env.DYNAMODB_LOCAL_SECRET_ACCESS_KEY;
    config.endpoint = process.env.DYNAMODB_LOCAL_ENDPOINT;
}

AWS.config.update(config);

const dynamodb = new AWS.DynamoDB.DocumentClient();

const connectionTable = process.env.CONNECTIONS_TABLE;

export async function sendMessage(connectionId, body) {
    try {
        const endpoint = process.env.APIG_ENDPOINT;
        const apig = new AWS.ApiGatewayManagementApi({
            apiVersion: '2018-11-29',
            endpoint
        });
        await apig.postToConnection({
            ConnectionId: connectionId,
            Data: JSON.stringify(body)
        }).promise();
    } catch (err) {
        // Ignore if connection no longer exists
        if (err.statusCode !== 400 && err.statusCode !== 410) {
            throw err;
        }
    }
}

export async function getAllConnections() {
    const { Items, LastEvaluatedKey } = await dynamodb.scan({
        TableName: connectionTable,
        AttributesToGet: ['connectionId']
    }).promise();

    const connections = Items.map(({ connectionId }) => connectionId);
    if (LastEvaluatedKey) {
        connections.push(...await getAllConnections());
    }

    return connections;
}

Step 4. Websocket Running Locally

In this step, we will run websocket locally, using serverless offline:

Before we start, Compare & confirm serverless.ts file as given below

// serverless.ts

import type { AWS } from '@serverless/typescript';

import hello from '@functions/hello';
import { wsHandler } from '@functions/websocket';
import dynamoDbTables from './dynamodb-tables';


const serverlessConfiguration: AWS = {
  service: 'serverless-websocket-ts',
  frameworkVersion: '2',
  custom: {
    region: '${opt:region, self:provider.region}',
    stage: '${opt:stage, self:provider.stage}',
    prefix: '${self:service}-${self:custom.stage}',
    connections_table: '${self:service}-connections-table-${opt:stage, self:provider.stage}',
    ['serverless-offline']: {
      httpPort: 3000,
    },
    ['bundle']: {
      linting: false
    },
    table_throughputs: {
      prod: 5,
      default: 1,
    },
    table_throughput: '${self:custom.TABLE_THROUGHPUTS.${self:custom.stage}, self:custom.table_throughputs.default}',
    dynamodb: {
      stages: ['dev'],
      start: {
        port: 8008,
        inMemory: true,
        heapInitial: '200m',
        heapMax: '1g',
        migrate: true,
        seed: true,
        convertEmptyValues: true,
        // Uncomment only if you already have a DynamoDB running locally
        // noStart: true
      }
    }
  },
  plugins: [
    'serverless-bundle',
    'serverless-dynamodb-local',
    'serverless-offline',
    'serverless-dotenv-plugin',
  ],
  package: {
    individually: true,
  },
  provider: {
    name: 'aws',
    runtime: 'nodejs14.x',
    apiGateway: {
      minimumCompressionSize: 1024,
      shouldStartNameWithService: true,
    },
    environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
      REGION: '${self:custom.region}',
      STAGE: '${self:custom.stage}',
      APIG_ENDPOINT: 'http://localhost:3001',
      CONNECTIONS_TABLE: '${self:custom.connections_table}',
    },
    lambdaHashingVersion: '20201221',
  },
  // import the function via paths
  functions: { 
    hello,
    wsHandler
  },

  resources: {
    Resources: dynamoDbTables,
  }
};

module.exports = serverlessConfiguration;

$ serverless offline start

Serverless: Bundling with Webpack...
Serverless: Watching for changes...
Dynamodb Local Started, Visit: http://localhost:8008/shell
Issues checking in progress...
No issues found.
Serverless: DynamoDB - created table serverless-websocket-ts-connections-table-dev
offline: Starting Offline: dev/us-east-1.
offline: Offline [http for lambda] listening on http://localhost:3002
offline: Function names exposed for local invocation by aws-sdk:
           * hello: serverless-websocket-ts-dev-hello
           * wsHandler: serverless-websocket-ts-dev-wsHandler   
offline: route '$connect'
offline: route '$disconnect'
offline: route '$default'
offline: Offline [websocket] listening on ws://localhost:3001

   ┌─────────────────────────────────────────────────────────────────────────┐
   │                                                                         │
   │   POST | http://localhost:3000/dev/hello                                │
   │   POST | http://localhost:3000/2015-03-31/functions/hello/invocations   │
   │                                                                         │
   └─────────────────────────────────────────────────────────────────────────┘

offline: Offline [http for websocket] listening on http://localhost:3001
offline: [HTTP] server ready: http://localhost:3000  
offline:
offline: Enter "rp" to replay the last request

Step5: Websocket Demo

I hope you find this post useful!

Unimedia Technology

Here at Unimedia Technology we have a team of BackEnd Developers that can help you develop your most complex Applications