Serverless workflow for Crypto and Stock price volatility on AWS Cloud
There are times when we frequently check price movements due to the volatility of the market. Constant monitoring is required to overcome the fear of missing out when the market moves up and anxiety when it moves down.
Market volatility is inevitable and it’s very difficult to time the market. I wanted to engineer a system that sends an alert message or an email when there are price movements on my favorite stocks or crypto. The need for the tediously continuous monitoring of the market becomes obsolete in this case.
In this project, live data will be collected from Yahoo Finance for crypto and stocks based on the Ticker on required time intervals and compared with the most recent data for change in the price. If there is a change in the price of certain percent then an email is sent to the user.
Here is the architecture of this workflow:
Here is the code repository for this project: https://github.com/manideep1116/serverless-workflow
I have used multiple AWS services to build this serverless system on cloud. These include AWS Lambda, DynamoDB, SQS, SES, CloudWatch events and SAM for Infrastructure as code. AWS SAM is an extension of CloudFormation with simpler syntax for configuring serverless resources.
Overview:
Referring to the above diagram, CloudWatch events trigger SQS on minute intervals or hour intervals. For every trigger or message, SQS invokes Lambda function and it gets the live Stock/Crypto price data from Yahoo Finance and adds data as records to the DynamoDB table.
For every row or record added in the table for a particular stock/crypto, DynamoDB streams will trigger another Lambda function which performs a query on the DynamoDB table and compares the price from the new record to the most recent record. If the price rises or drops by a certain percent, lambda function will email the user using SES with the message on the price movement.
For example, if the CloudWatch events are set for 1 hour time intervals and the first Lambda stores the price for “ETH-USD” at 12:00 pm as $3000 in the DynamoDB table and after an hour when it adds another value for “ETH-USD” price at 1:00pm as $3100, DynamoDB streams trigger the second Lambda to query the table and compare the price at 1:00 pm to the price at 12:00pm and if there is desired percentage change then user will get an email.
CloudWatch Events:
CloudWatch Events delivers a near real-time stream of system events when there are changes in any AWS resources.
For example, if we want to monitor the state of an EC2 instance and get notified when the state of EC2 changes to ‘stopped’ or ‘terminated’, we can create a customized Event pattern in CloudWatch events and trigger some actions to notify the team through any target such as SNS when state of the EC2 changes.
CloudWatch events can also be used as cron jobs in the cloud. Using CloudWatch events we can schedule automated actions that self-trigger at certain times using cron or rate expressions. In this case, the CloudWatch event triggers the SQS on every minute or every hour as required. Every trigger is stored as a message in the SQS queue.
Amazon Simple Queue Service(SQS):
SQS is the message queuing service that lets you store or retrieve messages in the queue, which basically tells that there is a job that needs to be done.
Using SQS here is optional as we can directly let CloudWatch events trigger lambda. The main purpose of SQS here is when the compute time is more than the event time or if there is any logic issue in the Lambda, the messages are stored in SQS and will be processed without missing them. In this case, the Lambda function is integrated with SQS so that every time a message gets stored in the SQS the Lambda function is Invoked and performs its job.
Amazon Lambda:
Lambda is the serverless compute service that runs the code without provisioning or managing any servers. Lambda functions can be invoked in response to events from other AWS services.
When the Lambda function is invoked, it will get the data from Yahoo Finance and stores it in the DynamoDB table. Data is collected using an open source Python package called yahoo_fin.
Lambda will also add a record of its invocations to the CloudWatch logs.
All the required stocks/crypto tickers are stored in a Python list as shown below.
Amazon DynamoDB:
DynamoDB is a fully-managed NoSQL database service that allows us to create a database table that can store and retrieve any amount of data and serve any level of traffic.
For every lambda invocation, a new item is added with three columns or attributes. Here, ticker is the partition key, timestamp is the sort key and price is an attribute. We can potentially have more and different attributes specific to an item or row.
partition key — A simple primary key, composed of one attribute known as the partition key.
Partition key and sort key — Referred to as a composite primary key, this type of key is composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key.
To retrieve the data from the table we can either perform a SCAN or QUERY on the data.
Scan performs a scan on the entire table, if we want we can add filters to it as shown below. The filter applied below is to retrieve items with attribute “BTC-USD” and Type String.
In order to perform a Query, we have to provide the value for the primary key because the Query table basically goes and searches for the record in the primary key. As shown below, the Query is performed for ticker(partition key) that matches with “ETH-USD” and we can also provide a specific timestamp for sort key for a more narrow search. We can query the items in descending order or for most recent items on top using the check box “Sort descending”
Scan is typically not recommended because Scan picks up all the records in the table and then applies filter. Scan operations can be slow for large tables. Whereas, Query will only pick up the records that will match the primary key. We can query any table or secondary index that has a composite primary key (partition key + sort key).
DynamoDB Streams:
DynamoDB Streams is a very powerful feature that captures information about every modification to data in the table and emits events on record modifications occurring on a DynamoDB table.
There are three types of events INSERT(adding data), UPDATE(changing data) and REMOVE(deleting data). Below is the sample record with INSERT event.
Events are customizable, that means we can have new or old versions of data or both in the event — Keys only, New Image, Old Image, New & Old Image.
Streams can be integrated with AWS Lambda easily and triggers Lambda for every change in the table.
How streams work?
Streams consists of a number of shards managed by DynamoDB. Shards are like containers for change events in the table. When a change event is created it is added to the shard and associated with a sequence number. Lambda concurrently polls the shards(4 times / sec / shard) for any events. If it finds any event it will execute the lambda function.
If there are any issues in the lambda code and it fails to process the events, the events are still present in the shard. We can place them in the dead letter queue and process them later.
In our case, Whenever a new item is added to the table, DynamoDB streams will trigger the second lambda by sending the similar event with records as shown above. In order for Streams to work make sure to enable the Stream view type as shown in below image with “New and old images”.
For every new event, the second Lambda is triggered and performs a query for the item added in the table with sort descending order checked. This gets the most recent items on the top of the table for that particular ticker. Now, it takes the price value from the top two items which are most recent including the new item and compares the price for a certain percentage change. If the condition is true then it sends out an email using SES. Refer to stream.py in the code/ directory of the repository for this code.
Amazon SES:
Simple Email Service is an email service that provides an easy, cost-effective way to send and receive email using your own email addresses and domains.
Sender’s email address needs to be verified from the AWS Console in SES section to send an email. We can email recipients without their consent and verification.
In our case, the second Lambda will email the users on price movements using the boto3 library.
Pricing:
All the services can be used under a free tier account. To be cautious, set the CloudWatch events in the template.yaml to trigger less frequently as needed.
Lambda - Free Tier: 1 million requests per month, 400,000 GB-seconds of compute time per month
DynamoDB - DynamoDB charges for reading, writing, and storing data in your DynamoDB tables. Data storage of 25GB pr month is free.
SQS - First 1 Million Requests/Month are FREE.
SES - $0.10 for every 1,000 emails you send.
CloudWatch events- $1.00/million custom events published
Usage:
Pre-requisites:-
* AWS FREE Tier Account.
* SAM CLI — [Install the SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
* [Python 3 installed](https://www.python.org/downloads/)
* Docker — [Install Docker community edition](https://hub.docker.com/search/?type=edition&offering=community)
After installing the above pre-requisites, clone the repo and add any tickers for stock and crypto which are available on Yahoo Finance in the code/stream.py tickers variable which is a list and run the “build.sh” script in the repository. It will deploy the required infrastructure on AWS.
You can also make necessary changes for the desired infrastructure by changing the default values of parameters in the template.yaml.
Make sure to destroy the stack after usage using following command.
aws cloudformation delete-stack — stack-name Serverless-workflowReferences:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
