Trigger Lambda function every 15 minutes to process Kinesis data by device ID, calculate average, and store in DB

Question

I'm working on a serverless architecture using AWS Kinesis Data Streams and Lambda functions. I want to achieve the following:

1. Read data from the Kinesis stream in 15-minute blocks, starting from 00:00-00:15, 00:15-00:30, and so on throughout the day.

2. For each 15-minute block, trigger a Lambda function that processes the data for a specific device ID.

3. Within the Lambda function:

    i.Extract data relevant to the device ID.
    ii.Calculate the average value for that data over the 15-minute block.
   

4. Store the calculated average for the device ID and time block in a database

Challenges:

1. I'm unsure how to configure Kinesis or Lambda to achieve the periodic triggering based on 15-minute intervals.
2. There might be a more efficient way to handle data aggregation within the Lambda function.

I'm looking for guidance on the best approach to implement this functionality. Any help will be greatly appreciated.

Answer 1

In order to achieve periodic triggering of your lambda you can use Eventbridge. While configuring the eventbridge you can provide the cron expression for every 15 minutes trigger. The cron expression for the same is 0/15 * * * ? * You must also refer to the AWS documentation FYR https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-cron-expressions.html

Data aggregation within Lambda is majorly under your control as you can write your application and upload the executable in your lamda. You can use AWS Glue to do the ETL works. Glue can read and write data from multiple systems and databases including:

• Amazon S3

• Amazon DynamoDB

• Amazon Redshift

• Amazon Relational Database Service (Amazon RDS)

• Third-party JDBC-accessible databases

• MongoDB and Amazon DocumentDB (with MongoDB compatibility)

• Other marketplace connectors and Apache Spark plugins

AWS Glue for Spark can stream data from the following systems:

• Amazon Kinesis Data Streams

• Apache Kafka

FYR: https://docs.aws.amazon.com/glue/latest/dg/how-it-works.html