aws-samples
New serverless pattern - sqs-lambda-ddb-cdk-ts
Issue #, if available: #2031
Description of changes: The purpose of this pattern is to address write throttling issues of Provisioned Capacity DynamoDB Tables by offloading the requests to SQS to handle writes asynchronously. This pattern is deployed through an AWS Cloud Development Kit (CDK) app which provisions DynamoDB tables, SQS queues and DLQs, Lambda functions, and Cloudwatch alarms. Test scripts have been provided for testing.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Hi, it looks like you have a subfolder sqs-lambda-dynamodb-cloudwatch-cdk-ts with another pattern? Is that intentional?
I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?
Hi, it looks like you have a subfolder sqs-lambda-dynamodb-cloudwatch-cdk-ts with another pattern? Is that intentional?
Hi Ben, addressed the outdated pattern folder (this was from a previous submission where the pattern also included a cloudwatch alarm)
I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?
The pattern is meant to show how to offload dynamodb writes using sqs + lambda per table pipelines to avoid ddb table throttling. This is done in collaboration with gulid@ and their pattern for the same architecture but using SAM instead (under the sqs-lambda-ddb-sam-ts subfolder
I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?
The pattern is meant to show how to offload dynamodb writes using sqs + lambda per table pipelines to avoid ddb table throttling. This is done in collaboration with gulid@ and their pattern for the same architecture but using SAM instead (under the
sqs-lambda-ddb-sam-tssubfolder
Hi, why would a customer not use Application Auto Scaling to adjust the amount of Provisioned Concurrency instead? https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html#managing-provisioned-concurency
I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?
The pattern is meant to show how to offload dynamodb writes using sqs + lambda per table pipelines to avoid ddb table throttling. This is done in collaboration with gulid@ and their pattern for the same architecture but using SAM instead (under the sqs-lambda-ddb-sam-ts subfolder
Hi, why would a customer not use Application Auto Scaling to adjust the amount of Provisioned Concurrency instead? https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html#managing-provisioned-concurency
The Provisioned Concurrency value would only affect the number of available execution environments for the Lambda handler, not limit the maximum amount. With this pattern, we are trying to offload the DynamoDB writes to lambda to allow the supposed application side to write as frequently as possible, rather than having to time requests in line with the DynamoDB WCUs on the application side. For this purpose, the Reserved Concurrency value is more useful since it represents the max number of concurrent requests allowed, which limits how fast the Lambda handler is allowed to handle the requests to keep the table from throttling as much as possible and avoid redriving requests.
Hi @astan54321 , thanks for your reply. I understand the use-case better now. In this case, I think it would be best to integrate maximum concurrency on the SQS side as well: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/
Hi @astan54321 , thanks for your reply. I understand the use-case better now. In this case, I think it would be best to integrate maximum concurrency on the SQS side as well: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/
Tanks for sharing this @bfreiberg! This helps a lot. Will go ahead and integrate this and update the PR.
Hi Ben, yes, we are still working on making edits to the README and making sure the testing stop makes sense. Should have an updated commit early next week.
Hi Ben and Eric, yes just pushed updates with updated functionality to use SQS max concurrency instead of Lambda's reserved concurrency, which fits the goal of the pattern better. Changes were required to the README by another team member and I during the past few weeks which is what took so long with this (along with changing schedules). Let me now if anhy final changes need to be made if this is to be merged into the base ServerLess land repo. Thanks.
Looks good, thanks for your contribution. A Developer Advocate will merge the PR and publish the changes to serverlessland.com.