Feature request: Support for MSK in Batch Processing

[einarjohnson]

Use case

Add support for batch processing partial failures when using a MSK/Kafka event source map. This is currently supported for Amazon SQS, Amazon Kinesis Data Streams, and Amazon DynamoDB Streams and it would be great to have this feature for Kafka also.

Solution/User Experience

Having the same interface, or something similar, to what we already have for SQS.

Alternative solutions

No response

Acknowledgment

• [X] This feature request meets Powertools for AWS Lambda (Python) Tenets
• [X] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[boring-cyborg[bot]]

Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hello @einarjohnson! In fact this is a nice feature, we see a growing number of customers asking for Batch Processing support on EventSource other than SQS, DynamoDB, and Kinesis.

The main benefit of our batch processing utility is the ability to handle partial failures and return only failed messages to the source, and currently, Lambda does not support PartialFailure when using MSK.

I will keep this issue open to allow the community to give it a thumbs up and we will pass this need from our customers to the Lambda team. I don't have an estimated date of when I can update this issue with any news, but as soon as I have news I'll come back here to update.

Thank you so much for bringing these ideas to Powertools, this is priceless.

[einarjohnson]

Thanks for the quick response @leandrodamascena . Given the nature of Kafka/consumer offsets this feature might be tricky to implement. But it would be great to have at some stage, very exited to see if this gains some momentum with the AWS lambda team and gets implemented in some way.

[leandrodamascena]

Hello everyone! We don't know if BatchPartialFailure support will be integrated into MSK, so I'm closing this issue as unplanned for now.

However, we are keeping an eye on all releases from the Lambda/Poller team and if support is added, we will reopen this issue.

Thank you.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.