aws-powertools
Feature request: Better failures handling while using both BatchProcessing and LargeMessageHandling
Is your feature request related to a problem? Please describe.
While using both BatchProcessing and LargeMessageHandling on a lambda, there could be a case when the lambda failed to download the large payload from S3, hence it would fail the whole batch before it could even reach the batch processor. The result is that we have to reprocess/redownload the whole batch, even though most of the messages in the batch could have been processed in the previous try.
Describe the solution you'd like The failures when downloading payload are handled individually as it were an exception inside the batch processor.
Additional context
My use case is that I would like to batch process large messages. I would prefer to process any messages downloaded successfully without relying on other messages' downloading status. Only the failed-to-load messages should be retried.
What I am thinking is to give possibility to users to handler specific message exceptions while enriching in some way.
So if during enriching message from S3, if we get an exception, we can probably accept an exception handler class on LargeMessage annotation which users can implement on how they want to deal with the exception.
That way, they can decide themselves, if they would like to throw exception to fail entire batch or would liketo silently deal with the exception and let enriching of other message continue.
Something like:
public class SqsLambdaHandler implements RequestHandler<SQSEvent, String> {
@Override
@SqsLargeMessage(enrichFailureHandler = ExceptionHandler.class)
public String handleRequest(SQSEvent sqsEvent, Context context) {
return sqsEvent.getRecords().toString();
}
class ExceptionHandler implements EnrichFailureHandler {
@Override
public void handleFailure(SQSMessage message,
FailedProcessingLargePayloadException e) {
// Handle exception specific to a message
}
}
}
So if during enriching message from S3, if we get an exception, we can probably accept an exception handler class on LargeMessage annotation which users can implement on how they want to deal with the exception.
Yeah, I agree. This makes more sense and provides more flexibility.
