WriteUnhandledExceptionToLog in LambdaBootstrap seems to break structured logging setup

[LarsAndreasEk]

Description

In 80c440c2f8db276bf71e946ed36d012e9aed4b2a, the logging behaviour for unhandled exceptions was changed. Exceptions are ToString()-ed and written to Console.Error.

This seems to "break" existing setups where structured logging is in place (ie Serilog). Logs are written twice, once by the application's own structured logging setup, then again to Console.Error using a non-structured approach.

We seem to be left with these options:

1. Live with exceptions being logged twice, in two different ways.
2. Skip the structured logging for unhandled exceptions, and live with the unstructured logging for unhandled exceptions.
3. Downgrade to Amazon.Lambda.RuntimeSupport 1.5

If you have any other suggestions, please let me know!

Reproduction Steps

1. Create a dotnet lambda (I use a custom runtime, but I don't think that changes anything)
2. Setup Serilog or another structured logging library, configured to write to stdout
3. Have the lambda listen to sqs messages, and deliberately throw an exception
4. Make sure to have a try/catch around message processing that logs thrown exceptions using Serilog.
5. Rethrow the exception to let Lambda retry the SQS message.
6. Inspect the logs (you will now see a json string from Serilog AND numerous lines from the WriteUnhandledExceptionToLog method)
7. Ship the logs to Datadog, or similar, that tries to understand the not-so-structured logs :)

Resolution

It would be nice if it was possible to disable the logging that the LambdaBootstrap does when an unhandled exception is thrown. Or perhaps have a way to provide a delegate that does the logging itself, so that we could have the LambdaBootstrap itself use structure logging.

---

Might not be a :bug: bug-report, but causes problems for us at least :)

[ashishdhingra]

Most likely a feature request rather than a bug.

[martincostello]

Might be related to #1000.

[bruce-lindsay]

@LarsAndreasEk this was an issue for me as well. As a workaround Console.SetError from the handler constructor with a custom implementation of a TextWriter which does nothing when WriteLine(object o) is called suppresses the output. I'm leaving this in case it helps anyone else. This worked for me because I don't use Console.Error from application code.

The new behavior should provide an opt-out for customers who are already using tools like Serilog.

[LarsAndreasEk]

Thanks @bruce-lindsay! I put another workaround in place: Catch exceptions in our own code, log, then throw a custom exception that has a ToString implementation that returns null.