[Feature Request] Improved logging for troubleshooting (2024 Roadmap item)

[AndreaNassisi]

Community Note

• Please use a 👍 reaction to provide a +1/vote. This helps the community and maintainers prioritize this request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Is your feature request related to a problem? Please describe. The current Graph Explorer logging lacks of coverage and verbosity to enable troubleshooting issues. This is affecting the support operations enterprise deployments on Neptune Notebook and self-hosted instances.

Describe the solution you'd like Logging need to cover:

1. the system information in which the Graph Explorer is deployed
2. the log of the Node.js startup
3. the startup of the Graph Explorer processes on Node.js
4. The user connections
5. The user database schema loading
6. The user queries, not the resulting data
7. The user visualization settings

Additional context [TBD] Log lifecycle [TBD] integration with AWS services like CloudWatch

[kmcginnes]

We'll also want to make sure unhandled exceptions are properly logged, and investigate what is possible to log when the process is shutting down unexpectedly.

[kmcginnes]

Nearly all of the proposed solution has been covered. Here is a breakdown:

1. the system information in which the Graph Explorer is deployed

This was not done because we ran out of time and Graph Explorer is typically hosted within a Docker container running on a Neptune Notebook EC2 instance. Also, the specs of the Docker container are rarely important to diagnosing the issue.

It does come up occasionally, so I hope we can revisit this one in the future.

2. the log of the Node.js startup
3. the startup of the Graph Explorer processes on Node.js

The Express.js startup process is now logging out more information.

• Where the client side files are being hosted from
• The environment values being used by the server (not all, just the ones the node server is concerned with)
• Environment values are validated with helpful errors and the process refuses to start if the environment values are invalid
• The shutdown process attempts to be a graceful shutdown
• Unhandled exceptions are handled and logged

6. The user queries, not the resulting data

The database queries can now be logged to the server log stream. This is optional and off by default. If turned on, each database request that flows through the proxy server will be logged to the server's log stream (i.e. the terminal, Docker, or CloudWatch).

4. The user connections
5. The user database schema loading
7. The user visualization settings

These can be gathered by using the "Save Configuration" option in settings. This provides a full backup copy of the client configuration data in a JSON file. This can be provided for support and will include the connections, styles, any synced schemas, and any additional settings.