Ghost icon indicating copy to clipboard operation
Ghost copied to clipboard

Added OpenTelemetry instrumentation to Ghost backend

Open cmraible opened this issue 1 year ago • 1 comments

This PR adds OpenTelemetry instrumentation to Ghost's backend code, which allows us to view traces similar to what we see in Sentry Performance locally.

It also refactors the ConnectionPoolInstrumentation (that was seeming to cause CPU utilization issues) —instead of creating so many logs and metrics and trying to pipe them into elastic, this class now simply creates spans in OpenTelemetry for the acquire and query stage of the query, so you can see both if there is contention in the pool and if the query itself is taking a long time. The new version of this instrumentation lives at ghost/core/core/shared/OpenTelemetryKnexTracing.js.

Note: there is likely some opportunity to combine this file with ghost/core/core/shared/SentryKnexTracingIntegration.js since there is a lot of overlap in functionality, but for now this seems to work well.

OpenTelemetry is enabled if NODE_ENV === 'development' or if it is explicitly enabled via config with opentelemetry:enabled.

It also adds a Jaeger container to Ghost's docker-compose file for viewing the traces. There's no setup required (beyond running yarn docker:reset to pickup the changes in the docker-compose file the first time — but this will also reset your DB so be careful). This will launch the Jaeger container, and you can view the UI to see the traces at http://localhost:16686/search.

Example trace in Jaeger: CleanShot 2024-05-22 at 07 00 26@2x

Todo:

  • [ ] Combine OpenTelemetryKnexTracing.js and SentryKnexTracingIntegration.js if it makes sense
  • [ ] Add more attributes to the tags as needed
  • [ ] Add tests for unhappy paths and make sure any potential errors are caught
  • [ ] Check if CPU usage is affected by this

cmraible avatar May 07 '24 00:05 cmraible

Codecov Report

Attention: Patch coverage is 94.21053% with 11 lines in your changes missing coverage. Please review.

Project coverage is 73.29%. Comparing base (e5592aa) to head (6845204). Report is 21 commits behind head on main.

:exclamation: Current head 6845204 differs from pull request most recent head fce98b7

Please upload reports for the commit fce98b7 to get more accurate results.

Files Patch % Lines
ghost/core/core/shared/OpenTelemetryKnexTracing.js 92.48% 10 Missing :warning:
ghost/core/core/shared/instrumentation.js 97.72% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #20144      +/-   ##
==========================================
- Coverage   73.40%   73.29%   -0.11%     
==========================================
  Files        1269     1265       -4     
  Lines       74790    74314     -476     
  Branches     9971     9868     -103     
==========================================
- Hits        54897    54467     -430     
+ Misses      19079    19047      -32     
+ Partials      814      800      -14     
Flag Coverage Δ
admin-tests 42.07% <ø> (-0.82%) :arrow_down:
e2e-tests 81.84% <94.21%> (+<0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar May 07 '24 19:05 codecov[bot]