Added OpenTelemetry instrumentation to Ghost backend
This PR adds OpenTelemetry instrumentation to Ghost's backend code, which allows us to view traces similar to what we see in Sentry Performance locally.
It also refactors the ConnectionPoolInstrumentation (that was seeming to cause CPU utilization issues) —instead of creating so many logs and metrics and trying to pipe them into elastic, this class now simply creates spans in OpenTelemetry for the acquire and query stage of the query, so you can see both if there is contention in the pool and if the query itself is taking a long time. The new version of this instrumentation lives at ghost/core/core/shared/OpenTelemetryKnexTracing.js.
Note: there is likely some opportunity to combine this file with ghost/core/core/shared/SentryKnexTracingIntegration.js since there is a lot of overlap in functionality, but for now this seems to work well.
OpenTelemetry is enabled if NODE_ENV === 'development' or if it is explicitly enabled via config with opentelemetry:enabled.
It also adds a Jaeger container to Ghost's docker-compose file for viewing the traces. There's no setup required (beyond running yarn docker:reset to pickup the changes in the docker-compose file the first time — but this will also reset your DB so be careful). This will launch the Jaeger container, and you can view the UI to see the traces at http://localhost:16686/search.
Example trace in Jaeger:
Todo:
- [ ] Combine
OpenTelemetryKnexTracing.jsandSentryKnexTracingIntegration.jsif it makes sense - [ ] Add more attributes to the tags as needed
- [ ] Add tests for unhappy paths and make sure any potential errors are caught
- [ ] Check if CPU usage is affected by this
Codecov Report
Attention: Patch coverage is 94.21053% with 11 lines in your changes missing coverage. Please review.
Project coverage is 73.29%. Comparing base (
e5592aa) to head (6845204). Report is 21 commits behind head on main.
:exclamation: Current head 6845204 differs from pull request most recent head fce98b7
Please upload reports for the commit fce98b7 to get more accurate results.
| Files | Patch % | Lines |
|---|---|---|
| ghost/core/core/shared/OpenTelemetryKnexTracing.js | 92.48% | 10 Missing :warning: |
| ghost/core/core/shared/instrumentation.js | 97.72% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #20144 +/- ##
==========================================
- Coverage 73.40% 73.29% -0.11%
==========================================
Files 1269 1265 -4
Lines 74790 74314 -476
Branches 9971 9868 -103
==========================================
- Hits 54897 54467 -430
+ Misses 19079 19047 -32
+ Partials 814 800 -14
| Flag | Coverage Δ | |
|---|---|---|
| admin-tests | 42.07% <ø> (-0.82%) |
:arrow_down: |
| e2e-tests | 81.84% <94.21%> (+<0.01%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.