cloud_controller_ng
cloud_controller_ng copied to clipboard
Metrics may not be emitted for cleanup jobs
Issue
Currently it is possible for the periodic updater not to emit metrics for daily cleanup jobs.
Context
The periodic updater looks at the delayed_jobs table to populate the known jobs and their queue lengths in order to emit metrics.
The metrics are emitted by cloud controller on the api nodes, if no api node sees the temporary insertion of the row in the delayed_jobs table before cc-worker picks it up then the job will not have a metric emitted.
It seems like for similar behaviour with the local worker a change was made to ensure that metrics would always be emitted: 0fddfd413e89f5c14fe9f71241628e7aeadb68c7
Steps to Reproduce
- Deploy Cloud Foundry with a small number of
apiinstances - Wait for a cleanup job to run
- Occasionally observe that job metrics are not emitted
Expected result
Job queue metrics are emitted.
Current result
Job queue metrics are sometimes not emitted.