Feature Request: customizable requeue time per monitor

[tthompson-figma]

👋 We're investigating a migration from Terraform-defined monitors to operator-based ones. We semi-officially support a pattern where, during incidents, responders temporarily tweak monitor thresholds from the UI and let later Terraform runs reset the tweak. However, from what I can tell, the monitor controller reconciles monitor configuration every minute by default. This prevents us from supporting this use case in a realistic way.

Would you be open to contribution of a new supported DatadogMonitorControllerOptions field that customizes the requeue interval per-monitor?

Strawman API:

type DatadogMonitorControllerOptions struct {
	// RequeueInterval specifies how many seconds to wait before reconciling the DatadogMonitor with the API monitor.
	RequeueInterval *int64 `json:requeueInterval,omitempty`

	// Other properties omitted for brevity
}

[xavixendit]

We also looking to migrate TF to CRD but we have thousands of monitors per cluster, a dozen clusters. What happens to the DD API at this point?

Whats the point on knowing the monitor status wether is Triggered or not? Is not that you will make "alerts" or "monitors" out of this.. in fact, im not sure wether you can even make alerts out of the status field of a CRD in a cluster with datadog at all..

The operator is handy, a good idea.. but seems over-engineered with behaviours like this, if you were in need to receive this status update.. u should consider integrations/webhooks or communication between the DD agent and the operator.

Can we get some flag to turn off this unnecessary overhead ? In theory projecting the creation, deletion and modification of the CRD into the monitor should be enough from a controller perspective.

[dd-octo-sts[bot]]

This issue has been automatically marked as stale because it has not had activity in the past 15 days.

It will be closed in 30 days if no further activity occurs. If this issue is still relevant, adding a comment will keep it open. Also, you can always reopen the issue if you missed the window.

Thank you for your contributions!