ECS Service Discovery without Writing EMF Logs

[GreasyAvocado]

Hi,

I have an ECS cluster and a self-hosted Prometheus. I'd like to use CW agent in order to perform service discovery on my ECS services, and then have Prometheus scrape the metrics of the discovered containers.

I implemented it according to this doc: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure-ECS.html

The discovery process works well, and I get a targets file under /tmp/cwagent_ecs_auto_sd.yaml, which Prometheus can then use. The problem is that the agent also writes EMF logs to CloudWatch Logs (/aws/ecs/containerinsights/${cluster_name}/prometheus), which I don't need.

Is it possible to somehow keep the ecs_service_discovery flag in the agent's config, while disabling the agent from sending EMF logs?

I've tried using this config, but EMF logs were still being sent this way:

  "logs": {
    "metrics_collected": {
      "prometheus": {
        "prometheus_config_path": "env:PROMETHEUS_CONFIG_CONTENT",
        "ecs_service_discovery": {
          "sd_frequency": "1m",
          "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml",
          "docker_label": {
            "sd_port_label": "ECS_PROMETHEUS_EXPORTER_PORT"
          }
        },
        "emf_processor": {
          "metric_declaration": [
            {
              "dimensions": [["Namespace"]],
              "source_labels": ["Namespace"],
              "label_matcher": "dont_send_metrics",
              "metric_selectors": ["dont_send_metrics"]
            }
          ]
        }
      }
    },
    "force_flush_interval": 5
  }
}

Any sugestions?

Thanks!

[github-actions[bot]]

This issue was marked stale due to lack of activity.

[pingleig]

If you use ADOT, you can use the https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/observer/ecsobserver to generate the target file . You can set the exporter to noop or use its prometheus related exporters (remote write etc.), it won't generate the EMF log.

I think https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/observer/ecstaskobserver won't generate the sd file for prometheus but I am not the author of that plugin so I don't know much about it.

[GreasyAvocado]

That's exactly what I ended up doing. Works great!

[github-actions[bot]]

This issue was marked stale due to lack of activity.

[github-actions[bot]]

Closing this because it has stalled. Feel free to reopen if this issue is still relevant, or to ping the collaborator who labeled it stalled if you have any questions.