Sending prometheus's metrics which scrapped from each job to multiple namespace

[khanhntd]

Description of the issue

Currently, there is no way for us to send prometheus's metrics which scrapped from each job to multiple namespace (not include running multiple agents). Therefore, adding Metric Namespace in each Metric Declaration so the namespace in Metric Declaration can override the default namespace. By doing this, it would have the same benefits as namespace's benefits.

Tracking issues

https://github.com/aws/amazon-cloudwatch-agent/issues/377

Description of changes

Adding metric_namespace property in MetricDeclaration structure for customer to define an override namespace.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Test that I had done and failed:

• Scrapping prometheus's metric on MacOS: the toml translator does not accept prometheus configuration on that particular OS, thanks @ZhenyuTan-amz for pointing it out . Need to find more use case in the future for supporting Prometheus on Mac.
• Scrapping prometheus's metric on Linux On Premetheus; however, CWAgent on Linux needs a systemd unit before running it and I was not able to config the systemd unit with CWAgent. For long term needs to figure it out to better support with customer.

Test that I had done and succeeded:

• Step 1: Build the agent's docker image locally by using docker make dockerized-build (One thing to note is that the docker image does not have the default json configuration when running the image)
• Step 2: Set up a sample ECS Cluster. An example would be

ecs-cli up --capability-iam --size 1 \
--instance-type t3.medium \
--cluster $ECS_CLUSTER_NAME \
--region $AWS_REGION

• Step 3: Install a memcached workload on the sample ECS as a service.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_traffic/memcached/memcached-traffic-sample.yaml
MEMCACHED_ECS_TASK_ROLE_NAME=memcached-prometheus-demo-ecs-task-role-name
MEMCACHED_ECS_EXECUTION_ROLE_NAME=memcached-prometheus-demo-ecs-execution-role-name
MEMCACHED_ECS_NETWORK_MODE=host

aws cloudformation create-stack --stack-name Memcached-Prometheus-Demo-ECS-$ECS_CLUSTER_NAME-EC2-$MEMCACHED_ECS_NETWORK_MODE \
    --template-body file://memcached-traffic-sample.yaml \
    --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                 ParameterKey=ECSNetworkMode,ParameterValue=$MEMCACHED_ECS_NETWORK_MODE \
                 ParameterKey=TaskRoleName,ParameterValue=$MEMCACHED_ECS_TASK_ROLE_NAME \
                 ParameterKey=ExecutionRoleName,ParameterValue=$MEMCACHED_ECS_EXECUTION_ROLE_NAME \
    --capabilities CAPABILITY_NAMED_IAM \
    --region $AWS_REGION

• Step 4: Install CW Agent on the sample ECS as a service by using the following template with CMF. An example would be

ECS_NETWORK_MODEE=bridge
CREATE_IAM_ROLES=True
ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name

aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
    --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
    --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                 ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                 ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                 ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                 ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
    --capabilities CAPABILITY_NAMED_IAM \
    --region $AWS_REGION 

• Step 5: Check the result in CloudWatch console. The first picture is before having the override namespace and the second is after having it. Before: After:

Documents

• Setup scrapping prometheus's metrics with CW Agent: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-memcached-ecs.html#ContainerInsights-Prometheus-Setup-memcached-ecs-environment

[codecov-commenter]

Codecov Report

Merging #386 (995a867) into master (dd62a46) will increase coverage by 0.28%. The diff coverage is 65.38%.

@@            Coverage Diff             @@
##           master     #386      +/-   ##
==========================================
+ Coverage   57.27%   57.55%   +0.28%     
==========================================
  Files         370      369       -1     
  Lines       17351    17270      -81     
==========================================
+ Hits         9937     9939       +2     
+ Misses       6827     6745      -82     
+ Partials      587      586       -1     

Impacted Files	Coverage Δ
...gins/inputs/windows_event_log/wineventlog/utils.go	69.60% <ø> (ø)
plugins/processors/emfProcessor/emfProcessor.go	54.71% <0.00%> (-1.06%)	:arrow_down:
translator/config/schema.go	81.81% <ø> (ø)
plugins/inputs/logfile/tailersrc.go	88.23% <33.33%> (+1.47%)	:arrow_up:
...lugins/processors/emfProcessor/metricDefinition.go	89.09% <57.14%> (-4.91%)	:arrow_down:
plugins/inputs/logfile/tail/tail.go	33.92% <87.50%> (+0.76%)	:arrow_up:
plugins/inputs/logfile/logfile.go	64.59% <100.00%> (+0.26%)	:arrow_up:
translator/cmdutil/userutil_darwin.go

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update dd61eea...995a867. Read the comment docs.

[github-actions[bot]]

This PR was marked stale due to lack of activity.

[github-actions[bot]]

This PR was marked stale due to lack of activity.

[github-actions[bot]]

This PR was marked stale due to lack of activity.