onPremise default profile configuration broken

[julian-meter]

Describe the bug

Commit 6119858864c317ff26f41f576c169148d1250837 changes the config.ModeOnPrem string from onPremise to onPrem, however several programs receive this string as a command-line argument (e.g. config-downloader) from amazon-cloudwatch-agent-ctl and fail to recognize the region or credentials as a consequence. For example, config-downloader takes the string onPremise as its mode flag, which used to compare equal with config.ModeOnPrem, but no longer does after this change, which means GetCredentials no longer sets the credential profile to DEFAULT_PROFILE.

Steps to reproduce

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -c file:cloudwatch.json -s where common-config.toml doesn't contain a credentials block.

What did you expect to see?

I! SDKRegionWithCredsMap region:  us-west-2

or

****** processing amazon-cloudwatch-agent ******
Got Home directory: /root I! Set home dir Linux: /root I! SDKRegionWithCredsMap region: us-west-2 Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_cloudwatch.json.tmp

and so on, which is the output with 1.247354 or with shared credentials explicitly set.

What did you see instead?

Unable to determine aws-region.

or sometimes

****** processing amazon-cloudwatch-agent ******
2022/09/01 19:21:25 E! Please make sure the credentials and region set correctly on your hosts.

depending on the state of existing configuration on the host.

What version did you use?

v1.247355.0. Confirmed that v1.247354.0 works fine.

Environment

Ubuntu 18.04.6 LTS (Bionic Beaver)

Additional context

I recommend either reverting the change, or a patch like this: (note that anything that specifies mode arguments to the programs that consume this string must be adjusted)

diff --git a/packaging/dependencies/amazon-cloudwatch-agent-ctl b/packaging/dependencies/amazon-cloudwatch-agent-ctl
index 5c5fbcf..1e9c722 100755
--- a/packaging/dependencies/amazon-cloudwatch-agent-ctl
+++ b/packaging/dependencies/amazon-cloudwatch-agent-ctl
@@ -514,7 +514,7 @@ main() {
 
      case "${mode}" in
      ec2) ;;
-     onPremise) ;;
+     onPremise) mode=onPrem ;;
      auto) ;;
      *)
           echo "Invalid mode: ${mode} ${UsageString}" >&2

[antoine-sachet]

We're badly impacted by this. How can I install the previous version? AWS only provide links to the latest version which is broken for onPremise use unless we change the config everywhere.

[mark-poscablo]

One of our released projects at Varian Medical Systems is also badly impacted by this (its proprietary so it's not on GitHub). Please fix this bug - we'd very much appreciate it. Thanks!

[SaxyPandaBear]

Sorry for the radio silence. We're discussing this internally on the best approach to mitigate this. @antoine-sachet right now, we don't explicitly publish how to install older versions of the agent. You should be able to reach out to AWS support in order to get help on downgrading.

[stankolubomir]

Is it possible to get the status of this. We are also heavily impacted by this. We can not spawn new servers because they depends on aws metrics and aletring system. Please fix this bug ASAP. Thank you.

[adam-mateen]

@stankolubomir and @antoine-sachet It looks like @ymtaye is fixing the issue now. Until then, you should be able to download the previous version with this link: https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/1.247354.0b251981/amazon-cloudwatch-agent.rpm

(amazon_linux / AMD64)

[stankolubomir]

@adam-mateen How is it possible to get this kind of link? Just asking if something similar happens in the future. We prefer to lock to the specific version of the package instead of risking issues with "latest" version.

[adam-mateen]

Short answer is - you cannot.

AWS only supports the latest release of CloudWatch Agent. So please treat this link as temporary.

[valof]

Can somebody please clarify this for me - is "latest" have anything to do with the release tags here? https://github.com/aws/amazon-cloudwatch-agent/releases The commit 6119858 is NOT in the latest release tag v1.247354.0 SO... when we download the "latest" are we download daily build of some kind? (for the record, I am 100% OK with it - I just need to know what "latest" mean) Also, if it is a daily build, I suspect the same commit may have something to do with the use case 11005207991 , which I opened recently.

[valof]

@SaxyPandaBear Can you take a look? 👆🏻 I can help reproducing the case 11005207991 if need be.

[SaxyPandaBear]

@valof if you are communicating through AWS Support already, that's the best communication channel to go through.

[mark-poscablo]

@SaxyPandaBear @adam-mateen Is there an estimate of when an official release (not a nightly build) that contains #598 / 7c4cdbf is coming out?

[jupe]

We're also badly impacted by this. Please provide fix for this asap.

[ahmgithubahm]

If it helps anyone, here's a link to the working Windows version: https://amazoncloudwatch-agent.s3.amazonaws.com/windows/amd64/1.247354.0b251981/amazon-cloudwatch-agent.msi

[TreeKat71]

Thanks @ahmgithubahm , it works for me. Save my day

[adam-mateen]

Sorry for the long delay. The was a delay in creating an official release due to the November holidays as well as other "restricted" / "blocked" days.

v1.247357 release in in progress. The global S3 links will return this version, however some of the regional S3 links will take a week or so to get updated.

Please verify this latest version resolves the issue and re open an issue if not.

Thanks.

[mark-poscablo]

This has fixed our issue at Varian. Thank you all for the fix!