THREESCALE-8772 - AWS secret format with STS

[valerymo]

what

Jira: https://issues.redhat.com/browse/THREESCALE-8772 AWS secret format with STS

STS object will be added to ApiManager CR to recognize if it's STS cluster or IAM

• ApiManager CR example:

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata: 
  name: apimanager-sample
  namespace: 3scale-test
spec: 
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
        sts:
          enabled: true
          audience: openshift
  wildcardDomain:<wildcardDomain>

• STS s3 secret example

kind: Secret
apiVersion: v1
metadata:
  name: s3-credentials
  namespace: 3scale
data:
  AWS_ROLE_ARN: arn:aws:iam::<ID>:role/<ROLE>
  AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/openshift/serviceaccount/token
  AWS_BUCKET: <bucket_name>
  AWS_REGION: <region>
type: Opaque 

Validation

TESTS scenarios

• Test 1 - CR: NON STS, Secret: Non STS expected: 3scale operator installed test of image upload (in DevPortal/Content/Images) - Successfull
• Test 2 - CR: NON STS, Secret: STS expected: error - secret field 'AWS_ACCESS_KEY_ID' is required in secret 's3-credentials'
• Test 3 - CR: STS, Secret: STS , sts.enabled: false, sts.audience missing; expected: error - secret field 'AWS_ACCESS_KEY_ID' is required in secret 's3-credentials'
• Test 4 - CR: STS, Secret: STS , sts.enabled: false, sts.audience: "123" expected: error "secret field 'AWS_ACCESS_KEY_ID' is required in secret 's3-credentials'
• Test 5 - CR: STS, Secret: STS , sts.enabled: true, sts.audience missing expected: 3scale operator installed, audience (in pod system-app): openshift, test of image upload - Successfull
• Test 6 - CR: STS, Secret: STS , sts.enabled: st.audience "123" expected: 3scale operator installed, audience(in pod system-app): "123", test upload of image - Failed, Internal error notification appears in 3scale-operator. - Developer Portal/Content - Sections templates are missing (https://3scale-admin.xxxxxx.devshift.org/p/admin/cms/templates) - Images section and other missing. If create new section and load file - Internal error.

Install 3scale operator

$ cd 3scale-operator
$ make install

Notes: make install required to create/recreate ApiManager CRD. The following commands to be done before each test. It's better to delete 3scale-test before each test, but in some cases, it's enough to delete the secret and CR, and stop running operator

$ export NAMESPACE=3scale-test
$ oc new-project $NAMESPACE
$ oc project $NAMESPACE
$ make download
$ make run

Secret and CR to be installed on separate terminal window.

Secrets & CRs for test

• Non STS Secret

oc apply -f - <<EOF
---
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: XXX=
  AWS_SECRET_ACCESS_KEY: XXX=
  AWS_BUCKET: XXX=
  AWS_REGION: dXMtZWFzdC0y
type: Opaque
EOF

• STS Secret

oc apply -f - <<EOF
---
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ROLE_ARN: XXX=
  AWS_WEB_IDENTITY_TOKEN_FILE: XXX=
  AWS_BUCKET: XXX=
  AWS_REGION: dXMtZWFzdC0y
type: Opaque
EOF

• Non STS CR

oc apply -f - <<EOF
---
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata: 
  name: apimanager-sample
  namespace: 3scale-test
spec: 
  system: 
    image: 'quay.io/3scale/porta:fix-cms-s3-hostname'
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: apps.vmogilev-rosa.upri.s1.devshift.org
EOF

• STS CR

oc apply -f - <<EOF
---
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata: 
  name: apimanager-sample
  namespace: 3scale-test
spec: 
  system: 
    image: 'quay.io/3scale/porta:fix-cms-s3-hostname'
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
        sts:
          enabled: true
          audience: openshift
  wildcardDomain: apps.vmogilev-rosa.upri.s1.devshift.org
EOF

Validation done: All tests passed as expected

[openshift-ci[bot]]

Hi @valerymo. Thanks for your PR.

I'm waiting for a 3scale member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[austincunningham]

/ok-to-test

[austincunningham]

@valerymo have you checked that the system-sidekiq pods environment variables have been applied when you changed to the STS formatted secret ?

[eguzki]

The tests are checking the existence or omission of the fields. I think that one e2e manual test would be necessary for this particular case. It would be nice to test that 3scale works with AWS secret with STS format. It should be simple (maybe not easy). The operator needs to be run on a STS supported openshift instance and then upload one asset for the developer portal (image or whatever). Check that the asset is there in the developer portal.

[valerymo]

@valerymo have you checked that the system-sidekiq pods environment variables have been applied when you changed to the STS formatted secret ?

Hi @austincunningham , I'm working to add volumes now (not pushed yet), and I see env vars in system-sidekiq , I added it. Will submit EOD

oc describe pod system-sidekiq-1-wclkf |grep aws-token ...

  AWS_ROLE_ARN:                                  <set to the key 'AWS_ROLE_ARN' in secret 'aws-token'>                      Optional: false
  AWS_WEB_IDENTITY_TOKEN_FILE:                   <set to the key 'AWS_WEB_IDENTITY_TOKEN_FILE' in secret 'aws-token'>       Optional: false

. . . /var/run/secrets/eks.amazonaws.com/serviceaccount/ from aws-token (ro)

[valerymo]

• comments addressed.
• After update for setFileStorageOptions method now should return an error. - issue in Unit test TestGetSystemOptionsProvider/WithS3 (checked in IDE, not clear meanwhile)
• Tested 3scale-operator on sts cluster without Rhoam. Looks correct. CC @eguzki , @laurafitzgerald

[laurafitzgerald]

Suggested Verification Steps:

Setup

Deploy this copy of 3scale-operator to a ROSA STS enabled cluster. Seed the secret with keys AWS_BUCKET, AWS_REGION, AWS_ROLE_ARN , AWS_WEB_IDENTITY_TOKEN_FILE with name s3-credentials Create an api-manager cr named with the value spec.system.fileStorage.siimpleStorageService to have key:value entry sts: true You'll may need to redeploy the system app components, can it be done by deleting the system-app deployment config and letting 3sclae-operator recreate it? or will it update it by default?

Verify the following:

For pods system-app system-app-X-hook-pre system-sidekiq Check that Env vars AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are present Check that Env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are NOT present Verify that Env vars AWS_BUCKET and AWS_REGION are also present Volume Mount - mountPath: /var/run/secrets/openshift/serviceaccount/token name: s3-credentials readOnly: true is present Volume is also present

[laurafitzgerald]

Verifying the following on @valerymo cluster

Aipmanager cr has the value (variable name needs a cleanup but this is just cosmetic)

For each pods system-app system-app-X-hook-pre system-sidekiq the below is verified, please see screenshots Check that Env vars AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are present Check that Env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are NOT present Verify that Env vars AWS_BUCKET and AWS_REGION are also present Volume Mount - mountPath: /var/run/secrets/openshift/serviceaccount/token name: s3-credentials readOnly: true is present Volume is also present

system-app pod

Env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are NOT present

system-app-X-hook-pre

Env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are NOT present

system-sidekiq

Unless I am missing any other configuration in my verification, it appears that all configuration is done correctly.

[laurafitzgerald]

Verifying the following on @valerymo cluster

Aipmanager cr doesn't have the value so sts should be false

For each pods system-app system-app-X-hook-pre system-sidekiq the below is verified, please see screenshots Check that Env vars AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are NOT present Check that Env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY ARE present Verify that Env vars AWS_BUCKET and AWS_REGION are also present Verify that mounts are not present: Volume Mount - mountPath: /var/run/secrets/openshift/serviceaccount/token name: s3-credentials readOnly: true is present Volume is also present

### system-app pod

system-app-X-hook-pre

### system-sidekiq

It works as expected when sts is enabled and the secret is configured with IAM credentials.

[laurafitzgerald]

Verification Invalid config of secret in case of sts true produces error.

confirm that error is produced by the operator

[laurafitzgerald]

## Re-verifing after recent changes

STS enabled true in apimanager cr.

system-app envs and volumes present and correct as above. system-app-X-hook-pre envs and volumes present and correct as above. syttem-sidekiq envs and volumes are present and correct as above

[laurafitzgerald]

## Re-verifing after recent changes

### STS not present in apimanager cr.

system-app envs and volumes not present and correct as above. system-app-X-hook-pre envs and volumes not present and correct as above. system-sidekiq envs and volumes not present and correct as above.

[laurafitzgerald]

Confirming that in sts enabled true if AWS_WEB_IDENTITY_TOKEN_FILE is missing is reporting an error.

[eguzki]

I suggest you push commits with small contributions. And only when it is ready to merge, it is your call if you want to squash them in a single commit or instead keep all the commits. But for reviewing, it makes it easier to have small commits

[valerymo]

I suggest you push commits with small contributions. And only when it is ready to merge, it is your call if you want to squash them in a single commit or instead keep all the commits. But for reviewing, it makes it easier to have small commits

ok, I will not use commit --amend

[valerymo]

Docs updated. STS section was moved from openapi-user-guide.md (I wrote there by mistake before) to operator-user-guide.md.

[valerymo]

/test test-e2e

[qlty-cloud-legacy[bot]]

Code Climate has analyzed commit 7f7abc1b and detected 15 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	7
Duplication	2
Style	6

View more on Code Climate.

[eguzki]

/test test-e2e

[laurafitzgerald]

Confirming that diff before I pushed the sqaush produce 0 differences. @valerymo can you also confirm from your local copy pre squash.