[Enhancement] Vendor pull from terraform private registries

[Amos-85]

Describe the Feature

This feature is an enhancement for Atmos Components to be able to fetch terraform modules from private registries based on https.

Expected Behavior

Ability to fetch terraform modules from private terraform registries

Use Case

Many companies use private registries and repositories to:

• Decoupled from vendor servers.
• Fasten their pipeline builds.
• Avoid breaking routine work when problems appear with vendor servers.
• Improve security.
• etc...

E.g:

• Docker
• Apt
• Rpm
• Npm
• Maven
• Helm
• Terraform
• etc...

Describe Ideal Solution

Option 1:

After running terraform login <some terraform private registry>, credentials file being created under ~/.terraform.d/credentials.tfrc.json . If Atmos can look for this path on component initialization, it might solve the problem for any terraform registry use case as long as it work with native terraform registry authentication.

Reference: ==> Terraform Credentials Storage

Option 2 (might be the preferred):

Atmos will fetch the following environment variable in case of private registries TF_TOKEN_cloudposse_jfrog_io=<terraform-private-registry-token> , terraform translate the uri to cloudposse.jfrog.io .

Reference: ==> Environment Variables Credentials

Alternatives Considered

No response

Additional Context

No response

[jamengual]

Hi @Amos-85 I'm using vendor pull right now and I'm trying to understand your usecase.

you are talking about the module instantiation source or the URL on the component.yaml that is read by atmos when you run atmos vendor pull?

[Amos-85]

Hi @jamengual , The use case is actually similar to terragrunt tfr protocol .

Currently I'm trying to run atmos vendor pull -c aws/vpc after setup ComponenetVendorConfig as below :

apiVersion: atmos/v1
kind: ComponentVendorConfig
metadata:
  name: vpc-vendor-config
  description: Source and mixins config for vendoring of 'vpc' component
spec:
  source:
    uri: "https://<company-domain>.jfrog.io/artifactory/<terraform-remote-repository>/terraform-aws-modules/terraform-aws-vpc/{{ .Version }}.zip"
    version: v4.0.1
    included_paths:
    - "**/*.tf"
    - "**/*.tfvars"
    - "**/*.md"

but it's failing to fetch the module with error bad response code: 401 because of missing terraform private registry api token

It's fetching the module as expected if I'll use the uri with username:password as following:

uri: "https://<username>:<password>@<company-domain>.jfrog.io/artifactory/<terraform-remote-repository>/terraform-aws-modules/terraform-aws-vpc/{{ .Version }}.zip"

but it's not secure way to use credentials in configuration files.