aws
Add extra-index-url support for CodeArtifact login with --tool pip
Is your feature request related to a problem? Please describe.
Running aws codeartifact login --tool pip ... overrides the default index url (pypi.org), which becomes an issue if CodeArtifact becomes out of sync (a separate issue)
Describe the solution you'd like
Add an --extra-index-url flag to put the CodeArtifact URL in (only necessary for --tool pip)
Should be as easy as changing global.index-url to global.extra-index-url if the flag is passed: https://github.com/aws/aws-cli/blob/develop/awscli/customizations/codeartifact/login.py#L157
Describe alternatives you've considered Currently setting the url "manually":
CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token --domain ... --domain-owner ... --query authorizationToken --output text)
pip config set global.extra-index-url https://aws:$CODEARTIFACT_TOKEN@...-....d.codeartifact.us-east-1.amazonaws.com/pypi/.../simple/
Hi @nayaverdier ,
Thanks for the feature request. I'm going to bring this to the attention of the CodeArtifact team.
Hi @kdaily,
Do you know if the CodeArtifact team has been able to take a look at the PR for this issue?
This was a major issue for us. Too many chicken&egg issues with tokens auto-expiring after 12 hours. Machines (especially ubuntu 18) in order to simply use the code artifact autologin often need as their first step to upgrade pip, which requires a newer pip than the latest system default. But they can't do that if they have to login to codeartifact first, which they will be forced to do if they use the default autologin command from the CLI.
As a result, we don't use the auto login feature, and manually grab tokens. We then use sed to update our local requirements.txt files and with the combination of a git filter, are able to successfully ignore the first line changes with --extra-index-url=...CODEARTIFACT_AUTH_LOGIN.... So our requirements.txt files store the tokens and they never get checked into source control. Then we can leave the user/system default pypi registry unchanged, grab the necessary upgrade prerequisites, and seamlessly pull down the private artifacts from codeartifact as needed.
Definitely not a simple solution to set up.
Chiming in from CodeArtifact:
Thanks for the PR and bringing this to our attention. We will be looking at this soon
I'm not sure this will help anyone else, but I found a sort of workaround for my needs, and particularly on Ubuntu 18.04. It involves installing a newer version of python, by way of compiling from source. That way, you get a newer version of pip, too. I had luck with both 3.8 and 3.9 on different machines. In the following example code, I used 3.9.
The following commands should get you through the process. Beware the the compilation process is somewhat lengthy. Also, I got some of this from https://tecadmin.net/how-to-install-python-3-9-on-ubuntu-18-04/ and some was trial and error. Beware that this may take some time since, again, you are compiling from source.
# install prereqs
sudo apt install wget build-essential checkinstall
sudo apt install libreadline-gplv2-dev libncursesw5-dev libssl-dev \
libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev
# get and install python 3.9
cd /opt
sudo wget https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz
sudo tar xzf Python-3.9.6.tgz
cd Python-3.9.6
sudo ./configure --enable-optimizations # this part takes a while
sudo make altinstall # as noted in the page from the link above, **it is important not to use make install**
# remove the unused files
sudo rm -f /opt/Python-3.9.6.tgz
sudo rm -rf /opt/Python-3.9.6
# optional - get venv for 3.9
sudo apt update
sudo apt install python3.9-venv
You can now use python3.9 in place of python3
In my specific case, I am using a virtual environment on a host computer to facilitate linting and code following in vscode, so I had to then do something like this:
python3.9 -m venv ~/envs/venv
. ~/envs/venv/bin/activate
aws codeartifact login --tool pip --repository [repo name] --domain [domain name] --domain-owner [domain owner id]
# make sure you install any libraries which support any pip packages here
pip install -r requirements.txt # or wherever you have it
+1
This is what we do. It works, but a bit clunky...
DOMAIN=XXX
REPO=XXX
AWS_ACCOUNT=XXX
REGION=XXX
export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain ${DOMAIN} --domain-owner ${AWS_ACCOUNT} --query authorizationToken --output text`
export CODE_ARTIFACT_URL="https://aws:$CODEARTIFACT_AUTH_TOKEN@${DOMAIN}-${AWS_ACCOUNT}.d.codeartifact.${REGION}.amazonaws.com/pypi/${REPO}/simple/"
export PIP_EXTRA_INDEX_URL="${CODE_ARTIFACT_URL}"
+1
It would be nice to add extra-index-url support for auto-login, for example via a --extra-index-url flag parameter.
Anyway, you can do it manually by retrieving the token and manually modify the pip .ini config file.
For my use case, I had to perform such operation in a Python script, I leave here a code snippet:
# init boto3 client
boto3_client = boto3.client('codeartifact',
region_name=REGION,
aws_access_key_id=AWS_ACCES_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCES_KEY)
# get auth-token manually
response = boto3_client.get_authorization_token(
domain=DOMAIN,
domainOwner=OWNER,
durationSeconds=43200
)
token = response['authorizationToken']
# build the extra-index-url entry
extra_index_url = f'https://aws:{token}@{DOMAIN}-{OWNER}.d.codeartifact.{REGION}.amazonaws.com/pypi/{REPOSITORY}/simple/'
def run_bash_command(command):
""" Run a system bash command and returns the output """
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
output, error = process.communicate()
if process.returncode != 0:
print("Error:", error.decode())
return output.decode().strip()
# modify the pip .ini file
run_bash_command(f'pip config set global.extra-index-url {url}')```
In your GitHub action, set
env:
PIP_EXTRA_INDEX_URL: "https://pypi.org/simple"
If whatever AWS auth mechanism you're using overwrites pip behavior to only point to CodeArtifact, you can use that env var to get pip to also pull from PyPi, you can remove PyPi as an upstream for your CodeArtifact repo, and you avoid paying Amazon's egress and storage costs to download copies of open source dependencies.
Facing a similar issue here ... primarily because CodeArtifact is a factor 2 slower than PyPi in our CI/CD pipelines I also wanted to use a combination of the 2.
Most people seem to suggest a solution to have PyPi as the default index url, and CodeArtifact as an extra index url but couldn't that lead to security vulnerabilities as people would be able to hijack your private packages by uploading them to PyPi ? (https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-24105)
