Issues/3922 aws auth profile

[Hexotical]

Changelog Entry

To be copied to the draft changelog by merger:

Will address issue 3922 by adding an environment variable to specify a specific AWS profile with which to use Toil

Reviewer Checklist

• [ ] Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • [ ] If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • [ ] If there is no associated issue, create one.
• [ ] Read through the code changes. Make sure that it doesn't have:
  • [ ] Addition of trailing whitespace.
  • [ ] New variable or member names in camelCase that want to be in snake_case.
  • [ ] New functions without type hints.
  • [ ] New functions or classes without informative docstrings.
  • [ ] Changes to semantics not reflected in the relevant docstrings.
  • [ ] New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • [ ] New features without tests.
• [ ] Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• [ ] Finish the review with an overall description of your opinion.

Merger Checklist

• [ ] Make sure the PR passes tests.
• [ ] Make sure the PR has been reviewed since its last modification. If not, review it.
• [ ] Merge with the Github "Squash and merge" feature.
  • [ ] If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• [ ] Copy its recommended changelog entry to the Draft Changelog.
• [ ] Append the issue number in parentheses to the changelog entry.

[adamnovak]

Looks like defaulting to default might not actually work:

INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT: ==================================== ERRORS ====================================
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT: ________________ ERROR collecting src/toil/test/cwl/cwlTest.py _________________
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT: Traceback (most recent call last):
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/tmp/toil/cwlTests/toil/src/toil/test/cwl/cwlTest.py", line 61, in <module>
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     from toil.test.provisioners.aws.awsProvisionerTest import \
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/tmp/toil/cwlTests/toil/src/toil/test/provisioners/aws/awsProvisionerTest.py", line 27, in <module>
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     from toil.provisioners.aws.awsProvisioner import AWSProvisioner
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/tmp/toil/cwlTests/toil/src/toil/provisioners/aws/awsProvisioner.py", line 54, in <module>
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     from toil.lib.ec2 import (a_short_time,
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/tmp/toil/cwlTests/toil/src/toil/lib/ec2.py", line 285, in <module>
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     iam_client = establish_boto3_session().client('iam')
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/tmp/toil/cwlTests/toil/src/toil/lib/aws/session.py", line 62, in establish_boto3_session
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     return Session(botocore_session=botocore_session, region_name=region_name, profile_name=os.environ.get("TOIL_AWS_PROFILE", "default"))
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/boto3/session.py", line 90, in __init__
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     self._setup_loader()
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/boto3/session.py", line 131, in _setup_loader
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     self._loader = self._session.get_component('data_loader')
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/session.py", line 796, in get_component
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     return self._components.get_component(name)
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/session.py", line 1108, in get_component
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     self._components[name] = factory()
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/session.py", line 193, in <lambda>
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     lambda: create_loader(self.get_config_variable('data_path')),
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/session.py", line 317, in get_config_variable
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     return self.get_component('config_store').get_config_variable(
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/configprovider.py", line 426, in get_config_variable
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     return provider.provide()
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/configprovider.py", line 628, in provide
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     value = provider.provide()
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/configprovider.py", line 718, in provide
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     scoped_config = self._session.get_scoped_config()
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:   File "/usr/local/lib/python3.9/dist-packages/botocore/session.py", line 416, in get_scoped_config
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT:     raise ProfileNotFound(profile=profile_name)
INFO     toil.test.provisioners.clusterTest:clusterTest.py:112 STDOUT: botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

This made https://ucsc-ci.com/databiosphere/toil/-/jobs/24520 fail.

I think maybe when we're using the AWS instance's identity to authenticate our AWS requests, we can't ask for the profile default, because it isn't going to be there. So we probably have to either try None, or do **kwargs expansion, or go back to having two copies of the calls.

The Kubernetes CWL tests also had one test randomly time out; I don't think that was related.