AWS TooManyBuckets

Open larryns opened this issue 2 years ago • 1 comments

Hi,

I'm getting an error:

2023-02-22T13:51:06+0000] [MainThread] [E] [toil.lib.retry] Got a <class 'botocore.exceptions.ClientError'>: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed which is not retriable according to <function AWSJobStore._bindBucket.<locals>.bucket_retry_predicate at 0x7fa049814b80>

I checked my account and 83 buckets were created. I'm running the test_data example for CAT (Comparative Annotation Toolkit). So, I'm guessing the buckets aren't being deleted for some reason?

Any pointers on what I can try to fix this?

Thanks, Larry.

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1294

Feb 22 '23 19:02 larryns

If your AWS account is full of buckets from Toil, you can try running toil clean aws:<region name>:<job store name> for the old job stores that the buckets belong to. A bucket foo-job-store--files would belong to a job store names foo-job-store.

We also have a cleanup script https://github.com/DataBiosphere/toil/blob/42299c7b688d52a59a2bfcfaae9aab82b6280bab/contrib/admin/cleanup_aws_resources.py which will clean up abandoned buckets from Toil tests.

When running workflows, you can pass --clean=always if you don't want the job store data to be left behind to allow for restarting failed workflows.

Are you seeing buckets left behind from workflows that claimed to succeed?

Feb 07 '25 20:02 adamnovak