Imagebuilder command cannot receive all images

[jaypan13]

Describe the bug

aws imagebuilder list-images command always returns nextToken causing it to not get all images from ec2 image builder.

I found that it is getting stopped after 25 results and then it sends back nextToken no matter what option I use. So, I tried to filter it with --query like below

aws imagebuilder list-images --query "imageVersionList[?version=='2024.10.15']" --output table

and even though I have less than 20 images with that version, it showed only 3 images in the result. Next I tried using --filters option and it still returned only 3 results with nextToken

aws imagebuilder list-images --filters "name=version,values='2024.10.15'" | jq -r '.nextToken' | cut -c1-4

In summary, I used all options and still cannot return my version = 2024.10.15 images in cli.

P.S. Just a suggestion – the global --no-paginate option could really benefit from a different name. Many people assume that using this option means pagination is disabled and that all results will be returned in one go. Although

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

It should return 20 images

Current Behavior

I tried all options and still cannot return my version = 2024.10.15 images when using aws cli.

Reproduction Steps

1. Create 20 image builder pipelines
2. Create images using those pipelines (Probably you need to create more images so the count gets more than 25 - I cannot test this unfortunately and I have more than 100 images generated with different versions)
3. Using aws cli, try to get all 20 images with specific version.

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.17.13 Python/3.11.9 Linux/5.15.153.1-microsoft-standard-WSL2 exe/x86_64.ubuntu.22

Environment details (OS name and version, etc.)

Ubuntu in wsl

[tim-finnigan]

Thanks for reaching out. Here is a link to the documentation for that command: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/imagebuilder/list-images.html. There is actually a limitation here with the underlying ListImages API, where it will only return up to 25 results at a time. The --query parameter is for client-side filtering, so that won't be able to filter on anything not returned by the API.

You can try setting the --by-name boolean parameter for filtering by name, for example:

aws imagebuilder list-images  --by-name  --filters "name=name,values=test-recipe"

Another option you could try using is a boto3 script like this to fetch all of the results:

import boto3

list = []
client = boto3.client('imagebuilder')

response = client.list_images(filters=[
    {'name': 'name', 'values': ['test-recipe']}])
list.append(response['imageVersionList'])

while 'nextToken' in response:
    response = client.list_images(filters=[
        {'name': 'name', 'values': ['test-recipe']}], nextToken=response['nextToken'])
    list.append(response['imageVersionList'])

print(list)

Upon searching internally I founder there was a previous issue related to this filtering behavior. I think the EC2 Image Builder team was planning improvements to this — I'm not sure if there are any updates on that but will try following up with them.

[jaypan13]

Thank you for boto script. I used bash to do similar kind of task.

I cannot use --by-name as image name contains some string that I need to look for and it is not exact match.

I understand that issue relies with underlying API returning back 25 results. But even though I have less than 25 results on the server, filters command is not returning back all the results at the same time. I assume --filters is happening on the server side unlike --query, which is happening at client side as you mentioned. I tried below query too.

aws imagebuilder list-images --filters "name=version,values='2024.10.15'" --max-results 25

Not sure if that other issue you mentioned in your comment is related with the same thing that I mentioned above, but just putting it here for later knowledge.

[tim-finnigan]

We heard back from the service team who noted that there are two issues here: their service doesn't auto-paginate in the CLI due to an explicit exclusion in the AWS CLI, and their API sometimes gives empty pages due to how filtering works.

They are working on addressing the second issue with empty pages, but this has not been prioritized yet. Please refer to the CHANGELOG for updates going forward and feel free to check back in the future. But since this cannot be addressed directly via the AWS CLI and must be supported by the service team, I will close this issue as not planned.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

[tarunb12]

Just an update here: Image Builder does now auto-paginate responses in the CLI, and supports paginators for all List APIs in boto3.