[Bug]: Unable to pull the full SWE-Bench test set

[cngarrison]

Is there an existing issue for the same bug?

• [X] I have checked the existing issues.

Describe the bug and reproduction steps

When trying to pull the full test set using:

evaluation/swe_bench/scripts/docker/pull_all_eval_docker.sh instance full

It fails with this error:

Using namespace: xingyaoww
Pulling docker images for [instance] level
Pattern: sweb.base\|sweb.env\|sweb.eval
Image file: evaluation/swe_bench/scripts/docker/all-swebench-full-instance-images.txt
Pulling xingyaoww/sweb.eval.x86_64.sphinx-doc__sphinx-8713:latest into sweb.eval.x86_64.sphinx-doc__sphinx-8713:latest

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview xingyaoww/sweb.eval.x86_64.sphinx-doc__sphinx-8713:latest
Error response from daemon: pull access denied for xingyaoww/sweb.eval.x86_64.sphinx-doc__sphinx-8713, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I have attempted from two systems and get the same error. I have ensured I have logged in with docker.

Pulling the lite test set is successful.

OpenHands Installation

Docker command in README

OpenHands Version

main

Operating System

MacOS

Logs, Errors, Screenshots, and Additional Context

No response

[mamoodi]

@xingyaoww FYI.

[xingyaoww]

Actually i think we can pull this image by doing docker pull xingyaoww/sweb.eval.x86_64.sphinx-doc_s_sphinx-8713.

There's a _s_ in the docker image name now since DockerHub doesn't seems to support __ now.

I think the bug comes from evaluation/swe_bench/scripts/docker/all-swebench-full-instance-images.txt

We should just do a global replace in that file to replace __ with _s_

[github-actions[bot]]

OpenHands started fixing the issue! You can monitor the progress here.

[github-actions[bot]]

An attempt was made to automatically fix this issue, but it was unsuccessful. A branch named 'openhands-fix-issue-4583' has been created with the attempted changes. You can view the branch here. Manual intervention may be required.