build-system icon indicating copy to clipboard operation
build-system copied to clipboard

Add the ability in bootstrap_repositories.py to resume the process from where it stopped

Open forbiddenpotato opened this issue 1 year ago • 0 comments

Tried to import the repos into pulp using the bootstrap_repositories.py scrip and at some point, it failed with the following error

INFO:repo-bootstrapper:Syncing Repository: almalinux-8-nfv i686 http://pulp/pulp/content/prod/almalinux-8-nfv-i686/ from RepositoryRemote: almalinux-8-nfv-i686 i686 http://rsync.repo.almalinux.org/almalinux/8/NFV/i686/os/...
Traceback (most recent call last):
  File "/code/scripts/bootstrap_repositories.py", line 273, in <module>
    main()
  File "/code/scripts/bootstrap_repositories.py", line 262, in main
    sync(pulp_client.sync_rpm_repo_from_remote(
  File "/usr/lib64/python3.9/functools.py", line 888, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/code/env/lib/python3.9/site-packages/syncer.py", line 33, in sync_co
    return asyncio.get_event_loop().run_until_complete(co)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/code/alws/utils/pulp_client.py", line 500, in sync_rpm_repo_from_remote
    result = await self.wait_for_task(task['task'])
  File "/code/alws/utils/pulp_client.py", line 615, in wait_for_task
    raise Exception(f'Task {str(task)} has failed')
Exception: Task {'pulp_href': '/pulp/api/v3/tasks/dcc80ace-f3ea-419e-9152-6aa7de768fe1/', 'pulp_created': '2022-07-15T09:17:29.108353Z', 'state': 'failed', 'name': 'pulp_rpm.app.tasks.synchronizing.synchronize', 'logging_cid': '97c71e1392b54b808348f7979d9efd93', 'started_at': '2022-07-15T09:17:29.224114Z', 'finished_at': '2022-07-15T09:17:29.819785Z', 'error': {'traceback': '  File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 410, in _perform_task\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 464, in synchronize\n    remote_url = fetch_remote_url(remote, url)\n  File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 298, in fetch_remote_url\n    remote_url = fetch_mirror(remote)\n  File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 250, in fetch_mirror\n    result = downloader.fetch()\n  File "/usr/local/lib/python3.8/site-packages/pulpcore/download/base.py", line 187, in fetch\n    return done.pop().result()\n  File "/usr/local/lib/python3.8/site-packages/pulpcore/download/http.py", line 271, in run\n    return await download_wrapper()\n  File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/pulpcore/download/http.py", line 256, in download_wrapper\n    return await self._run(extra_data=extra_data)\n  File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run\n    self.raise_for_status(response)\n  File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status\n    response.raise_for_status()\n  File "/usr/local/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status\n    raise ClientResponseError(\n', 'description': "404, message='Not Found', url=URL('http://rsync.repo.almalinux.org/almalinux/8/NFV/i686/os')"}, 'worker': '/pulp/api/v3/workers/a629915a-b048-4b35-8198-11baa7659196/', 'parent_task': None, 'child_tasks': [], 'task_group': None, 'progress_reports': [], 'created_resources': [], 'reserved_resources_record': ['/pulp/api/v3/repositories/rpm/rpm/01649371-8733-4f17-b2e2-7b9c2f40f17d/', 'shared:/pulp/api/v3/remotes/rpm/rpm/f2eb53aa-6842-4b9f-90a2-62b770779fee/']} has failed

The problem was already fixed upstream. Had to run the script again, which takes some time because it starts all over again. It would be great if we can bring to this script the ability to resume from where it failed.

forbiddenpotato avatar Apr 12 '24 09:04 forbiddenpotato