build-system
build-system copied to clipboard
Add the ability in bootstrap_repositories.py to resume the process from where it stopped
Tried to import the repos into pulp using the bootstrap_repositories.py scrip and at some point, it failed with the following error
INFO:repo-bootstrapper:Syncing Repository: almalinux-8-nfv i686 http://pulp/pulp/content/prod/almalinux-8-nfv-i686/ from RepositoryRemote: almalinux-8-nfv-i686 i686 http://rsync.repo.almalinux.org/almalinux/8/NFV/i686/os/...
Traceback (most recent call last):
File "/code/scripts/bootstrap_repositories.py", line 273, in <module>
main()
File "/code/scripts/bootstrap_repositories.py", line 262, in main
sync(pulp_client.sync_rpm_repo_from_remote(
File "/usr/lib64/python3.9/functools.py", line 888, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "/code/env/lib/python3.9/site-packages/syncer.py", line 33, in sync_co
return asyncio.get_event_loop().run_until_complete(co)
File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/code/alws/utils/pulp_client.py", line 500, in sync_rpm_repo_from_remote
result = await self.wait_for_task(task['task'])
File "/code/alws/utils/pulp_client.py", line 615, in wait_for_task
raise Exception(f'Task {str(task)} has failed')
Exception: Task {'pulp_href': '/pulp/api/v3/tasks/dcc80ace-f3ea-419e-9152-6aa7de768fe1/', 'pulp_created': '2022-07-15T09:17:29.108353Z', 'state': 'failed', 'name': 'pulp_rpm.app.tasks.synchronizing.synchronize', 'logging_cid': '97c71e1392b54b808348f7979d9efd93', 'started_at': '2022-07-15T09:17:29.224114Z', 'finished_at': '2022-07-15T09:17:29.819785Z', 'error': {'traceback': ' File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 410, in _perform_task\n result = func(*args, **kwargs)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 464, in synchronize\n remote_url = fetch_remote_url(remote, url)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 298, in fetch_remote_url\n remote_url = fetch_mirror(remote)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 250, in fetch_mirror\n result = downloader.fetch()\n File "/usr/local/lib/python3.8/site-packages/pulpcore/download/base.py", line 187, in fetch\n return done.pop().result()\n File "/usr/local/lib/python3.8/site-packages/pulpcore/download/http.py", line 271, in run\n return await download_wrapper()\n File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry\n ret = await target(*args, **kwargs)\n File "/usr/local/lib/python3.8/site-packages/pulpcore/download/http.py", line 256, in download_wrapper\n return await self._run(extra_data=extra_data)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run\n self.raise_for_status(response)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status\n response.raise_for_status()\n File "/usr/local/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status\n raise ClientResponseError(\n', 'description': "404, message='Not Found', url=URL('http://rsync.repo.almalinux.org/almalinux/8/NFV/i686/os')"}, 'worker': '/pulp/api/v3/workers/a629915a-b048-4b35-8198-11baa7659196/', 'parent_task': None, 'child_tasks': [], 'task_group': None, 'progress_reports': [], 'created_resources': [], 'reserved_resources_record': ['/pulp/api/v3/repositories/rpm/rpm/01649371-8733-4f17-b2e2-7b9c2f40f17d/', 'shared:/pulp/api/v3/remotes/rpm/rpm/f2eb53aa-6842-4b9f-90a2-62b770779fee/']} has failed
The problem was already fixed upstream. Had to run the script again, which takes some time because it starts all over again. It would be great if we can bring to this script the ability to resume from where it failed.