JuliaRegistries
Add check for rate limiting
The inexplicable 403 errors we're seeing on repos that once worked but started to fail may be rate limited.
We can probe the status here https://docs.github.com/en/rest/overview/rate-limits-for-the-rest-api?apiVersion=2022-11-28#checking-the-status-of-your-rate-limit
Note that my theory is that GitHub are serving the wrong error message. We're seeing
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "https://docs.github.com/rest/git/refs#create-a-reference"}
I suspect we should be seeing something along the docs here
Exceeding the rate limit If you exceed your primary rate limit, you will receive a 403 or 429 response, and the x-ratelimit-remaining header will be 0. You should not retry your request until after the time specified by the x-ratelimit-reset header.
If you exceed a secondary rate limit, you will receive a 403 or 429 response and an error message that indicates that you exceeded a secondary rate limit. If the retry-after response header is present, you should not retry your request until after that many seconds has elapsed. If the x-ratelimit-remaining header is 0, you should not retry your request until after the time, in UTC epoch seconds, specified by the x-ratelimit-reset header. Otherwise, wait for at least one minute before retrying. If your request continues to fail due to a secondary rate limit, wait for an exponentially increasing amount of time between retries, and throw an error after a specific number of retries.
Continuing to make requests while you are rate limited may result in the banning of your integration.
The easiest would be to (when we get a 4xx error) dump the HTTP headers that GitHub sent back as part of the 4xx. And then we can e.g. inspect the x-ratelimit-remaining header.
We can configure retries with PyGithub like in this example: https://github.com/PyGithub/PyGithub/issues/2759#issuecomment-1745643733
