devspace-sh
Update checks: prevent GitHub rate limit
Is your feature request related to a problem? Yes, when running commands, we always check for updates by calling the GitHub API. When running many commands in a short time (e.g. devspace status sync), we are hitting the GitHub API rate limit and see this error:
[WARN] Couldn't check for newest version: GET https://api.github.com/repos/covexo/devspace/releases: 403 API rate limit exceeded for 178.5.106.180. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 8m47s]
Which solution do you suggest? I suggest running an update service that does the GitHub API request regularly and caches the latest release. When checking for updates, the request should be made against the update service instead of against the GitHub API directly.
Which alternative solutions exist? Alternatively, we could ask the user for a personal access token for the GitHub API but that would require additional configuration and force users to have a GitHub account.
Additional context
/kind feature
@LukasGentele
Just some relevant information that affect your solution.
I recently ran into a similar issue in one of my projects.
Github's api supports conditional requests, which would cause the requests to not count against the rate limit if the requested object hadn't changed. I wonder if that would be helpful here? github conditional requests go-github supports this as well. see here
Switching to conditional requests greatly reduced the number of api calls in my project.
I notice you all are using github.com/rhysd/go-github-selfupdate which uses go-github to talk to the github api, however it does not look like the default updater implements conditional requests.
Wondering if maybe working with go-github-selfupdate to add an updater that supports conditional requests would be an option. I'm not familiar with go-github-selfupdate so maybe there is already a way to accomplish this as is.
Just something to look into.
Thank you for pointing this out. I will take a closer look and see if that would be a suitable solution. It looks like a very promising option.
I went ahead and created an issue in go-github-selfupdate for conditional requests: https://github.com/rhysd/go-github-selfupdate/issues/17
Is there any update on this, I am getting this quite often, the only way to fix it for me currently is to reconnect my ISP.
Another solution is to allow a parameter which downloads the file from github with input as the authorization header.
devspace upgrade --github-pat [PAT]
You can drive inspiration from this: https://stackoverflow.com/a/50145177/686131