Download speed on self-hosted runners is significantly slower

[ajmnz]

👋🏼

We've started working with self-hosted runners, but noticed that the download speed when downloading the cache is significantly slower than on hosted runners. Here's the breakdown of what is happening:

• Self-hosted runner version: 2.296.2
• Cache size: ~300MB
• Average download speed
  • Hosted runners: 50 MBs/s
  • Self-hosted runners: 3.5 MBs/s

Hosted:

Self-hosted:

Workflow:

      - name: Cache yarn dependencies
        uses: actions/cache@v3
        id: cache-dependencies
        with:
          path: node_modules
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: Install dependencies
        if: steps.cache-dependencies.outputs.cache-hit != 'true'
        run: |
          yarn install --frozen-lockfile

I'm pretty confident there's no limitation on our machine, and I'm not sure if this is a limitation on the runner itself, but at that speed installing dependencies takes practically the same time as pulling the cache.

Thanks in advance!

[anuragc617]

@ajmnz is your self-hosted runner hosted in public cloud or on your private network? One thing to note is that hosted runners are present in the same regional cloud network as the blob storage, that's why the download or upload speed is generally high as compared to self-hosted runners.

[ajmnz]

Hi @anuragc617, the runner is hosted in public cloud. More specifically, in a dedicated server with plenty of resources available, where the only thing running are the self-hosted runners.

I understand the region issue, but it just seems odd that there would be such a difference in download speeds, with hosted runners sometimes reaching +100 MB/s and self-hosted runners never going above 4-4.5 MB/s. To me, it just looks like there might be some throttling because of how consistent speeds are on self-hosted runners.

[klaus993]

I'm having the same issue (10 MB/s in self-hosted, 100 MB/s in GitHub hosted runners), and I'm wondering: why isn't the cache hosted locally (in the runner) so there is no need for internet transfer? Where exactly is it hosted?

[aristotaloss]

We're seeing the exact same, but just to bust some future "It's your network speed" comments:

There also doesn't seem to be a way to specify a custom endpoint. What can we do?

[omairvaiyani]

We moved to Self-Hosted Runners to reduce our build times with custom AMIs and bigger instances, but our efforts are being hindered by extremely slow cache download and post-job upload steps. What takes just 10-20 seconds on the GitHub hosted runner sometimes takes up to 15 minutes on our Self-Hosted runners.

[ajmnz]

Any news? https://github.com/actions/runner/issues/2146 looks like a pretty similar issue

[vsvipul]

We are tracking self-hosted runner slowness in https://github.com/actions/cache/issues/720 . Please continue the discussion there. Closing this in the meanwhile.