cache
cache copied to clipboard
actions
False positive cache hit
Overview
- Expected behavior: Observe a cache miss.
- Observed behavior: actions/cache reports a cache hit when no cache is present.
We are using actions/cache lookup only to check whether we have previously executed a job or not. It has been working OK until we identified two instances (there may be more) in which actions/cache reported a hit when it was supposed to be a miss. Here are a couple of examples:
Example 1
Originally posted here regarding this run:
This is what I see from calling gh cache ls --json id,key:
[END)
{
"id": 396,
"key": "docker-dependencies-Linux-6d19bd7b980701f22c2af8652e06f82ed05ebeefd65b4033dc89fe1c3abecd38-build-main-pool"
},
{
"id": 410,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-7f0445647fb5a2aac10e6a6845e489bbe0cff9662f88cc3cfca939c2fd8e1b7f"
},
{
"id": 412,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-84b70f73dd13fdbaa431ccc8dcd245057774fde673c98049e605e1f0ede96df8"
},
{
"id": 411,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-460bc19305d5154a0091cdf9ecf08b62e3e63297ae0fbad8ddc190de03304291"
},
{
"id": 408,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-84b70f73dd13fdbaa431ccc8dcd245057774fde673c98049e605e1f0ede96df8"
},
{
"id": 407,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-7f0445647fb5a2aac10e6a6845e489bbe0cff9662f88cc3cfca939c2fd8e1b7f"
},
{
"id": 409,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-460bc19305d5154a0091cdf9ecf08b62e3e63297ae0fbad8ddc190de03304291"
},
{
"id": 393,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-4c07a1f8fdd193a63c4c9ec0d23873a8a5822196c33e385f05398cd9b830e0aa"
},
{
"id": 406,
"key": "docker-dependencies-Linux-6d19bd7b980701f22c2af8652e06f82ed05ebeefd65b4033dc89fe1c3abecd38-clarify-impure"
},
{
"id": 397,
"key": "docker-dependencies-Linux-6d19bd7b980701f22c2af8652e06f82ed05ebeefd65b4033dc89fe1c3abecd38-128-remove-old-action"
}
]
And this is what I see from the step of that run:
Run actions/cache@v4
with:
key: docker-dependencies-Linux-f4bcfb55392f2a994483280942820d4c4affe91f35b437367ba7e4c0b6021444-51-check-gi-density
lookup-only: true
path: ./DESCRIPTION
enableCrossOsArchive: false
fail-on-cache-miss: false
save-always: false
env:
REGISTRY: cfaprdbatchcr.azurecr.io
IMAGE_NAME: cfa-epinow2-pipeline
Lookup only - skipping download
Cache found and can be restored from key: docker-dependencies-Linux-f4bcfb55392f2a994483280942820d4c4affe91f35b437367ba7e4c0b6021444-51-check-gi-density
So I have no idea why it's making a hit since there's no hash with f4bc* in the list of existing caches!
Example 2:
Thanks for the example, @dylanhmorris! Here is the output I'm getting from checking the cache:
gh cache ls --json id,key
[
{
"id": 1215,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-f85fa02bc57d3f407305e6d5bf93e59911ca7e0eb3a184f5c9f9518c5000d8af"
},
{
"id": 1214,
"key": "docker-dependencies-Linux-02b5e2d6aef8dd3a4ea8f51b074d3269bab5dfef8507c178d50fc75350e7aff1-dhm-run-pyrenew-h"
},
{
"id": 1191,
"key": "Windows Server 2022 x64 (build 20348)-R version 4.4.2 (2024-10-31 ucrt)-1-d47748fab30e34e26ffc10f8970c48c11008041b22e082e2ed8525c822f64f38"
},
{
"id": 1192,
"key": "Windows Server 2022 x64 (build 20348)-R version 4.4.0 (2024-04-24 ucrt)-1-2e815b11189127381077466a7f7b248e3f149541715601ebbe42fb3ef3b04309"
},
{
"id": 1213,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-3942e21f75d57e70130b5aba1366b5c20bad24e308e50eb7528285bb009c7895"
},
{
"id": 1212,
"key": "macOS Sonoma 14.7.2-R version 4.4.0 (2024-04-24)-1-3c831122a1d6cb7b3298ba9106e77bfd313c2a4953e41041c144b9abc49d59fb"
},
{
"id": 1211,
"key": "macOS Sonoma 14.7.2-R version 4.4.2 (2024-10-31)-1-552fe97f674812742e51a10283f3c4130b22593f0a62efba81d78ec07611d42d"
},
{
"id": 1210,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.0 (2024-04-24)-1-75a8387d30a9e32c76d0e5883aedd4e6c6742016baa21d0c84e59568d8b2b50b"
},
{
"id": 1189,
"key": "macOS Sonoma 14.7.2-R version 4.4.0 (2024-04-24)-1-a46748d991d0f385a4b67206662ead3630e350fec232e19861a724661f8d32ea"
},
{
"id": 1188,
"key": "macOS Sonoma 14.7.2-R version 4.4.2 (2024-10-31)-1-bcd1e597b03282c6da48a8b4f3146cdb20d08a37f1a3d18462118d6f533a828f"
},
{
"id": 1186,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-cd166cd05a4988ff0f6212575fe8b681837fb8f5ff5ef5c03324bef0b65f6c36"
},
{
"id": 1194,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-17787be3eeb6cd3d84f97d7de16d975cbd461fd2a86065be16d1bbd18fa6f341"
},
{
"id": 1187,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.0 (2024-04-24)-1-488e219a9a58ae5305b17158881bf08bba9ace8c082766f5f169607d7cd342eb"
},
{
"id": 650,
"key": "pre-commit-3|/opt/hostedtoolcache/Python/3.12.8/x64|7062add5de4c02de988411686969a115a1d328795dbff87a75840862398b679f"
},
{
"id": 651,
"key": "r-precommit-cache|/opt/hostedtoolcache/Python/3.12.8/x64|7062add5de4c02de988411686969a115a1d328795dbff87a75840862398b679f"
},
{
"id": 1184,
"key": "Linux-poetry"
},
{
"id": 1209,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-f85fa02bc57d3f407305e6d5bf93e59911ca7e0eb3a184f5c9f9518c5000d8af"
},
{
"id": 1208,
"key": "docker-dependencies-Linux-02b5e2d6aef8dd3a4ea8f51b074d3269bab5dfef8507c178d50fc75350e7aff1-dhm-pick-and-choose-denom"
},
{
"id": 1150,
"key": "Windows Server 2022 x64 (build 20348)-R version 4.4.0 (2024-04-24 ucrt)-1-2e815b11189127381077466a7f7b248e3f149541715601ebbe42fb3ef3b04309"
},
{
"id": 1151,
"key": "Windows Server 2022 x64 (build 20348)-R version 4.4.2 (2024-10-31 ucrt)-1-d47748fab30e34e26ffc10f8970c48c11008041b22e082e2ed8525c822f64f38"
},
{
"id": 1207,
"key": "macOS Sonoma 14.7.2-R version 4.4.0 (2024-04-24)-1-3c831122a1d6cb7b3298ba9106e77bfd313c2a4953e41041c144b9abc49d59fb"
},
{
"id": 1162,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-1f3f0940525e68bfdfd78468b47d5bdf086a8e7d193d498ec374308692ce80b2"
},
{
"id": 1206,
"key": "macOS Sonoma 14.7.2-R version 4.4.2 (2024-10-31)-1-552fe97f674812742e51a10283f3c4130b22593f0a62efba81d78ec07611d42d"
},
{
"id": 1148,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-cd166cd05a4988ff0f6212575fe8b681837fb8f5ff5ef5c03324bef0b65f6c36"
},
{
"id": 1169,
"key": "macOS Sonoma 14.7.2-R version 4.4.2 (2024-10-31)-1-0d5c47c21bb57e6fff68ef0cf1e77263b7fd8364f3dfe827cf5745d7ffb8c3a4"
},
{
"id": 1155,
"key": "macOS Sonoma 14.7.2-R version 4.4.0 (2024-04-24)-1-a46748d991d0f385a4b67206662ead3630e350fec232e19861a724661f8d32ea"
},
{
"id": 1147,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.2 (2024-10-31)-1-3942e21f75d57e70130b5aba1366b5c20bad24e308e50eb7528285bb009c7895"
},
{
"id": 1149,
"key": "Ubuntu 22.04.5 LTS-R version 4.4.0 (2024-04-24)-1-75a8387d30a9e32c76d0e5883aedd4e6c6742016baa21d0c84e59568d8b2b50b"
},
{
"id": 1143,
"key": "Linux-poetry"
},
{
"id": 1183,
"key": "docker-dependencies-Linux-02b5e2d6aef8dd3a4ea8f51b074d3269bab5dfef8507c178d50fc75350e7aff1-dhm-run-pyrenew-h"
}
]
The key docker-dependencies-Linux-a512b6c4b4ba68e0babeea1ad38120673c2112ebb935fdb37cda846422f06b9d-dhm-bump-default-pyrenew (which is the one generated by your job) is not present. Nonetheless, I noticed that you ran out of space for cache! For some reason, this is the message I see here (attn @damonbayer):
Approaching total cache storage limit (10.46 GB of 10 GB Used) Least recently used caches will be automatically evicted to limit the total cache storage to 10 GB. Learn more about cache usage.
This may be a bug the happens when you reach the cache limit? I see that cfa-epinow2-pipeline cleared the cache (@amondal2?).
This is also happening to us, but in a slightly different scenario.
We had this workflow seemingly create the cache, although with warnings.
Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/chainlink/chainlink --files-from manifest.txt --use-compress-program zstdmt
Warning: Failed to save: Cache service responded with 429 during upload chunk.
Warning: Cache upload failed because file read failed with EBADF: bad file descriptor, read
Warning: Cache upload failed because file read failed with EBADF: bad file descriptor, read
Warning: Cache upload failed because file read failed with EBADF: bad file descriptor, read
Cache saved with key: chainlink-arm64-15bf819648053976ae8ee9f36d25fc3c6d98977d
Then we had another workflow try and use the cache but it indicated a cache miss, then wasn't able to save the cache.
Run actions/cache@v4
Cache not found for input keys: chainlink-arm64-15bf819648053976ae8ee9f36d25fc3c6d98977d
Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/chainlink/chainlink --files-from manifest.txt --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key chainlink-arm64-15bf819648053976ae8ee9f36d25fc3c6d98977d, another job may be creating this cache. More details: Cache already exists. Scope: refs/pull/15874/merge, Key: chainlink-arm64-15bf819648053976ae8ee9f36d25fc3c6d98977d, Version: cf03ac9a549347c4069ec94d40fa6e18853da93fecd412255eee5526d8cfe9c7
Querying the cache API:
gh api \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
'/repos/smartcontractkit/chainlink/actions/caches?ref=refs/pull/15874/merge&per_page=100'
{
"total_count": 14,
"actions_caches": [
{
"id": 769058,
"ref": "refs/pull/15874/merge",
"key": "Linux-gobuild-ccip-deployment-cbb4806d943fd00054bbcc3b2bb79fee3531e12b6d6adac4be8ab565859ad664-wr-deploy"
},
{
"id": 769027,
"ref": "refs/pull/15874/merge",
"key": "setup-go-Linux-ubuntu24-go-1.23.4-cbb4806d943fd00054bbcc3b2bb79fee3531e12b6d6adac4be8ab565859ad664"
},
{
"id": 769054,
"ref": "refs/pull/15874/merge",
"key": "go-test-hashes-ccip-deployment-wr-deploy-5ce2105"
}
]
}
