Repository tests can are still cached by bazel if repository rules change but tests themselves do not

[fishcakez]

🐞 bug report

Affected Rule

The issue is caused by the rule:

Is this a regression?

Yes, the previous version in which this bug was not present was: ....

Description

A clear and concise description of the problem...

Repository tests can are marked as cached by bazel if repository rules change but tests themselves do not.

🔬 Minimal Reproduction

Run a test e.g. bazel test //python/tests/toolchains:python_3_8_10_x86_64-apple-darwin_test Add fail("oops") to a the rule,

diff --git a/python/repositories.bzl b/python/repositories.bzl
index d48e775..5baa0fe 100644
--- a/python/repositories.bzl
+++ b/python/repositories.bzl
@@ -38,6 +38,8 @@ def py_repositories():
 STANDALONE_INTERPRETER_FILENAME = "STANDALONE_INTERPRETER"

 def _python_repository_impl(rctx):
+    if rctx.attr.python_version == '3.8.10':
+        fail("oops")
     if rctx.attr.distutils and rctx.attr.distutils_content:
         fail("Only one of (distutils, distutils_content) should be set.")

Then run test again:

bazel test //python/tests/toolchains:python_3_8_10_x86_64-apple-darwin_test
...
//python/tests/toolchains:python_3_8_10_x86_64-apple-darwin_test (cached) PASSED in 93.4s

Retrying with cache tests results disabled:

bazel test //python/tests/toolchains:python_3_8_10_x86_64-apple-darwin_test --cache_test_results=no
...
//python/tests/toolchains:python_3_8_10_x86_64-apple-darwin_test         FAILED in 6.4s

🔥 Exception or Error

N/A

🌍 Your Environment

Operating System:

  
Any
  

Output of bazel version:

  
Any
  

Rules_python version:

  
Any
  

Anything else relevant?

This has bug has allowed the repo to introduce at least a couple of bugs, eg #803 #805