rules_python 1.6 cannot find python 3 (local toolchain)

[honglooker]

🐞 bug report

Affected Rule

rules_python/tools/wheelmaker

Is this a regression?

We did not encounter this in 1.4.2: https://github.com/protocolbuffers/protobuf/actions/runs/17471667264/job/49621377398

Description

I upgraded protobuf's rules_python version to 1.6 from 1.4.2. I'm now running into an issue when trying to mirror module.bazel deltas to WORKSPACE.

It looks like rules_python is having difficulty locating python3 for localtoolchains after updating to 1.6.

🔬 Minimal Reproduction

https://github.com/protocolbuffers/protobuf/pull/23341

🔥 Exception or Error

 File "/workspace/_build/out/sandbox/processwrapper-sandbox/1203/execroot/com_google_protobuf/bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/rules_python/tools/wheelmaker", line 411, in _RunExecv
    os.execv(python_program, argv)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/python3'
INFO: From Compiling upb_generator/reflection/header.cc [for tool]:
In file included from upb_generator/reflection/header.cc:15:
In file included from bazel-out/k8-opt-exec-ST-a828a81199fe/bin/src/google/protobuf/compiler/_virtual_includes/code_generator/google/protobuf/compiler/code_generator.h:2

🌍 Your Environment

Operating System:

Linux

Output of bazel version:

bzl 7., 8.

Rules_python version: 1.6.0

Passing all but 1 orthogonal test in 1.4.2: https://github.com/protocolbuffers/protobuf/actions/runs/17471667264/job/49621377398

Failing a lot of Linux in 1.6: https://github.com/protocolbuffers/protobuf/actions/runs/17469109780/job/49612587161?pr=23341

[honglooker]

fyi @rickeylev

[rickeylev]

I tried to repro this, but couldn't.

A longer excerpt from the log:

DEBUG: /workspace/_build/out/external/rules_python/python/private/repo_utils.bzl:83:16: 
rules_python:local_runtime_repo(@@local_python3) WARNING: No external python libraries found.

...
Python Location: [/opt/hostedtoolcache/gcloud/538.0.0/x64/platform/bundledpythonunix/bin/python3]

...

ERROR: /workspace/python/dist/BUILD.bazel:434:9: Building wheel @//python/dist:test_wheel failed: (Exit 1): wheelmaker failed: error executing PyWheel command (from target //python/dist:test_wheel) 
  (cd /workspace/_build/out/sandbox/processwrapper-sandbox/1193/execroot/com_google_protobuf && \
  exec env - \
    LD_LIBRARY_PATH=/opt/venv/lib: \
    PATH=/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/rules_python/tools/wheelmaker --name protobuftests --version 6.33.0 --python_tag py3 --abi none --platform any --out bazel-out/k8-fastbuild/bin/python/dist/protobuftests-6.33.0-py3-none-any.whl --name_file bazel-out/k8-fastbuild/bin/python/dist/test_wheel.name '--strip_path_prefix=python/' '--strip_path_prefix=src/' --input_file_list bazel-out/k8-fastbuild/bin/python/dist/test_wheel_target_wrapped_inputs.txt --metadata_file bazel-out/k8-fastbuild/bin/python/dist/test_wheel.metadata.txt --extra_distinfo_file 'LICENSE;LICENSE')
# Configuration: 82fbc2a81c67e1f0f00710ea1648f8717166a0ba8c3af4f17afee9beb37837b1
# Execution platform: @@internal_platforms_do_not_use//host:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
Traceback (most recent call last):
  File "/workspace/_build/out/sandbox/processwrapper-sandbox/1193/execroot/com_google_protobuf/bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/rules_python/tools/wheelmaker", line 625, in <module>
    Main()
  File "/workspace/_build/out/sandbox/processwrapper-sandbox/1193/execroot/com_google_protobuf/bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/rules_python/tools/wheelmaker", line 610, in Main
    ExecuteFile(
  File "/workspace/_build/out/sandbox/processwrapper-sandbox/1193/execroot/com_google_protobuf/bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/rules_python/tools/wheelmaker", line 385, in ExecuteFile
    _RunExecv(python_program, main_filename, args, env)
  File "/workspace/_build/out/sandbox/processwrapper-sandbox/1193/execroot/com_google_protobuf/bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/rules_python/tools/wheelmaker", line 411, in _RunExecv
    os.execv(python_program, argv)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/python3'

Some notable things from the CI logs:

1. Bazel is being executed under a docker image.
2. /usr/local/bin is on PATH. So, presumably, that directory and its contents are readable?
3. processwrapper-sandbox is being used
4. The failures/passes seem inconsistent:
   • Fail: linux 3.11, linux 3.12, linux 3.13
   • Pass: linux 3.19, Linux 3.10, mac 3.12, mac 3.13, mac
5. The "no external libraries found" means that the libpython*.so files couldn't be found -- not fatal, but does seem odd. It would mean whatever Python was found doesn't have shared libs.
6. The "Python Location" is very different than what bazel finds. It's not clear if that is something available in the environment, or some debugging info about what's been installed on the system as part of gcloud.

Some extra debug info that would be informative is what 1.4 reports for the python it finds. In particular, if it, too, uses /usr/local/bin/python3.