launching python subprocess can fail: `rules_python==1.7.0` + `bootstrap_impl=system_python`

[shayanhoshyari]

🐞 bug report

Affected Rule

py_binary() / py_test() that invokes python as subprocess. When using --@rules_python//python/config_settings:bootstrap_impl=system_python (default)

--@rules_python//python/config_settings:bootstrap_impl=script does not reproduce this issue.

Is this a regression?

Did this behavior use to work in the previous version? Yes

Description

I have had python executables where they launched a python subprocess as part of them. A basic one for example looked like

# Python code
import sys
 
# do some preprocess
subprocess.check_output([sys.executable, "-m", "datamodel_code_generator", ...
# do some post process

# BUILD code
py_binary(
  name = "pydantic_generator",
  deps = ["@pypi//:datamodel_code_generator", ...]
)

With rules_python==1.7.0 I get an error that no module named datamodel_code_generator found.

🔬 Minimal Reproduction

https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/subprocess_call

🔥 Exception or Error

/private/var/tmp/_bazel_hoshyari/5bf493163e144814d5d8244a8cb0233d/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/test.runfiles/rules_python++python+python_3_13_aarch64-apple-darwin/bin/python3: No module named datamodel_code_generator

🌍 Your Environment

Operating System:

Reproduces on both ubuntu (tried 22.04) MacOS (tried Sonoma)

Output of bazel version:

bazel 8.2.1

Rules_python version:

1.7.0

Anything else relevant?

Seems this is due to https://github.com/bazel-contrib/rules_python/commit/b8e32c454a1158cd78ce4ecaef809b99bef4e5da

1. 1.6.3 and older used PYTHONPATH so it propagated in subprocess calls.
2. 1.7.0 starts using the combination
   1. bootstrap stage 1 that calls
   2. bootstrap stage 2 that adds iii to site.addsitepackage, then runs main entrypiont with runpy.
   3. siteinit.py that adds the new imports.

The issue is that adding folders via site.addsitepackage do not propagate via subprocess.call. So when the main script launches a new python, it is not aware of ii and hence iii.

bootstrap_impl=script does not have this issue as sys.executable resolves to a different python (test.venv/bin/python) that always is aware of iii even when ii is out of picture.

(I suspect if we launch with -S even bootstrap_impl=script will stop working)

Fix ideas:

1. use same venv python for system_python too
2. use sitecustomize + PYTHONPATH in bootstrap stage 1 instead of site.addsitepackage in bootstrap stage 2. This should propagate.

[shayanhoshyari]

Seems https://github.com/bazel-contrib/rules_python/issues/3422 is relevant. But it might not be the same issue, as this is not strictly about adding the runfiles root.

[rickeylev]

use the venv for system_python

With the two bootstrap sharing more code now, this is pretty feasible now. The system_python bootstrap is basically just a stage1 bootstrap, just implemented in python instead of bash.

Hm, actually, looking at the python_bootstrap_template.txt, it seems the code for that is already there? Ah, I see -- L516 in py_executable.bzl skips creating a full venv for bootstrap=system_python. Looks like it just creates the site-packages parts. I think there was an obscure CI failure I couldn't figure out? In any case, that's probably the line that needs to be changed.

use sitecustomize + PYTHONPATH

I had considered this, but was concerned that it'd collide with a user's sitecustomize (e.g. one coming from a docker image that a py_binary gets installed into).

Using -S will break bootstrap=script

To some extent, yes, this is true. However, in stage2, the site.addsitepackage() call is invoked if the venv's site-package directory doesn't appear to be on sys.path, which works around the issue. This won't help direct invocations of sys.executable, though.

Also, a non-bazel venv will break with -S, too; venvs rely on the "find the site-packages directory" logic that having the "site" functionality enabled triggers.