rules_python 1.8.0 and venv

[gfrankliu]

Congrats on the 1.8.0 release! I just tried it with venv by adding below to my .bazelrc (I am using bazel 8.5.0 on Linux):

common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes

then bazel run a trial py_binary:

bazel run //util/validate_python_ml_requirements:validate_python_ml_requirements

(to see if tensorflow can finally find all cuda libraries via venv)

Now if I go to the directory:

bazel-bin/util/validate_python_ml_requirements/_validate_python_ml_requirements.venv/bin

I see "python3" file there, symlinked to something non-exist:

$ cksum python3
cksum: python3: No such file or directory

Am I missing anything?

[rickeylev]

The relative symlinks created within the venv are only valid relative to the runfiles location.

So the path to access it would be e.g.

bazel-bin/util/validate_python_ml_requirements/validate_python_ml_requirements.runfiles/_main/util/validate_python_ml_requirements/validate_python_ml_requirements.venv/bin/python3

[gfrankliu]

Thanks for the clarification!
So the "_validate_python_ml_requirements.venv" directory inside bazel-bin/util/validate_python_ml_requirements is not used? Can we not create it to avoid confusion if it is not used and has wrong symlinks in it?

[rickeylev]

Unfortunately no, we can't prevent it; it's part of how bazel creates and materializes files. I do agree their visibility is annoying and confusing (I've seen people (myself included 😅) use them, not realizing they're not technically the "correct" file path to use for their situation). (Trivia: in the past, such generated files were put in "bazel-genfiles", but that's been being phased out for a decade).

The files are used, just in an indirect way.

The bazel-bin/ files are what bazel calls the "exec root". Whenever a generate file is created, it gets materialized in the bazel exec root (i.e its the actual file that data gets written to). It's like the "backing store" for generated file content. Later, when the file is included in a binary's runfiles, it shows up as a symlink from bazel-bin/bin.runfiles/whatever.txt -> bazel-bin/whatever.txt.

[shayanhoshyari]

I had a colleague who decided to ignore rules_python, write his own python rules, and in that process not use .runfiles and just use bazel created files.

That ended badly 😂 🤦

(one random problem was runfiles are always kept clean by bazel, e.g. no stale file, but output tree can contain any random files)

[gfrankliu]

@rickeylev if the correct venv directory we should use is inside runfiles directory, can we update the runfiles_manifest file point to that venv directory instead? We depend on the runfiles_manifest file to pack the files for the py image layer tar.

[aignas]

can we update the runfiles_manifest file point to that venv directory instead?

I am looking at the manifest and I am not sure what you mean by that. We don't do anything in particular to the runfiles manifest and I'd be interested to learn if there is a way for us to manipulate that in some way.

[rickeylev]

While we specify the set of runfiles, we don't control the exact output format, that's handled by Bazel, so no, we can't make the runfiles manifest point anywhere specific.

we depend on runfiles_manifest to pack the files for py image layer tar

How so? It should work mostly the same as before. The main difference I can think of is the raw symlinks used by the venv. The runfiles_manifest won't carry information to distinguish "must be symlink" from "happens to be a symlink". Is that the problem?

[gfrankliu]

I was doing something like this to create a bsdtar mtree spec from the runfiles_manifest. Since the manifest file now has invalid links, the bsdtar using the resulting mtree spec complains and fails.

[aignas]

This is an upstream issue for the tar.bzl because during the mtree manifest creation, the ruleset does not understand that the files should be preserved as relative symlinks.

At some point bazel-contrib/tar.bzl#16 (comment) used to work for me but later I switched to rules_pkg as it has better performance and it understands symlinks.

I'll close this issue as there is little we can do here.

For now, the mtree manifest has to be post-processed to support relative symlinks. The rules_img ruleset might be supporting symlinks as well as I have noted in #3117 description.