Skip to content

Simplify CI build job Python setup #7113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
psamanoelton wants to merge 1 commit into
base: master
Choose a base branch
from
+33 −79
Open

Simplify CI build job Python setup #7113

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 33 additions & 79 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,39 @@ jobs:
python_version: ['3.10']
steps:
- uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # v3.3.0
- uses: actions/setup-python@13ae5bb136fac2878aff31522b9efb785519f984 # v4.3.0
with:
python-version: ${{ matrix.python_version }}
architecture: 'x64'
# The self-hosted ml-build container already has the system packages we
# need; installing Python inside the container keeps Bazel's
# @system_python repo pointed at a normal interpreter/header layout and
# avoids the hosted-toolcache header-copy workaround that was previously
# needed with actions/setup-python.
- name: 'Set up Python and system dependencies'
run: |
py_abi="python${{ matrix.python_version }}"
sudo apt-get update
sudo apt-get install -y \
"${py_abi}" \
"${py_abi}-dev" \
"${py_abi}-venv" \
libgbm-dev \
libxss1 \
libasound2
"/usr/bin/${py_abi}" -m venv /tmp/tb-build-venv
echo "/tmp/tb-build-venv/bin" >> "${GITHUB_PATH}"
echo "VIRTUAL_ENV=/tmp/tb-build-venv" >> "${GITHUB_ENV}"
/tmp/tb-build-venv/bin/python -m pip install -U pip setuptools wheel virtualenv
- name: 'Check Python toolchain'
run: |
python --version
python - <<'PY'
import pathlib
import sys
import sysconfig

print("sys.executable =", sys.executable)
include_dir = pathlib.Path(sysconfig.get_config_var("INCLUDEPY"))
print("INCLUDEPY =", include_dir)
print("Python.h exists =", (include_dir / "Python.h").exists())
PY
- name: 'Set up Bazel'
run: |
ci/download_bazel.sh "${BAZEL_VERSION}" "${BAZEL_SHA256SUM}" ~/bazel
Expand All @@ -78,85 +107,10 @@ jobs:
-r ./tensorboard/pip_package/requirements.txt \
-r ./tensorboard/pip_package/requirements_dev.txt \
;
- name: 'Install system dependencies'
run: |
py_abi="python${{ matrix.python_version }}"
sudo apt-get update
sudo apt-get install -y "${py_abi}-dev" libgbm-dev libxss1 libasound2
# Bazel's system_python repository resolves headers relative to the
# interpreter installed by actions/setup-python. The self-hosted
# ml-build container does not provide Python.h under that prefix, so
# copy the system headers into the managed interpreter include dir.
# A real directory copy is more robust here than a symlink because
# Bazel's repository setup may materialize or traverse this path in
# ways that do not preserve symlink behavior.
sudo mkdir -p "${pythonLocation}/include"
sudo rm -rf "${pythonLocation}/include/${py_abi}"
sudo mkdir -p "${pythonLocation}/include/${py_abi}"
sudo cp -a "/usr/include/${py_abi}/." "${pythonLocation}/include/${py_abi}/"
includepy="$(python - <<'PY'
import sysconfig
print(sysconfig.get_config_var("INCLUDEPY"))
PY
)"
# upb's system_python repository rule shells out to `python3` and
# uses sysconfig.get_config_var("INCLUDEPY"), which under
# actions/setup-python inside this container points at
# /opt/hostedtoolcache/... rather than ${pythonLocation}. Materialize
# that INCLUDEPY path too so Bazel's @system_python repo can symlink
# real headers into external/system_python/python.
sudo mkdir -p "$(dirname "${includepy}")"
sudo rm -rf "${includepy}"
sudo cp -a "${pythonLocation}/include/${py_abi}" "${includepy}"
test -f "${pythonLocation}/include/${py_abi}/Python.h"
test -f "${includepy}/Python.h"
- name: 'Debug Python toolchain'
run: |
py_abi="python${{ matrix.python_version }}"
echo "pythonLocation=${pythonLocation}"
echo "PATH=${PATH}"
which python
python --version
command -v python3.10-config || true
python3.10-config --includes || true
python - <<'PY'
import os
import pathlib
import sys
import sysconfig

print("sys.executable =", sys.executable)
for key in ("include", "platinclude", "stdlib", "platstdlib", "scripts", "data"):
print(f"{key} =", sysconfig.get_path(key))
for key in ("INCLUDEPY", "CONFINCLUDEPY", "LIBDIR", "LDLIBRARY", "MULTIARCH"):
print(f"{key} =", sysconfig.get_config_var(key))
include_dir = pathlib.Path(sysconfig.get_path("include"))
print("include_dir exists =", include_dir.exists(), include_dir)
print("Python.h exists =", (include_dir / "Python.h").exists())
print("pythonLocation =", os.environ.get("pythonLocation"))
PY
ls -la "${pythonLocation}" || true
ls -la "${pythonLocation}/include" || true
ls -la "${pythonLocation}/include/${py_abi}" || true
- name: 'Check Pip state'
run: pip freeze --all
- name: 'Bazel: fetch'
run: bazel fetch //tensorboard/...
- name: 'Debug Bazel system_python repo'
run: |
output_base="$(bazel info output_base)"
echo "output_base=${output_base}"
find "${output_base}/external/system_python" -maxdepth 3 \
\( -name 'Python.h' -o -name 'BUILD.bazel' -o -name 'WORKSPACE' \) \
-print | sort || true
if [ -f "${output_base}/external/system_python/BUILD.bazel" ]; then
sed -n '1,200p' "${output_base}/external/system_python/BUILD.bazel"
fi
if [ -d "${output_base}/external/system_python/python" ]; then
ls -la "${output_base}/external/system_python/python"
else
echo "system_python/python directory not found"
fi
- name: 'Bazel: build'
# Note we suppress a flood of warnings from the proto compiler.
# Googlers see b/222706811 & b/182876485 discussion and preconditions
Expand Down
Loading