-
Notifications
You must be signed in to change notification settings - Fork 49
feat(packagesettings): add dependency-chained build tags #1053
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
dhellmann
wants to merge
1
commit into
python-wheel-build:main
Choose a base branch
from
dhellmann:build-dependency-changelog-chaining
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,250 @@ | ||
| Dependency-Chained Build Tags | ||
| ============================== | ||
|
|
||
| When building packages that depend on system libraries or other packages, you may | ||
| want the build tag to automatically increment when any dependency's build | ||
| configuration changes. This ensures downstream packages are rebuilt when their | ||
| dependencies are updated, even if the downstream package itself hasn't changed. | ||
|
|
||
| Use Cases | ||
| --------- | ||
|
|
||
| Dependency-chained build tags are particularly useful for: | ||
|
|
||
| * **Platform Dependencies**: Packages that depend on CUDA, ROCm, or other | ||
| accelerator stacks where the platform version changes independently of your | ||
| package code. | ||
|
|
||
| * **Midstream Builds**: When you maintain patches or configuration for upstream | ||
| packages and need downstream dependents to rebuild when you change those | ||
| patches. | ||
|
|
||
| * **Transitive Rebuilds**: Ensuring an entire dependency chain rebuilds when a | ||
| base library changes (e.g., when updating OpenSSL, all packages that depend | ||
| on it should get new build tags). | ||
|
|
||
| How It Works | ||
| ------------ | ||
|
|
||
| Build tags are calculated as: | ||
|
|
||
| .. code-block:: text | ||
|
|
||
| build_tag = own_changelog_count + sum(dependency_build_tags) | ||
|
|
||
| Dependencies are resolved **recursively** and **transitively**. If package A | ||
| depends on B, and B depends on C, then A's build tag includes changes from both | ||
| B and C. | ||
|
|
||
| Basic Example | ||
| ------------- | ||
|
|
||
| Consider a simple dependency chain where ``torch`` depends on ``cuda-toolkit``: | ||
|
|
||
| **overrides/settings/cuda-toolkit.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| changelog: | ||
| "12.8": | ||
| - "Initial CUDA 12.8 support" | ||
| "12.9": | ||
| - "Updated to CUDA 12.9" | ||
|
|
||
| **overrides/settings/torch.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - cuda-toolkit | ||
| changelog: | ||
| "2.0.0": | ||
| - "Initial build" | ||
|
|
||
| When you build ``torch==2.0.0`` with CUDA 12.9: | ||
|
|
||
| * ``cuda-toolkit`` has 1 changelog entry for version 12.9 → build tag = ``1`` | ||
| * ``torch`` has 1 own changelog entry + 1 from cuda-toolkit → build tag = ``2`` | ||
|
|
||
| If you later add another CUDA changelog entry: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| changelog: | ||
| "12.9": | ||
| - "Updated to CUDA 12.9" | ||
| - "Fixed memory issue" # New entry | ||
|
|
||
| Now: | ||
|
|
||
| * ``cuda-toolkit`` version 12.9 → build tag = ``2`` | ||
| * ``torch`` version 2.0.0 → build tag = ``3`` (automatically incremented!) | ||
|
|
||
| The wheel filename becomes: ``torch-2.0.0-3-cp311-cp311-linux_x86_64.whl`` | ||
|
|
||
| Transitive Dependencies | ||
| ----------------------- | ||
|
|
||
| Dependencies are resolved transitively through the entire chain. | ||
|
|
||
| **overrides/settings/triton.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - cuda-toolkit | ||
| changelog: | ||
| "2.3.0": | ||
| - "Triton for CUDA 12.x" | ||
|
|
||
| **overrides/settings/torch.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - triton | ||
| changelog: | ||
| "2.0.0": | ||
| - "Initial build" | ||
|
|
||
| Build tags for ``torch==2.0.0``: | ||
|
|
||
| * ``cuda-toolkit`` (version 12.9): 2 changelog entries → build tag = ``2`` | ||
| * ``triton`` (version 2.3.0): 1 own + 2 from cuda-toolkit → build tag = ``3`` | ||
| * ``torch`` (version 2.0.0): 1 own + 3 from triton → build tag = ``4`` | ||
|
|
||
| Notice that ``torch`` **automatically includes** ``cuda-toolkit``'s changes even | ||
| though it only directly lists ``triton`` as a dependency. | ||
|
|
||
| Multiple Dependencies | ||
| --------------------- | ||
|
|
||
| A package can depend on multiple packages. All dependency build tags are summed. | ||
|
|
||
| **overrides/settings/vllm.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - torch | ||
| - triton | ||
| - cuda-toolkit | ||
| changelog: | ||
| "0.3.0": | ||
| - "Initial vLLM build" | ||
|
|
||
| If each dependency has a build tag of 2, then: | ||
|
|
||
| .. code-block:: text | ||
|
|
||
| vllm build_tag = 1 (own) + 2 (torch) + 2 (triton) + 2 (cuda-toolkit) = 7 | ||
|
|
||
| "Fake Packages" for Platform Dependencies | ||
| ------------------------------------------ | ||
|
|
||
| You can create settings files for platform dependencies (like CUDA, ROCm) that | ||
| don't actually exist as Python packages. These act as markers to track platform | ||
| version changes. | ||
|
|
||
| **overrides/settings/cuda-toolkit.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| # No source code - just a version marker | ||
| changelog: | ||
| "12.8": | ||
| - "CUDA 12.8.0 release" | ||
| "12.9": | ||
| - "CUDA 12.9.0 release" | ||
|
|
||
| **overrides/settings/rocm-runtime.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| changelog: | ||
| "6.0": | ||
| - "ROCm 6.0 release" | ||
| "6.1": | ||
| - "ROCm 6.1 release" | ||
|
|
||
| Now packages can depend on these platform markers: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - cuda-toolkit # or rocm-runtime for ROCm builds | ||
|
|
||
| Circular Dependency Detection | ||
| ------------------------------ | ||
|
|
||
| Fromager automatically detects and prevents circular dependencies: | ||
|
|
||
| **overrides/settings/package-a.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - package-b | ||
|
|
||
| **overrides/settings/package-b.yaml**: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - package-a | ||
|
|
||
| This will raise an error: | ||
|
|
||
| .. code-block:: text | ||
|
|
||
| ValueError: Circular dependency detected: package-a appears in | ||
| dependency chain: package-a -> package-b -> package-a | ||
|
|
||
| Scope and Limitations | ||
| ---------------------- | ||
|
|
||
| **Version-Independent** | ||
|
|
||
| Dependencies apply to **all versions** of a package. You cannot specify | ||
| different dependencies for different versions, or use version constraints: | ||
|
|
||
| .. code-block:: yaml | ||
|
|
||
| dependencies: | ||
| - torch # ✓ Correct | ||
| - torch>=2.0 # ✗ Not supported | ||
| - torch; sys_platform=='linux' # ✗ Not supported | ||
|
|
||
| **Per-Package, Not Per-Variant** | ||
|
|
||
| Dependencies are package-level, not variant-level. If you need different | ||
| dependencies for CUDA vs ROCm variants, use separate packages or platform markers. | ||
|
|
||
| **Build Tags Only** | ||
|
|
||
| The ``dependencies`` field only affects build tag calculation. It does **not**: | ||
|
|
||
| * Add runtime dependencies to wheel metadata | ||
| * Affect dependency resolution during bootstrap | ||
| * Change the build environment or compilation flags | ||
|
|
||
| Best Practices | ||
| -------------- | ||
|
|
||
| 1. **Use Semantic Changelog Entries**: Write clear changelog entries that | ||
| explain what changed and why a rebuild is needed. | ||
|
|
||
| 2. **Minimize Dependencies**: Only list direct dependencies that actually affect | ||
| the build. Transitive dependencies are included automatically. | ||
|
|
||
| 3. **Platform Markers**: Use fake packages for system dependencies (CUDA, ROCm, | ||
| OpenSSL) to track platform version changes separately from Python packages. | ||
|
|
||
| 4. **Testing**: When adding a dependency, verify the build tag increments as | ||
| expected by checking the wheel filename. | ||
|
|
||
| See Also | ||
| -------- | ||
|
|
||
| * :doc:`/reference/config-reference` - Full configuration reference | ||
| * :doc:`/customization` - Comprehensive customization guide |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolve each dependency against its own version.
Line 343 passes the parent package's
versioninto every dependency lookup. That means a build liketorch==2.0.0can only see changelog entries keyed as2.0.0incuda-toolkitortriton, so the mixed-version / fake-package cases described in this PR never contribute to the tag.build_tag()needs dependency-specific versions (or resolved requirements), not a single root version threaded through the whole recursion.🤖 Prompt for AI Agents
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is something we should fix
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
Let me look at the
Settingsclass and the test cases to understand the intended behavior and available APIs.🏁 Script executed:
Length of output: 41
🏁 Script executed:
Length of output: 1391
🏁 Script executed:
Length of output: 0
🏁 Script executed:
Length of output: 2209
🏁 Script executed:
Length of output: 314
🏁 Script executed:
Length of output: 1850
🏁 Script executed:
Length of output: 88
🏁 Script executed:
Length of output: 4068
@rd4398, the bug is confirmed and worth fixing. Here's the full analysis:The problem is real but currently hidden. All
dep_chaintests useVersion("1.0.0")which happens to match the changelog keys in all dep_chain YAML fixtures — so the version is coincidentally correct. The fake_cuda tests only exercise that package directly; there's no test for a package that depends ontest-fake-cuda. Iftorch==2.0.0listedtest-fake-cudaas a dependency,_calculate_build_tagwould calldep_pbi._calculate_build_tag(Version("2.0.0"), ...), find no entry for"2.0.0"in the CUDA changelog, and contribute 0.The right fix: For dependency contributions, don't pass the parent's version at all — instead sum all changelog entries across all versions of the dependency. The semantics shift from "what's the build tag of this dep at version X" to "how much has this dep chain changed in total". This is version-agnostic, which is consistent with the
dependenciesfield having no version specifier.Here's the suggested implementation:
Why this is correct for each case:
test_dep_chain_c(no deps, 1 entry for"1.0.0"):_own_total_changelog_count= 1 → same as beforetest_dep_chain_b(depends on C, 1 own entry): 1 + 1 = 2 → same as beforetest-fake-cudaas a dependency: sums entries for"12.9"+"12.8"= 2, increasing monotonically as new CUDA versions are addedYou'd also need a new test like:
_calculate_build_tagto delegate dependency contribution to a new version-agnostic helper_dependency_changelog_totaland_own_total_changelog_counthelpers in_pbi.pytest-fake-cuda🏁 Script executed:
Length of output: 242
Plan: Fix version-agnostic dependency changelog accumulation
_calculate_build_tagin_pbi.pyto separate own (versioned) count from dependency (version-agnostic) count_own_total_changelog_countrecursive helper for summing all versions of a deptest-fake-cudaFailed to handle agent chat message. Please try again.