Fix benchmark cache path and add forceBenchmarks pipeline variable#927
Open
RyAuld wants to merge 6 commits into
Open
Fix benchmark cache path and add forceBenchmarks pipeline variable#927RyAuld wants to merge 6 commits into
RyAuld wants to merge 6 commits into
Conversation
The Cache@2 ADO task requires a directory path, but .perf.baseline was a file. Move baseline storage into a .perf-baseline/ directory so the cache task can save/restore it correctly. - Change cache path to \/.perf-baseline - Update test_benchmark.py to store baseline in .perf-baseline/data - Ensure directory exists with mkdir -p before running benchmarks - Update .gitignore accordingly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the performance benchmark baseline caching to use a directory (compatible with Azure DevOps Cache@2) instead of a single baseline file, and aligns benchmark output storage under that directory.
Changes:
- Switch the benchmark baseline location to
.perf-baseline/dataso the cache target can be a directory. - Update the ADO cache path to
$(System.DefaultWorkingDirectory)/.perf-baselineand ensure the directory exists before running benchmarks. - Update
.gitignoreto ignore the new.perf-baseline/directory.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
tests/test_benchmark.py |
Moves baseline storage under .perf-baseline/ so it can be cached as a directory. |
azure-pipelines.yml |
Updates the Cache@2 path to a directory and creates it prior to running benchmarks. |
.gitignore |
Ignores the new .perf-baseline/ directory. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Allows benchmarks to run on non-dev branches by setting the pipeline variable forceBenchmarks=true when queuing a manual build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add os.makedirs in test_benchmark.py so local runs don't fail - Use absolute path in mkdir -p to align with Cache@2 path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RyAuld
changed the title
- Use __file__-based path resolution so baseline works from any CWD - Move stdlib imports before third-party imports - Update pipeline comment to reflect forceBenchmarks behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Python 3.14 intermittently stalls during test execution, consuming the full 30-min job timeout. Add a 5-minute timeoutInMinutes on the pytest step so stalls are caught quickly instead of wasting agent time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Require Build.Reason=Manual for forceBenchmarks override (case-insensitive) - Add 5-min timeoutInMinutes on pytest step to catch stalls early Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Build 1651184 fails with:
The
Cache@2ADO task requires a directory path, but.perf.baselinewas configured as a file.Additionally, Python 3.14 unit tests intermittently stall, consuming the full 30-minute job timeout before being canceled.
Changes
Fix benchmark cache path
$(System.DefaultWorkingDirectory)/.perf-baseline(a directory)test_benchmark.pyto store baseline data in.perf-baseline/data, anchored to repo root via__file__mkdir -p) and in test code (os.makedirs) so it works in CI and local runs.gitignoreto ignore the.perf-baseline/directoryAdd
forceBenchmarkspipeline variableforceBenchmarks=truewhen queuing a manual builddev)Add 5-minute step timeout to pytest
timeoutInMinutes: 5on the pytest step so stalls fail fast instead of consuming the full 30-min job timeoutValidation
Build 1651236 passed all stages including benchmarks (triggered with
forceBenchmarks=true).