Description
This supersedes #16809 - a lot of underlying work has been done on build process across the bindings since that issue was created to support a slightly better vision.
Problem
- Github Actions runs are taking too long, especially when multiple small PRs are being made.
- ci.yml workflow has significant overlap with ci-rbe.yml workflow
- There are things that can't be tested in RBE that aren't being tested on a GHA Runner
- Tests are inconsistent across bindings
Proposal
RBE remains the primary per-PR test gate. GHA Runners should only run tests that provide sufficient additional signal.
GHA runners should be scoped by affected Bazel targets (this is wired up but not used yet).
GHA Tests
Update: I think we move all browser tests that aren't se-manager or os-sensitive to manual/scheduled; Same with unit test matrix; we should trust that RBE will find enough of the unit test issues that it isn't worth rerunning every combination on every PR.
Run every time there are affected targets:
- Windows: Affected targets list filtered for
Edge + se_manager + os_sensitive -safari -ie (large & small tests)
- Mac: Affected targets list filtered for
Safari + se_manager + os_sensitive -ie (large & small tests)
- Linux: Affected targets list filtered for se_manager + rbe_skipped
-safari -ie
Run when tagged with [language]
4. Unit Test Matrix: first/last version and alternate interpreters as applicable
Run manually and/or scheduled:
- Additional unit tests for alternate interpreters or other versions
- Updated: Windows: chrome-beta, firefox-beta, edge (local)
- Updated: Mac: safari (local)
Additional/Future Considerations
- consider running periodically all targets on windows not filtering to just edge as a stopgap
Ideally we can replace .skipped-tests with bazel tags or test runner guards/annotations Update: Done
fix Java to use target_compatible_with instead of generating targets for everything and filtering Update: Done
- consider splitting skip-rbe tag into a reason for skipping
- consider running scheduled ci.yml workflow 1x a day instead of 2x a day. We've averaged 4 commits a day on trunk for the past year; 90% of days are < 9 commits. We run this much more frequently than we evaluate it.
Description
This supersedes #16809 - a lot of underlying work has been done on build process across the bindings since that issue was created to support a slightly better vision.
Problem
Proposal
RBE remains the primary per-PR test gate. GHA Runners should only run tests that provide sufficient additional signal.
GHA runners should be scoped by affected Bazel targets (this is wired up but not used yet).
GHA Tests
Update: I think we move all browser tests that aren't se-manager or os-sensitive to manual/scheduled; Same with unit test matrix; we should trust that RBE will find enough of the unit test issues that it isn't worth rerunning every combination on every PR.
Run every time there are affected targets:
Edge+ se_manager + os_sensitive-safari -ie(large & small tests)Safari+ se_manager + os_sensitive-ie(large & small tests)-safari -ieRun when tagged with [language]4. Unit Test Matrix: first/last version and alternate interpreters as applicableRun manually and/or scheduled:
Additional/Future Considerations
Ideally we can replace .skipped-tests with bazel tags or test runner guards/annotationsUpdate: Donefix Java to use target_compatible_with instead of generating targets for everything and filteringUpdate: Done