feat: add support for parallel kind processing with threads #840
+121
−41
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
2 times, most recently
from
2a48d4e to
05e295a
Compare
Contributor
|
Msgspec does support freethreading in python: https://github.com/jcrist/msgspec/pull/877/files Initially thought we are planning to have both support for msgspec and voluptuous |
Contributor
Author
Oh, that's wonderful to hear about msgspec! Supporting both should be fine as far as I'm concerned for this work; it just means I'll need to rebase this after the schema work lands. |
|
msgspec support for 3.14 has been released https://github.com/jcrist/msgspec/releases/tag/0.20.0 |
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
5 times, most recently
from
4b13fb9 to
2afc838
Compare
This will make a subsequent commit that adds support for a ThreadPoolExecutor cleaner.
Voluptuous contains thread-unsafe code; work around this by locking before calling into it.
Even with 3.14 free-threaded python, this is still a bit slower than multiprocessing on Linux, but it will allow us to start experimenting with it more, and may allow users on macOS and Windows to immediately see a speed-up.
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
from
2afc838 to
f4b9a78
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds experimental, opt-in support for generating kinds in parallel across multiple threads. When used in conjunction with a free-threaded python it can be a significant speed-up in configuration with many expensive to generate kinds (eg: Gecko) where multiprocessing is not available.
In the short and medium term this should stay opt-in for a few reasons:
zstandard) are not available, which makes taskgraph more painful to install with this python.In the long term, I hope to get rid of multiprocessing support entirely once the above are no longer issues, because it imposes many limitations on how taskgraph generation can happen, how cached data can be shared, etc.
Ideally this PR would come with some multithreaded testing on Windows and macOS. Once #869 is done we ought to be able to add mac tests. Windows is trickier -- we don't have a compiler available on the workers, and zstandard (and probably other packages) don't have wheels available for free threaded python, which means we can't even install taskgraph in a free-threaded python virtualenv.
The code coverage failure is a bit misleading -- there's little in the way of new code here, and the new paths for the Linux side is covered with the multithread test job, which coverage isn't aware of.