Minimize task creation for concurrent payload visiting#1414
Open
jmaeagle99 wants to merge 1 commit intotemporalio:mainfrom
Open
Minimize task creation for concurrent payload visiting#1414jmaeagle99 wants to merge 1 commit intotemporalio:mainfrom
jmaeagle99 wants to merge 1 commit intotemporalio:mainfrom
Conversation
jmaeagle99
force-pushed
the
better-concurrent-visitor
branch
from
47b6f79 to
4b4e741
Compare
jmaeagle99
force-pushed
the
better-concurrent-visitor
branch
from
4b4e741 to
ecee01a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
Update the payload visitor to create async tasks only when there is semaphore availability.
Why?
No need to spawn all async tasks for payload visitor. Task creation is minimized to not use up so many resources simultaneously.
Review Suggestion
This change looks substantial on its own. However, if you compare it to the state of the visitor before concurrent visiting was introduced in 3788785, you'll see it's much simpler. It effectively undoes the mass coroutine creation in that change and only creates async coroutines when visiting the payloads if there is semaphore availability. Here's a link to compare the current changes to just before concurrent visiting was introduced: jmaeagle99/temporalio-sdk-python@718266d...better-concurrent-visitor#diff-719b6ee0e76a1c3bf26e1e3b922da9fec696b6a4d93e8e9e8138cb86bf76bfe9
Checklist