-
Notifications
You must be signed in to change notification settings - Fork 10
Description
Issue
completion_config - Defines when the parallel operation completes:
Currently has:
CompletionConfig.all_successful() - Requires all branches to succeed (default)
CompletionConfig.first_successful() - Completes when any branch succeeds
CompletionConfig.all_completed() - Completes when branches finish; check started_count if completion criteria are met early
Custom configuration with specific success/failure thresholds
all_successful here means that once the first branch fails, the parallel execution completes (in your case all seem to complete before the failure branch terminates). The parallel operation (parent) successfully completes because it executed all branches (irrespective of their failure), conformed to your exit criteria and created a final parent checkpoint. Therefore, parallel is considered successful and you always need to check the batch result for successful/failed entries returned from those compound operations like parallel and map. Only if there’s something wrong with the parallel (parent) operation e.g., it couldn’t checkpoint, you’d see it entering a failed state.
Page/Location
Suggested Fix
Something like this for all_successful:
Completes when all branches succeed or immediately once a branch fails (default)
first_successful() - { minSuccessful: 1 }
Behavior:
- Stops starting new tasks when: successCount >= 1 (first success)
- Waits for already-running tasks to complete
- Tolerance: Unlimited failures allowed (no failure threshold set)
- Result: Completes when first item succeeds OR all items complete/fail
Completion reasons:
- If all complete:
"ALL_COMPLETED" - If first succeeds before all complete:
"MIN_SUCCESSFUL_REACHED"
all_successful() - { toleratedFailureCount: 0 }
Behavior:
- Stops starting new tasks when: failureCount > 0 (first failure)
- Waits for already-running tasks to complete
- Tolerance: Zero failures allowed (fail-fast)
- Result: Completes when first failure occurs OR all items succeed
Completion reasons:
- If any item fails:
"FAILURE_TOLERANCE_EXCEEDED" - If all items succeed:
"ALL_COMPLETED"
all_completed() - { toleratedFailureCount: None, toleratedFailurePercentage: None }
Behavior:
- Identical to all_successful() - stops starting new tasks on first failure
- Waits for already-running tasks to complete
- Tolerance: Zero failures allowed (fail-fast)
- Result: Completes when first failure occurs OR all items succeed
Completion reasons:
- If any item fails:
"FAILURE_TOLERANCE_EXCEEDED" - If all items succeed:
"ALL_COMPLETED"