Skip to content

refactor(engine): pass Python worker startup arguments by name#5597

Open
yangzhang75 wants to merge 1 commit into
apache:mainfrom
yangzhang75:refactor/python-worker-named-startup-args
Open

refactor(engine): pass Python worker startup arguments by name#5597
yangzhang75 wants to merge 1 commit into
apache:mainfrom
yangzhang75:refactor/python-worker-named-startup-args

Conversation

@yangzhang75

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Passes Python worker startup configuration by name instead of by argv position, as proposed in the issue.

PythonWorkflowWorker (JVM) previously built ~19 positional command-line arguments, and texera_run_python_worker.py unpacked them positionally. Because the two sides agreed only by index, adding/removing/reordering one argument could silently misassign values.

  • PythonWorkflowWorker.scala: build a single JSON object of named startup-config keys and pass it as one argument.
  • texera_run_python_worker.py: parse that JSON and read each value by key; a missing or renamed key now raises a clear KeyError instead of silently misaligning.

The set of keys written on the Scala side and read on the Python side is identical (19 keys). No behavior change otherwise.

Any related issues, documentation, discussions?

Closes #5547

How was this PR tested?

  • Verified the JSON key set written by PythonWorkflowWorker.scala exactly matches the keys read in texera_run_python_worker.py (19 keys, no drift).
  • WorkflowExecutionService/compile (amber) succeeds.
  • scalafmtCheckAll passes; scalafix rules (RemoveUnused, ProcedureSyntax) are satisfied (the new import is used).
  • Python entry script passes py_compile.
  • End-to-end worker launch is exercised by the existing CI integration jobs (amber-integration), which start a real Python worker.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

@codecov-commenter

codecov-commenter commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.36%. Comparing base (07ca5d4) to head (1e6d0ca).

Files with missing lines Patch % Lines
...chitecture/pythonworker/PythonWorkflowWorker.scala 0.00% 25 Missing ⚠️
amber/src/main/python/texera_run_python_worker.py 0.00% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5597      +/-   ##
============================================
- Coverage     52.38%   52.36%   -0.03%     
+ Complexity     2484     2481       -3     
============================================
  Files          1070     1070              
  Lines         41359    41367       +8     
  Branches       4441     4441              
============================================
- Hits          21666    21661       -5     
- Misses        18427    18438      +11     
- Partials       1266     1268       +2     
Flag Coverage Δ *Carryforward flag
access-control-service 64.61% <ø> (ø) Carriedforward from 07ca5d4
agent-service 33.76% <ø> (ø) Carriedforward from 07ca5d4
amber 53.26% <0.00%> (-0.06%) ⬇️
computing-unit-managing-service 1.65% <ø> (ø) Carriedforward from 07ca5d4
config-service 56.06% <ø> (ø) Carriedforward from 07ca5d4
file-service 38.21% <ø> (ø) Carriedforward from 07ca5d4
frontend 46.91% <ø> (ø) Carriedforward from 07ca5d4
pyamber 90.62% <0.00%> (-0.10%) ⬇️
python 90.80% <ø> (+0.04%) ⬆️ Carriedforward from 07ca5d4
workflow-compiling-service 58.69% <ø> (ø) Carriedforward from 07ca5d4

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xuang7

xuang7 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Could you assign a reviewer for this PR? @chenlica

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pass Python worker startup arguments by name instead of by position

3 participants