Skip to content

feat: pass partial_archive_url to evaluation workflow for resume support#2540

Merged
juanmichelini merged 2 commits intomainfrom
openhands/resume-partial-runs-from-archive
Mar 26, 2026
Merged

feat: pass partial_archive_url to evaluation workflow for resume support#2540
juanmichelini merged 2 commits intomainfrom
openhands/resume-partial-runs-from-archive

Conversation

@juanmichelini
Copy link
Copy Markdown
Collaborator

@juanmichelini juanmichelini commented Mar 20, 2026

Summary

Adds the partial_archive_url input to run-eval.yml so it can be forwarded to the evaluation repo's eval-job.yml workflow. This enables resuming evaluation jobs from a previous run's archive tar.gz.

Companion PR: OpenHands/evaluation#326

Changes to .github/workflows/run-eval.yml

  1. New partial_archive_url workflow_dispatch input
  2. Printed in the print-parameters step
  3. Passed as PARTIAL_ARCHIVE_URL env var to the dispatch step
  4. Included in the dispatch payload forwarded to the evaluation repo

How to test

  1. Merge OpenHands/evaluation#326 first (or use eval_branch pointing to that PR's branch)
  2. Trigger "Run Eval" workflow dispatch with:
    • eval_branch: openhands/resume-partial-runs-from-archive (points to the evaluation PR branch)
    • partial_archive_url: URL of a previous run's results.tar.gz
  3. The evaluation job should download the archive, extract it, and resume from existing output

Checklist

  • If the PR is changing/adding functionality, are there tests to reflect this?
    • Tests are in the companion PR (OpenHands/evaluation#326) — this PR is workflow config only
  • If there is an example, have you run the example to make sure that it works?
  • If there are instructions on how to run the code, have you followed the instructions and made sure that it works?
  • If the feature is significant enough to require documentation, is there a PR open on the OpenHands/docs repository with the same branch name?
  • Is the github CI passing?

Agent Server images for this PR

GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python amd64, arm64 nikolaik/python-nodejs:python3.13-nodejs22 Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:0365297-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-0365297-python \
  ghcr.io/openhands/agent-server:0365297-python

All tags pushed for this build

ghcr.io/openhands/agent-server:0365297-golang-amd64
ghcr.io/openhands/agent-server:0365297-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:0365297-golang-arm64
ghcr.io/openhands/agent-server:0365297-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:0365297-java-amd64
ghcr.io/openhands/agent-server:0365297-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:0365297-java-arm64
ghcr.io/openhands/agent-server:0365297-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:0365297-python-amd64
ghcr.io/openhands/agent-server:0365297-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-amd64
ghcr.io/openhands/agent-server:0365297-python-arm64
ghcr.io/openhands/agent-server:0365297-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-arm64
ghcr.io/openhands/agent-server:0365297-golang
ghcr.io/openhands/agent-server:0365297-java
ghcr.io/openhands/agent-server:0365297-python

About Multi-Architecture Support

  • Each variant tag (e.g., 0365297-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 0365297-python-amd64) are also available if needed

Add experimental partial_archive_url input to run-eval.yml that gets
forwarded to the evaluation repo's eval-job workflow. This enables
resuming evaluation jobs from a previous run's archive tar.gz.

Companion to OpenHands/evaluation#326

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 20, 2026

Python API breakage checks — ✅ PASSED

Result:PASSED

Action log

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 20, 2026

REST API breakage checks (OpenAPI) — ✅ PASSED

Result:PASSED

Action log

@juanmichelini juanmichelini changed the title DRAFT: feat: pass partial_archive_url to evaluation workflow for resume support feat: pass partial_archive_url to evaluation workflow for resume support Mar 20, 2026
@juanmichelini juanmichelini marked this pull request as ready for review March 23, 2026 19:45
Copy link
Copy Markdown
Collaborator

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Good taste - Clean parameter pass-through. Simple, solves a real problem (resuming expensive evals), no breaking changes. LGTM.

@juanmichelini juanmichelini marked this pull request as draft March 23, 2026 20:04
@juanmichelini juanmichelini marked this pull request as ready for review March 26, 2026 15:12
@juanmichelini juanmichelini merged commit 0f92b6e into main Mar 26, 2026
37 checks passed
@juanmichelini juanmichelini deleted the openhands/resume-partial-runs-from-archive branch March 26, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants