experiment: use hyperlight main directly (no custom branch)#99
Open
danbugs wants to merge 3 commits into
Open
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Linux Benchmarks
Details
| Benchmark suite | Current: b6f7241 | Previous: 4460ae2 | Ratio |
|---|---|---|---|
hello_world (median) |
20 ms |
20 ms |
1 |
pandas (median) |
110 ms |
110 ms |
1 |
density (per VM) |
8 MB |
11 MB |
0.73 |
snapshot (disk) |
1755 MiB |
656 MiB |
2.68 |
This comment was automatically generated by workflow using github-action-benchmark.
Contributor
There was a problem hiding this comment.
Windows Benchmarks
Details
| Benchmark suite | Current: b6f7241 | Previous: 4460ae2 | Ratio |
|---|---|---|---|
hello_world (median) |
196 ms |
301 ms |
0.65 |
pandas (median) |
481 ms |
768 ms |
0.63 |
density (per VM) |
7 MB |
656 MB |
0.010670731707317074 |
snapshot (disk) |
1756 MiB |
664 MiB |
2.64 |
This comment was automatically generated by workflow using github-action-benchmark.
danbugs
force-pushed
the
experiment/use-hyperlight-main
branch
2 times, most recently
from
ea01088 to
6b2c81d
Compare
- Point hyperlight-host at upstream hyperlight-dev/hyperlight main (no custom branch dependencies) - Adapt snapshot persistence to upstream OCI format (directory-based) - Re-map initrd via map_file_cow after every restore() since file mappings are separate hypervisor memory regions not in the snapshot - Reduce pyhl heap from 2560 MiB to 1280 MiB (minimum for Python warmup with pandas+docx) - Remove sparsify_snapshot code (was file-format specific) - Remove memmap2 and extra windows-sys features (no longer needed) - Update snapshot path references from file to directory in CI, tests, and examples Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
force-pushed
the
experiment/use-hyperlight-main
branch
from
6b2c81d to
b6f7241
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates hyperlight-unikraft’s host tooling to work directly against upstream hyperlight-dev/hyperlight main (pinned to 69c362b6), primarily by migrating snapshot persistence/loading from a single snapshot.hls file to an OCI image-layout directory (snapshot/) and removing custom-branch-only features/APIs.
Changes:
- Switch
hyperlight-hostdependency to upstreamhyperlight-dev/hyperlight@69c362b6and drop custom-branch-only features/deps. - Migrate snapshot persistence and all references from
snapshot.hlsto OCI layout insnapshot/(save/load withOciTag). - Update restore behavior to re-map initrd after restore (replacing the old “preserve file mappings” restore path).
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| host/tests/pyhl_runtime.rs | Updates test install probing from snapshot.hls to snapshot/. |
| host/src/pyhl.rs | Updates library install/runtime flow to use OCI snapshot directory and upstream snapshot APIs. |
| host/src/lib.rs | Adapts snapshot save/load to upstream OCI format; removes preserving-restore path and snapshot sparsification. |
| host/src/bin/pyhl.rs | Updates CLI setup/run to use snapshot/ directory layout and upstream APIs. |
| host/src/bin/pydriver_run.rs | Updates heap sizing consistent with other host entrypoints. |
| host/examples/test_cpiovfs.rs | Updates snapshot save path to directory form and simplifies output. |
| host/examples/pyhl_as_library.rs | Updates example docs to refer to .pyhl/snapshot/. |
| host/Cargo.toml | Points hyperlight-host to upstream repo/rev; drops no-longer-needed deps/features. |
| host/Cargo.lock | Lockfile refresh consistent with upstream dependency graph changes. |
| .github/workflows/benchmarks.yml | Updates snapshot size reporting to handle snapshot/ directory on Linux/Windows. |
Comments suppressed due to low confidence (1)
host/src/bin/pyhl.rs:378
- The setup early-return condition only checks that
snapshot/is a directory. A partially-written/empty snapshot dir (e.g., interrupted save) would incorrectly be treated as “installed” unless--force, leavingpyhl runbroken. Consider checking for an OCI-layout marker (e.g.,snapshot/index.json) and, when proceeding with setup, deleting any existing snapshot path (file or dir) before saving a new snapshot to avoid accumulating stale blobs across repeated--forceruns.
if image_installed(&home) && dst_snapshot.is_dir() && !args.force {
eprintln!(
"pyhl: image already installed at {} (use --force to overwrite)",
home.display()
);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+486
to
+489
| $snap = ".pyhl\snapshot" | ||
| $files = Get-ChildItem -Path $snap -Recurse -File | ||
| $apparentMiB = [math]::Round(($files | Measure-Object -Property Length -Sum).Sum / 1MB) | ||
| $diskMiB = $apparentMiB |
Comment on lines
+149
to
153
| let dst_snapshot = home.join(SNAPSHOT_DIR); | ||
| let dst_version = home.join(VERSION_FILE); | ||
|
|
||
| let already = dst_kernel.is_file() && dst_initrd.is_file() && dst_snapshot.is_file(); | ||
| let already = dst_kernel.is_file() && dst_initrd.is_file() && dst_snapshot.is_dir(); | ||
| if already && !opts.force { |
Comment on lines
+40
to
41
| if workspace.join("snapshot").is_dir() { | ||
| return Some(workspace); |
Comment on lines
+270
to
272
| let snap = home.join(SNAPSHOT_DIR); | ||
| if !snap.is_dir() { | ||
| bail!( |
Comment on lines
+529
to
533
| let snapshot = home.join(SNAPSHOT_DIR); | ||
|
|
||
| // Fast path: `pyhl setup` already warmed up a sandbox, ran | ||
| // Py_Initialize + preloaded modules, captured the state, and | ||
| // persisted it to snapshot.hls. Here we mmap that file back and | ||
| // instantiate a sandbox directly — no kernel boot, no Python init. | ||
| if !snapshot.is_file() { | ||
| if !snapshot.is_dir() { | ||
| return Err(anyhow!( | ||
| "no warmed-up snapshot at {}.\n\ |
Signed-off-by: danbugs <danilochiarlone@gmail.com>
Expose the runtime HYPERLIGHT_MAX_SURROGATES env var through: - configure_surrogates() in pyhl module - --max-surrogates CLI flag on hyperlight-unikraft, pyhl setup, pyhl run - max_surrogates field on InstallOptions - 5th parameter on Runtime::new() Passing 0 disables surrogate processes on Windows (VirtualAlloc + WHvMapGpaRange path), replacing the compile-time whp-no-surrogate feature gate. Signed-off-by: danbugs <danilochiarlone@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Experiment to see if hyperlight-unikraft can work with upstream
hyperlight-dev/hyperlightmain directly, without thehyperlight-unikraftbranch.hyperlight-dev/hyperlight@69c362b6(current main HEAD)whp-no-surrogatefeature (not in upstream)restore_preserving_file_mappingsusage (not in upstream)file_mapping_pathfield (dead without preserving restores)save/loadwithOciTag)sparsify_snapshot(not applicable with OCI layout)snapshot.hlsreferences tosnapshot/directoryTest plan