Persist snapshot to disk

[ludfjig]

persist sandbox snapshots to disk

Adds Snapshot::to_file / Snapshot::from_file and MultiUseSandbox::from_snapshot, so a sandbox can be reconstructed from a saved snapshot without going through UninitializedSandbox::evolve(). Works in-process and across processes.

public API

let snapshot = sandbox.snapshot()?;
snapshot.to_file("guest.hls")?;

let snapshot = Arc::new(Snapshot::from_file("guest.hls")?);
let cfg: Option<SandboxConfiguration> = None;
let mut sandbox = MultiUseSandbox::from_snapshot(snapshot, HostFunctions::default(), cfg)?;

HostFunctions must be a superset (by name and signature) of those registered when the snapshot was taken. The optional SandboxConfiguration overrides runtime fields (interrupt knobs, guest_core_dump, guest_debug_info). Layout fields (input_data_size, output_data_size, heap_size, scratch_size) are always taken from the snapshot.

from_file_unchecked is provided for trusted environments. It still verifies the header hash but skips the memory blob hash, making large snapshots load in constant time.

file format

+-----------------+ 0
| RawPreamble     |  magic "HLS\0" + format_version
+-----------------+
| RawHeaderV1     |  arch, abi_version, layout, entrypoint, sregs flag,
|                 |  hypervisor, memory_size, memory_offset, host_funcs_size
+-----------------+
| RawSregs        |  8 segment regs, 2 table regs, 7 control regs,
|                 |  4-element interrupt bitmap. Always written.
+-----------------+
| RawHashes       |  header_hash (always verified) + blob_hash
+-----------------+
| host functions  |  flatbuffer of names + signatures (variable, optional)
+-----------------+ <- memory_offset (page-aligned)
| memory blob     |  memory_size bytes, mmap target
+-----------------+
| trailing guard  |  PAGE_SIZE of zeros (Windows guard backing)
+-----------------+

All header structs are #[repr(C)] POD types deriving bytemuck::Pod + Zeroable, so the byte layout is whatever those structs declare and there are no separate offsets to keep in sync.

The trailing PAGE_SIZE padding exists because Windows read-only file mappings cannot extend beyond the file's actual size, so the file must contain bytes for the trailing guard. Linux ignores the padding (its guard pages come from an anonymous mmap reservation).

versioning and portability tags

• RawPreamble.format_version. Wire format of the file. Bumped when header byte layout or section ordering changes. May be convertible across versions.
• RawHeaderV1.abi_version. ABI of the memory blob contents. Bumped independently when in-guest data layouts change. A mismatch means the snapshot must be regenerated from the guest binary.
• arch tag distinguishes guest arch (x86_64, aarch64, i686), so an i686-guest snapshot cannot be loaded into an amd64-guest build.
• hypervisor tag distinguishes KVM, MSHV, and WHP. Segment register hidden-cache fields (unusable, type_, granularity, db) differ between hypervisors for the same architectural state, so cross-hypervisor loads are rejected by default.

what is and is not persisted

Persisted: the snapshot region (guest code, PEB, heap, init data, page tables), all sregs, and the names + signatures of host functions registered at snapshot time.

Not persisted:

• sandbox_id. Process-local counter, fresh ID assigned on load (see "sandbox identity and restore" below).
• LoadInfo. Debug-only, reconstructible from ELF if needed (see "known limitations").
• regions. Always empty after snapshot construction (mapped-region contents are absorbed into the memory blob, see "known limitations").

The scratch region is recreated fresh on load and re-initialised by copying page tables from snapshot to scratch and writing I/O buffer metadata.

The vCPU special registers are persisted because the guest init code installs a GDT, IDT, TSS, and segment descriptors that differ from the standard 64-bit defaults.

A header_hash over preamble || header || sregs || host_funcs is always verified, even by from_file_unchecked. A separate blob_hash over the memory blob is verified by from_file and skipped by from_file_unchecked. Because blob_hash is itself one of the bytes covered by header_hash, swapping a memory blob without rewriting the header invalidates the always-checked header hash.

gdb / crashdump

• guest_debug_info and guest_core_dump fields of SandboxConfiguration are honored by from_snapshot, so gdb and core dumps work after loading from disk.
• For gdb, HyperlightVm::new installs a one-shot entry breakpoint for both Initialise and Call snapshots so the gdb stub event loop enters on the first vCPU run regardless of how the sandbox was constructed. The breakpoint is removed on first hit by the run loop.

sandbox identity and restore

MultiUseSandbox::restore requires the supplied snapshot to share the sandbox's sandbox_id so it can reuse the underlying memory layout safely. Ids are process-local atomic counters and are not persisted to disk. Every call to Snapshot::from_file assigns a fresh id, and from_snapshot copies that id onto the resulting sandbox.

So sandboxes built from clones of the same in-memory Arc<Snapshot> are mutually restore-compatible, while sandboxes from independent from_file calls (even of the same path) are not.

performance (Linux/KVM)

End-to-end wall-clock from zero state to a completed Echo guest call.

Heap size	evolve	snapshot (check-hash)	snapshot (ignore-hash)	speedup
128 KB (default)	3.80 ms	2.73 ms	2.78 ms	1.4x
8 MB	8.89 ms	5.18 ms	2.96 ms	3.0x
64 MB	26.7 ms	19.9 ms	2.92 ms	9.1x
256 MB	88.3 ms	57.4 ms	3.33 ms	26.5x

known limitations

• core dumps from a from_snapshot sandbox lack binary_path and AT_ENTRY for Call snapshots, and mem_profile lacks accurate traces. The file format would need extending to fix these.
• max_guest_log_level is not plumbed through from_snapshot. It is also intrinsically ineffective for Call snapshots, this should be addressed orthogonally to this PR.
• guest-counter does not work on from_snapshot-built sandboxes.
• Mapped-region metadata is not preserved (region contents are absorbed into the memory blob, but snapshot.regions() is empty after load).
• to_file overwrites the target path non-atomically. A crash mid-write can leave a partially written file. Concurrent writers to the same path are not serialised.
• The file backing a loaded Snapshot (and any MultiUseSandbox built from it) must not be modified, truncated, renamed over, or deleted for the lifetime of the snapshot. On Linux this is unenforced. On Windows the OS refuses the operation with ERROR_USER_MAPPED_FILE (1224). (firecracker does this as well, see here

The memory file (pointed by backend_path when using File backend type, or pointed by mem_file_path) must be considered immutable from Firecracker and host point of view. It backs the guest OS memory for read access through the page cache. External modification to this file corrupts the guest memory and leads to undefined behavior.

future work

Typed error variants. Fuzz target for from_file. CoW overlay layers. Cross-hypervisor portability via sregs normalisation. Huge page support (MAP_HUGETLB). Atomic to_file via temp + rename + fsync. OCI distribution.

[Copilot]

Pull request overview

Adds first-class support for persisting Snapshots to disk and rehydrating sandboxes from those snapshots to avoid ELF parsing / guest init on cold start, using file-backed mappings for (near) zero-copy loads.

Changes:

• Implement Snapshot::to_file(), Snapshot::from_file(), and Snapshot::from_file_unchecked() with a versioned on-disk header + mmappable memory blob.
• Add MultiUseSandbox::from_snapshot() fast-path for sandbox creation directly from an in-memory or disk-loaded snapshot.
• Introduce file-backed ReadonlySharedMemory::from_file() and refactor SandboxMemoryLayout to expose enough stable layout fields for serialization.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Updates layout API usage (peb_address() accessor).
src/hyperlight_host/src/sandbox/uninitialized.rs	Ensures snapshot-based sandbox creation registers default HostPrint via FunctionRegistry::with_default_host_print().
src/hyperlight_host/src/sandbox/snapshot.rs	Defines snapshot file format + (de)serialization, file load/save APIs, sregs I/O, and extensive tests.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Adds MultiUseSandbox::from_snapshot() instantiation path.
src/hyperlight_host/src/sandbox/host_funcs.rs	Adds FunctionRegistry::with_default_host_print() helper and makes default writer private.
src/hyperlight_host/src/mem/shared_mem.rs	Adds ReadonlySharedMemory::from_file() for file-backed snapshot memory mappings.
src/hyperlight_host/src/mem/mgr.rs	Updates layout field access for scratch I/O buffer sizes.
src/hyperlight_host/src/mem/memory_region.rs	Routes Snapshot regions through the Windows surrogate “ReadOnlyFile” mapping path.
src/hyperlight_host/src/mem/layout.rs	Refactors SandboxMemoryLayout to store key sizes directly and computes offsets via methods; updates PEB writing.
src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs	Factors out apply_sregs() and updates peb_address() usage.
src/hyperlight_host/benches/benchmarks.rs	Adds benchmarks for snapshot file save/load and cold start via snapshot.
src/hyperlight_common/src/mem.rs	Adds write_to() helpers for GuestMemoryRegion and HyperlightPEB.
docs/snapshot-file-implementation-plan.md	Adds a detailed design/format plan and future work notes.

[andreiltd]

Looks good! I think the biggest issue, that have to be addressed here, is to NOT trust the parsed from binary header values and validate them before transforming it to the layout.

I would also recommend using winnow to improve mechanics and safety of parsing.

[simongdavies]

This is great work @ludfjig !!

I've left a couple of bits of feedback but my biggest comments are:

Discovery

There should be a way for a host to inspect a hls file , so it can discover what host functions it requires, what hypervisor it needs etc.

Something like

let snapshot_info = Snapshot::inspect-file("some.hls")

Process Wide Snapshot Cache

We should do the validation of the snapshot once and cache it (and remove the unchecked load). This way we pay the price for the validation once per process, this is likely the happy path for usage and should give the advantages of unchecked whilst retaining integrity checks.

static SNAPSHOT_CACHE: OnceLock<Mutex<HashMap<[u8; 32], Weak<Snapshot>>>> = OnceLock::new();

 pub fn from_file_cached(
        path: impl AsRef<std::path::Path>,
    ) -> crate::Result<Arc<Self>> {
        let cache = SNAPSHOT_CACHE.get_or_init(|| Mutex::new(HashMap::new()));

        // Read just enough to extract the blob_hash (cache key).
        // This is ~1.2 KiB — one syscall, no mmap.
        let declared_hash = Self::read_blob_hash(path.as_ref())?;

        // Fast path: cache hit.
        {
            let map = cache.lock().map_err(|_| {
                crate::new_error!("snapshot cache mutex poisoned")
            })?;
            if let Some(weak) = map.get(&declared_hash) {
                if let Some(arc) = weak.upgrade() {
                    return Ok(arc);
                }
            }
        }

        // Slow path: full load with hash verification.
        let snapshot = Arc::new(Self::from_file(path)?);

        // Double-check: another thread may have loaded the same
        // file while we were doing the full load. Prefer theirs
        // to avoid duplicate mmaps.
        {
            let mut map = cache.lock().map_err(|_| {
                crate::new_error!("snapshot cache mutex poisoned")
            })?;
            if let Some(weak) = map.get(&snapshot.hash) {
                if let Some(existing) = weak.upgrade() {
                    // Someone else loaded it first — use theirs,
                    // our load is dropped.
                    return Ok(existing);
                }
            }
            map.insert(snapshot.hash, Arc::downgrade(&snapshot));
            // Opportunistic cleanup of expired entries.
            map.retain(|_, v| v.strong_count() > 0);
        }

        Ok(snapshot)
    }

Last thing can we break this up into smaller PRs please, will make it easier to do a more thorough review of the changes. Thanks !

[simongdavies]

the comment above says that the blob_hash is part of the header_hash but it doesn't seem to be included in the data hashed for the header_hash ?

[simongdavies]

Is this correct, shouldn't it be an error if there are no sregs when NextAction::Call since VM should already be initialised and the sregs should have been captured?

[simongdavies]

I think its a bit fragile to hope that someone will increment this if there are any breaking changes , I think a combination of tests and const asserts would help massively, we should be able to get pretty much full coverage such that the asserts/tests fail forcing the developer to update the tests/asserts/ABI version.

[simongdavies]

should we be checking the length of these before truncating (or storing the correct sizes?) this is done for init_data_permissions on line 448?

[ludfjig]

This is great work @ludfjig !!

I've left a couple of bits of feedback but my biggest comments are:

Discovery

There should be a way for a host to inspect a hls file , so it can discover what host functions it requires, what hypervisor it needs etc.

Something like

let snapshot_info = Snapshot::inspect-file("some.hls")

Process Wide Snapshot Cache

We should do the validation of the snapshot once and cache it (and remove the unchecked load). This way we pay the price for the validation once per process, this is likely the happy path for usage and should give the advantages of unchecked whilst retaining integrity checks.

static SNAPSHOT_CACHE: OnceLock<Mutex<HashMap<[u8; 32], Weak<Snapshot>>>> = OnceLock::new();

 pub fn from_file_cached(
        path: impl AsRef<std::path::Path>,
    ) -> crate::Result<Arc<Self>> {
        let cache = SNAPSHOT_CACHE.get_or_init(|| Mutex::new(HashMap::new()));

        // Read just enough to extract the blob_hash (cache key).
        // This is ~1.2 KiB — one syscall, no mmap.
        let declared_hash = Self::read_blob_hash(path.as_ref())?;

        // Fast path: cache hit.
        {
            let map = cache.lock().map_err(|_| {
                crate::new_error!("snapshot cache mutex poisoned")
            })?;
            if let Some(weak) = map.get(&declared_hash) {
                if let Some(arc) = weak.upgrade() {
                    return Ok(arc);
                }
            }
        }

        // Slow path: full load with hash verification.
        let snapshot = Arc::new(Self::from_file(path)?);

        // Double-check: another thread may have loaded the same
        // file while we were doing the full load. Prefer theirs
        // to avoid duplicate mmaps.
        {
            let mut map = cache.lock().map_err(|_| {
                crate::new_error!("snapshot cache mutex poisoned")
            })?;
            if let Some(weak) = map.get(&snapshot.hash) {
                if let Some(existing) = weak.upgrade() {
                    // Someone else loaded it first — use theirs,
                    // our load is dropped.
                    return Ok(existing);
                }
            }
            map.insert(snapshot.hash, Arc::downgrade(&snapshot));
            // Opportunistic cleanup of expired entries.
            map.retain(|_, v| v.strong_count() > 0);
        }

        Ok(snapshot)
    }

Last thing can we break this up into smaller PRs please, will make it easier to do a more thorough review of the changes. Thanks !

I agree we should let users inspect a snapshot, and maybe even provide a caching mechanism like you propose (although on first glance my impression is that let users do this themselves), but maybe those could be follow ups as there's definitely some more design churn required for them?

Last thing can we break this up into smaller PRs please, will make it easier to do a more thorough review of the changes. Thanks !

I have 15 pretty atomic and separate commits that should aid review, do you feel strongly about separate prs still? More than half of the entire diff is just tests too

[simongdavies]

I agree we should let users inspect a snapshot, and maybe even provide a caching mechanism like you propose (although on first glance my impression is that let users do this themselves), but maybe those could be follow ups as there's definitely some more design churn required for them?

I actually think both these things are pretty important to the fundamental use case for this feature, as a user without inspect there is no way for me to know what host_funcs I should be providing, so I load a snapshot and I don't provide the host functions it expects then there is an error message telling me that but with the inspect API I can detect this and not even try and load the file, more so with the hypervisor tag , its trial and error which file to load, with this API a host could enumerate available snapshots and only attempt to load ones which should work.

The caching mechanism again is I think something we should offer as a core part of the API:

It allows is to get rid of the unchecked version of the API , we shouldn't have this API, we should verify and validate snapshots always, anything else is just opening a path to hard to debug/diagnose issues. It seems the only reason we have this API is to avoid the cost of the hash verification, having the cache allows us to amortize the cost of the verification across every sandbox created with that snapshot file across the life of the process. This is the primary use case for snapshot persistence.

It uses less resources (most of this arguably is trivial but everything helps, for example if we have N sandboxes created from one snapshot file it use 1xSnapshotSize Virtual Memory vs NxSnapshotSize Virtual Memory)

The cache also helps in some of the snapshot file modification scenarios , for example if the snapshot files was modified on disk after the first load (assumming headers were still intact) the subsequent snapshot loads would be consistent.

[simongdavies]

I have 15 pretty atomic and separate commits that should aid review, do you feel strongly about separate prs still? More than half of the entire diff is just tests too

Agreed the commits make it easier, but as a reviewer I have to review the entire PR in one session, splitting it up means , I can do it in smaller digestible chunks. I think this would be a great test of GH PR stacks, I just asked for access to the preview, if we get it we should try using it on this PR I think.