petri: add ability to mark vmm test variants as unstable

[tjones60]

Adds the ability to mark individual vmm test variants as unstable. Includes changes to petri and the vmm tests macros to make this possible.

[jstarks]

Needs guide updates.

[Copilot]

Pull request overview

Adds first-class support for marking individual Petri/VMM test variants as unstable (non-gating), moving away from name-suffix heuristics and enabling per-variant configuration via the VMM test macros.

Changes:

• Extend Petri’s test model to track “unstable” status per test and suppress unstable failures in libtest-mimic while still logging them distinctly.
• Enhance vmm_test_macros to support unstable_ per-variant prefixes and introduce #[vmm_test_with(...)] override syntax (e.g. noagent(...)).
• Update VMM tests, TMK tests, and the GitHub workflow to emit and report unstable-failure markers (petri.failed_unstable).

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
vmm_tests/vmm_tests/tests/tests/x86_64/storage.rs	Converts a Hyper-V-only test to use the unified vmm_test path and generic backend.
vmm_tests/vmm_tests/tests/tests/x86_64/openhcl_uefi.rs	Replaces *_no_agent macros with vmm_test_with(noagent(...)) usage.
vmm_tests/vmm_tests/tests/tests/x86_64.rs	Switches no-agent variants to vmm_test_with override form.
vmm_tests/vmm_tests/tests/tests/multiarch/vmgs.rs	Migrates to vmm_test_with(noagent(...)) for no-agent test variants.
vmm_tests/vmm_tests/tests/tests/multiarch/tpm.rs	Marks a specific OpenVMM OpenHCL UEFI x64 VBS variant as unstable via unstable_ prefix.
vmm_tests/vmm_tests/tests/tests/multiarch.rs	Migrates macros and marks selected variants as unstable via unstable_ prefixes.
vmm_tests/vmm_test_macros/src/lib.rs	Implements unstable handling + override parsing (vmm_test_with) and removes hyperv_test.
vmm_tests/prep_steps/src/main.rs	Updates call site for new log_test_result(..., unstable) signature.
tmk/tmk_tests/src/lib.rs	Switches to petri::unstable_test! for a non-gating TMK test.
petri/src/tracing.rs	Adds unstable-specific result marker/logging (petri.failed_unstable) and new API param.
petri/src/test.rs	Adds unstable flag to RunTest/SimpleTest, new unstable_test! macro, and new suppression logic.
.github/workflows/upload-petri-results.yml	Counts and reports unstable failures separately in PR comments.

Comments suppressed due to low confidence (2)

petri/src/tracing.rs:154

• Introducing a new failure marker file (petri.failed_unstable) may require updating other repo tooling that scans for failures. For example, repo_support/investigate_ci.py currently only searches for petri.failed, so unstable failures won’t show up in its summary unless it’s updated to also consider petri.failed_unstable.

            Err(err) if unstable => {
                tracing::warn!(
                    error = err.as_ref() as &dyn std::error::Error,
                    "unstable test failed"
                );
                "petri.failed_unstable"
            }
            Err(err) => {
                tracing::error!(
                    error = err.as_ref() as &dyn std::error::Error,
                    "test failed"
                );
                "petri.failed"
            }

petri/src/tracing.rs:159

• log_test_result writes the marker file with unwrap(), which can panic on ordinary I/O failures (permissions, disk full, etc.) and prevent results from being recorded. Prefer handling the error (e.g., log an error and continue) instead of panicking.

    /// Traces and logs the result of a test run in the format expected by our tooling.
    pub fn log_test_result(&self, name: &str, r: &anyhow::Result<()>, unstable: bool) {
        let result_path = match &r {
            Ok(()) => {
                tracing::info!("test passed");
                "petri.passed"
            }
            Err(err) if unstable => {
                tracing::warn!(
                    error = err.as_ref() as &dyn std::error::Error,
                    "unstable test failed"
                );
                "petri.failed_unstable"
            }
            Err(err) => {
                tracing::error!(
                    error = err.as_ref() as &dyn std::error::Error,
                    "test failed"
                );
                "petri.failed"
            }
        };
        // Write a file to the output directory to indicate whether the test
        // passed, for easy scanning via tools.
        fs_err::write(self.0.root_path.join(result_path), name).unwrap();
    }

[jstarks]

Can we have a way (env var?) to optionally count unstable test failures as failures? This seems important for doing stress test loops locally, for example.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

[github-actions]

At least one Petri test failed.

[jstarks]

Do we need to update logview to cope with this new file?

[tjones60]

Log view will just show all failures as failures for now, we can differentiate in the future if we want.

[tjones60]

You can see one of the flaky tests actually failed on this PR, but the github actions passed despite the tests appearing as failed on the test results website, as expected.

[tjones60]

This is because failures are identified as the lack of a petri.passed file not the presence of a petri.failed file.

[chris-oo]

i guess you can infer from the test name, but we should file an issue to follow up to make this more obvious in logview