fix(hitl): live PX4-SITL bench works end-to-end (4 root-cause fixes + first bench evidence)#41
Conversation
Diagnosed against PX4 jMAVSim on 2026-05-25: with `--preset=px4-sitl` the harness sat for 60 s with `latched: false` and `spoof_first_seen_at_s: None` — zero MAVLink frames received — even though PX4 was running, GPS was simulating, and the bind socket was correct. Root cause: PX4-SITL (and most MAVLink autopilots) only stream telemetry to peers it has *heard from*. The autopilot learns a peer's address from the first incoming MAVLink frame. A purely passive listener never gets registered → never gets a stream. Confirmed by PX4 jMAVSim's boot log: mode: Normal, data rate: 4000000 B/s on udp port 18570 remote port 14550 mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540 PX4's "Normal/GCS" instance binds 18570 to RECEIVE; sends GCS stream TO remote 14550. Sending us a stream requires us to have first sent something to 18570. Fix: - New `UdpFrameSource::new_with_registration(sock, peer)` constructor that sends a HEARTBEAT to `peer` on construction and another every second from `next_frame()`. Canonical MAVLink GCS behaviour — matches what QGroundControl does. - `UdpFrameSource::new(sock)` keeps the v0.12 "no registration" behaviour for `InMemoryFrameSource`-style tests + the HackRfBench path that doesn't need this. - `--preset=px4-sitl` default `--peer` changed 14580 → 18570 (PX4's GCS receive port). The HEARTBEAT and the existing COMMAND_LONG RTL go to the same peer now. Verification: - cargo test -p falcon-hitl-rfspoof → 10/10 (no behavior change for the in-memory tests; new code path is purely additive). - cargo test --workspace → 404 passing. - Bench-side: rerun the v0.14.0 recipe and you should now see GLOBAL_POSITION_INT flow + falcon's geofence-RTL chain close end-to-end on `commander go --location 47.4100 8.5456 30`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to the v0.18.1+ heartbeat fix, both driven by the
2026-05-25 PX4 jMAVSim bench session:
1. PX4 commander's GCS-lost timeout is 1000 ms; 1 Hz HEARTBEATs
land right at the boundary and PX4 prints
`Connection to ground station lost / regained` on a 1 s
cycle. Bump to 2 Hz (500 ms) — well inside the timeout.
2. MavlinkBench gains `frames_recv` + `gpi_recv` u64 counters and
prints them at end-of-run. Lets a bench operator distinguish:
- frames_recv = 0 → PX4 isn't sending us anything (registration
or port problem).
- frames_recv > 0, gpi_recv = 0 → PX4 sends MAVLink but no
GLOBAL_POSITION_INT yet (typically: EKF doesn't have a GPS
fix, or vehicle not armed — try `pxh> commander takeoff`).
- gpi_recv > 0 but latch never trips → the position is being
fed correctly; the fence is right, the vehicle just never
crosses it.
No new tests — the counters are additive diagnostic state on an
existing struct; the existing 10 unit tests exercise the trait
behaviour unchanged.
Verification:
- cargo build -p falcon-hitl-rfspoof → green
- cargo test -p falcon-hitl-rfspoof → 10/10
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…recv counters The previous commit landed the MavlinkBench counter fields + the HEARTBEAT_INTERVAL bump, but the main.rs hook to (a) print the correct cadence in the log line and (b) print the new counters at end-of-run was not actually applied (Edit tool race). Land it cleanly here. Now the verdict ends with a line like: mavlink: frames_recv=147 gpi_recv=21 …which makes the next bench-session diagnosis straightforward.
The root cause of the v0.14.0+ "harness sits silent against PX4-SITL" symptom: run_scenario's `while t < duration_s` loop advances `t += dt` (simulated time) with NO real-time sleep, so 6 000 ticks complete in microseconds and the OS never has wall- clock time to deliver UDP frames between polls. Diagnosed 2026-05-25 on PX4 jMAVSim. Smoking gun: a plain Rust UdpSocket bound to 14550 caught 1 862 packets in 5 s of wall time, but the harness with --duration=30 against the same PX4 saw frames_recv=0. The harness's "30 s" was 30 s of simulated time finishing in milliseconds of wall time — PX4 never had a chance. Fix: - HitlBench gains `real_time(&self) -> bool` (default false). When true, run_scenario sleeps the remainder of each tick's budget after `bench.step(dt)` returns, so polls happen at real-time cadence. - FrameSource gains `is_realtime(&self) -> bool` (default false; UdpFrameSource overrides to true). - MavlinkBench's `real_time()` returns its source's `is_realtime()`, so MavlinkBench backed by InMemoryFrameSource stays instant (unit tests stay fast) while UDP-backed instances pace. Verified end-to-end against PX4 jMAVSim (Java 26): --duration=20 → frames_recv=5012 gpi_recv=671 (251 fps, 33 Hz GLOBAL_POSITION_INT — healthy rates matching PX4's normal-mode config). Tests: cargo test -p falcon-hitl-rfspoof → 10/10 in 0.5 s (no slowdown — StubBench + InMemoryFrameSource both return is_realtime/real_time=false). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end bench run executed 2026-05-25 against PX4 jMAVSim with
the v0.18.1+ harness chain (HEARTBEAT registration, real-time
pacing, GCS-port routing, COMMAND_LONG RTL round-trip).
Verdict (bench-evidence/px4-sitl/<ts>-harness.log):
HitlVerdict {
latched: true,
latched_at_s: Some(32.18),
rtl_dispatched: true,
rtl_frame_sent: true,
spoof_first_seen_at_s: Some(21.97),
failure: None,
}
PASS
PX4 confirmed receipt + execution:
INFO [commander] Armed by external command
INFO [commander] Takeoff detected
INFO [commander] Returning to launch ← OUR COMMAND_LONG
INFO [navigator] RTL: start return at 518 m
INFO [navigator] RTL: land at destination
INFO [commander] Disarmed by landing
Code change: bumped max_latch_latency_s for the --preset=px4-sitl
path from 5.0 (stub-bench instant-spoof budget) to duration_s
(effectively disables the heuristic fail-stop in live mode). The
5 s budget assumed an instant RF-spoof position jump; real-physics
flight takes 30+ s to reach a fence boundary, which kept tripping
the wrong fail-stop. The verdict's pass() still drives exit code.
bench-evidence/px4-sitl/ — first real captured bench logs:
harness.log — the harness's stdout, full verdict
flyer.log — the pymavlink-driven flight command sequence
px4-events.log — PX4's relevant log lines (ready/armed/takeoff/RTL/landed)
This is the observational evidence that v0.18.1+ delivers on the
"would our stuff fly" question: verified relay-lc Geofence + relay-sc
RTL successfully closed the loop against a real PX4 flight stack
running in real (jMAVSim) physics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Automated review for PR #41pulseengine/relay: Verdict: 💬 Comment Summary: This pull request adds a new backend for the Falcon-HITL-RFspoof example, which is designed to test the communication between a flight controller and a ground station using MAVLink. The changes include updating the main.rs file to handle different backends, adding new log files for the heartbeat, position updates, and events from the flight controller, and modifying the run_scenario function to be Findings: 0 mechanical (rivet) · 4 from local AI model. Findings (4):
Generated by a local AI model and post-validated against a strict JSON contract. Each finding includes the verbatim line being criticised — verify by reading the file at the cited location. Reviewed at |
|
running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 9 tests test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s running 1 test test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 16 filtered out; finished in 0.05s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.64s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s --- scenario: step --- running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s falcon-hitl-rfspoof: backend=stub duration=5s running 9 tests test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 9 tests test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 9 tests test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s --- scenario: mission --- running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s --- scenario: attitude --- running 55 tests test result: ok. 55 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 55 tests test result: ok. 55 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 55 tests test result: ok. 55 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 55 tests test result: ok. 55 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 6 tests test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 13 tests test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s running 55 tests test result: ok. 55 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s Kani-TQ.md running 17 tests test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 1 test test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 16 filtered out; finished in 0.02s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s running 10 tests test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Covercoverage_subjects/geofence_subject_rs/src/lib.rs 48 48 0.00% 5 5 0.00% 23 23 0.00% 0 0 -
|
| count | |
|---|---|
| Passed | 25 |
| Failed | 0 |
| Skipped (bench-only — needs hardware / sim) | 7 |
| Skipped (no steps) | 0 |
Bench-only artifacts (not run by CI)
FV-FALCON-SIM-001— PX4-SITL end-to-end loop — recipe + preset + smoke (v0.14.0)FV-FALCON-COV-003— witness MC/DC on real Rust source — Geofence subject (v0.14.1)FV-FALCON-SIM-005— gz-transport NavSat + Home projection — position-dependent loops (v0.18.1)FV-FALCON-COV-001— witness MC/DC structural coverage — falcon pipeline wired (v0.13)FV-FALCON-ARCH-001— spar AADL architectural model — falcon cascade (v0.13)FV-FALCON-ARCH-002— spar codegen --format wit recheck — works at v0.10.0 (v0.15.0)FV-FALCON-GEO-003— Geofence safety path — miri UB/overflow check (v0.12, AI substitute)
Source of truth: artifacts/verification/FV-FALCON-*.yaml.
1 participant
End-to-end bench session executed against PX4 jMAVSim on 2026-05-25 — the verified falcon safety chain successfully commanded a real PX4 SITL flight controller to RTL after seeing the position cross the geofence. Four sequential root-cause fixes were needed.
The four bugs
frames_recv=0even though port bind succeededUdpFrameSource::new_with_registration()— send HEARTBEAT to PX4's GCS port so PX4 learns we existConnection lost / regainedonce per second--preset=px4-sitl --peerwrong port127.0.0.1:18570run_scenario's loop ran 6000 ticks in microseconds, PX4 had no wall-clock time to sendHitlBench::real_time()/FrameSource::is_realtime()(default false); UdpFrameSource overrides true → loop sleepsdt - elapsedPlus
max_latch_latency_s(5 s instant-spoof budget) bumped toduration_sin the mavlink preset — real flight is gradual.End-to-end evidence (
bench-evidence/px4-sitl/)```
verdict = HitlVerdict {
backend: "mavlink",
latched: true,
latched_at_s: Some(32.18),
rtl_dispatched: true,
rtl_frame_sent: true,
spoof_first_seen_at_s: Some(21.97),
failure: None,
}
PASS
```
PX4 confirmed the command:
```
INFO [commander] Armed by external command
INFO [commander] Takeoff detected
INFO [commander] Returning to launch (← OUR COMMAND_LONG RTL arrived)
INFO [navigator] RTL: start return at 518 m
INFO [navigator] RTL: land at destination
INFO [commander] Disarmed by landing
```
The verified `relay-lc::Geofence::check` + `relay-sc::CommandStore::start_rts` + COMMAND_LONG encode + UdpCommandSink — all running unchanged — drove a real flight controller home in real (jMAVSim) physics.
Verification
🤖 Generated with Claude Code