feat(fuzz): add egfx_avc444_decode oracle and target#1327
Open
Greg Lamberson (glamberson) wants to merge 2 commits into
Open
feat(fuzz): add egfx_avc444_decode oracle and target#1327Greg Lamberson (glamberson) wants to merge 2 commits into
Greg Lamberson (glamberson) wants to merge 2 commits into
Conversation
This change adds an assertion-or-panic fuzz oracle for the AVC length-prefix to Annex-B conversion that runs inside `OpenH264Decoder::decode` before any OpenH264 entry point. The same change refactors the conversion from a private method on `OpenH264Decoder` into two public free functions in `ironrdp_egfx::pdu::avc`. The first function, `avc_to_annex_b`, returns a fresh `Vec<u8>` and is symmetric with the existing `annex_b_to_avc`. The second function, `avc_to_annex_b_into`, writes into a caller-provided buffer and preserves the per-frame buffer-reuse optimization that `OpenH264Decoder` relied on. The conversion is now available unconditionally to any consumer of `ironrdp-egfx`. The previous `#[cfg(feature = "openh264")]` gating went with the location, not the bytes-to-bytes logic, so lifting the function out of `decode.rs` removed the gate too. The new oracle exercises two input distributions on every fuzz call: - Direct path: the oracle calls `avc_to_annex_b(data)` on the raw fuzz input. This exercises the wrapper on arbitrary byte distributions, including inputs that do not parse as `Avc420BitmapStream`. - Decode-chain path: the oracle tries `Avc420BitmapStream::decode(data)`; on success it calls `avc_to_annex_b(stream.data)`. This exercises the wrapper on the realistic post-decode payload distribution. The oracle catches panics in the wrapper, OOM allocation from attacker-controlled NAL length encoding, and contract violations on the produced Annex-B byte stream. The oracle does NOT catch OpenH264 internal bugs (OSS-Fuzz coverage), the YUV-to-RGBA conversion path downstream of OpenH264 (separate workstream), or AVC444 luma plus chroma split (sibling target). Smoke fuzz ran 10,922,695 iterations in 31 seconds at ~352K exec/s sustained with zero crashes. Coverage settled at 158 lines, 456 features, 69 corpus entries. The new target auto-discovers into CI via the `cargo xtask fuzz list` dynamic fan-out mechanism. The new `check_egfx_avc420_decode` regression-replay test in `crates/ironrdp-testsuite-core/tests/fuzz_regression.rs` runs against the seed corpus and passes. Refs Devolutions#1316.
Sibling of `egfx_avc420_decode` (PR Devolutions#1326), applied to the AVC444 wire shape. AVC444 wraps two `Avc420BitmapStream`s plus a 2-bit encoding tag per MS-RDPEGFX 2.2.4.4 (LUMA_AND_CHROMA, LUMA, or CHROMA). The wrapper layer for each sub-stream is the same `avc_to_annex_b` function that Devolutions#1326 lifted to a public free function; this PR adds an oracle that exercises the wrapper on both sub-streams. This PR is stacked on PR Devolutions#1326 (which introduced `avc_to_annex_b` as a public function in `ironrdp_egfx::pdu`). It depends on Devolutions#1326 landing first. CI on this branch will not be green against `Devolutions/IronRDP:master` until Devolutions#1326 merges; that is the correct mechanical signal of the dependency. After Devolutions#1326 merges, this PR rebases cleanly because the only commit unique to this branch is the AVC444 oracle scaffolding. The new oracle exercises three paths on every fuzz call: - Direct path: the oracle calls `avc_to_annex_b(data)` on the raw fuzz input. This exercises the wrapper on arbitrary byte distributions, including inputs that do not parse as `Avc444BitmapStream`. - Decode-chain path 1: the oracle tries `Avc444BitmapStream::decode(data)`; on success it calls `avc_to_annex_b(stream.stream1.data)`. Stream 1 is the luma stream (or the combined luma-and-chroma stream when `encoding == LUMA_AND_CHROMA`). - Decode-chain path 2: when `stream.stream2.is_some()`, the oracle also calls `avc_to_annex_b(stream2.data)`. Stream 2 carries the auxiliary chroma data per the tagged-encoding split. The oracle catches panics in the wrapper, OOM allocation from attacker-controlled NAL length encoding inside either sub-stream, contract violations on the produced Annex-B byte stream, and any framing slip in `Avc444BitmapStream::decode` that misallocates `stream1` vs `stream2` byte ranges. The oracle does NOT catch OpenH264 internal bugs (OSS-Fuzz coverage), the post-OpenH264 YUV-to-RGBA conversion path downstream of OpenH264, the AVC444 luma plus chroma packing on the encoder side, or cross-stream state correlation across frames (multi-frame oracle is a future sibling target under the same umbrella). Smoke fuzz ran 10,246,473 iterations in 31 seconds at ~330K exec/s sustained with zero crashes. The new target auto-discovers into CI via the `cargo xtask fuzz list` dynamic fan-out mechanism. The new `check_egfx_avc444_decode` regression-replay test in `crates/ironrdp-testsuite-core/tests/fuzz_regression.rs` runs against the seed corpus and passes. Refs Devolutions#1316. Stacked on Devolutions#1326.
This was referenced May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sibling of
egfx_avc420_decode(PR #1326), applied to the AVC444 wireshape. AVC444 wraps two
Avc420BitmapStreams plus a 2-bit encoding tagper MS-RDPEGFX 2.2.4.4 (LUMA_AND_CHROMA, LUMA, or CHROMA). The wrapper
layer for each sub-stream is the same
avc_to_annex_bfunction that#1326 lifted to a public free function; this PR adds an oracle that
exercises the wrapper on both sub-streams.
This PR is stacked on PR #1326 (which introduced
avc_to_annex_bas a public function inironrdp_egfx::pdu). Itdepends on #1326 landing first. CI on this branch will not be green
against
Devolutions/IronRDP:masteruntil #1326 merges; that is thecorrect mechanical signal of the dependency. After #1326 merges, this
PR rebases cleanly because the only commit unique to this branch is
the AVC444 oracle scaffolding.
The new oracle exercises three paths on every fuzz call:
avc_to_annex_b(data)on the raw fuzzinput. This exercises the wrapper on arbitrary byte distributions,
including inputs that do not parse as
Avc444BitmapStream.Avc444BitmapStream::decode(data);on success it calls
avc_to_annex_b(stream.stream1.data). Stream 1is the luma stream (or the combined luma-and-chroma stream when
encoding == LUMA_AND_CHROMA).stream.stream2.is_some(), the oracle alsocalls
avc_to_annex_b(stream2.data). Stream 2 carries the auxiliarychroma data per the tagged-encoding split.
The oracle catches panics in the wrapper, OOM allocation from
attacker-controlled NAL length encoding inside either sub-stream,
contract violations on the produced Annex-B byte stream, and any
framing slip in
Avc444BitmapStream::decodethat misallocatesstream1vsstream2byte ranges.The oracle does NOT catch OpenH264 internal bugs (OSS-Fuzz coverage),
the post-OpenH264 YUV-to-RGBA conversion path downstream of OpenH264,
the AVC444 luma plus chroma packing on the encoder side, or cross-stream
state correlation across frames (multi-frame oracle is a future
sibling target under the same umbrella).
Smoke fuzz ran 10,246,473 iterations in 31 seconds at ~330K exec/s
sustained with zero crashes.
The new target auto-discovers into CI via the
cargo xtask fuzz listdynamic fan-out mechanism. The new
check_egfx_avc444_decoderegression-replay test in
crates/ironrdp-testsuite-core/tests/fuzz_regression.rsruns againstthe seed corpus and passes.
Refs #1316. Stacked on #1326.