Skip to content

Split traits for plain and bitpacked decoding and fix soundness issue in BitReader::get_batch#10172

Open
jhorstmann wants to merge 2 commits into
apache:mainfrom
jhorstmann:from-bitpacked-safety
Open

Split traits for plain and bitpacked decoding and fix soundness issue in BitReader::get_batch#10172
jhorstmann wants to merge 2 commits into
apache:mainfrom
jhorstmann:from-bitpacked-safety

Conversation

@jhorstmann

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The FromBitpacked trait was introduced in #9662 as a marker for types that could be decoded from bitpacked data, but the implementation still mostly used methods from its FromBytes super-trait. The implementation of BitReader::get_batch that dispatched on the size of the type then required a complex safety contract, which meant that FromBytes had to be an unsafe trait.

Instead of that size dispatch, we can move the unsafe code into a method of the FromBitpacked trait, which makes BitReader::get_batch completely safe. This also removes the need to make any of those traits sealed as proposed in #10166.

What changes are included in this PR?

The responsibilities of the FromBytes and FromBitpacked traits should be much clearer now:

  • FromBytes is used for plain encoded data
  • FromBitpacked is used for bitpacked data

Unsafe code is moved from BitReader::get_batch into some of the FromBitpacked implementations.

Are these changes tested?

All existing tests pass.

Are there any user-facing changes?

FromBitpacked gains a new method and is no longer a sub-trait of FromBytes, so this might be breaking change. Code that previously implemented FromBytes will need to change to remove the unsafe keyword and BIT_CAPACITY constant.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Soundness: Unsound alignment contract in public FromBytes trait and BitReader::get_batch

1 participant