Skip to content

fix: sync structured primary key fields after metadata update#6326

Open
wombatu-kun wants to merge 9 commits intolance-format:mainfrom
wombatu-kun:fix-sync-unenforced-pk-fields-with-metadata
Open

fix: sync structured primary key fields after metadata update#6326
wombatu-kun wants to merge 9 commits intolance-format:mainfrom
wombatu-kun:fix-sync-unenforced-pk-fields-with-metadata

Conversation

@wombatu-kun
Copy link
Copy Markdown
Contributor

Summary
Fixes #6324

UpdateConfig.fieldMetadataUpdates() correctly persists lance-schema:unenforced-primary-key and lance-schema:unenforced-primary-key:position in the field's metadata HashMap, but the corresponding structured field unenforced_primary_key_position (used for protobuf serialization and is_unenforced_primary_key()) was never updated. This caused primary key information set via metadata updates to be lost on the next dataset reload.

Root cause
apply_update_map() in transaction.rs only modifies field.metadata (a HashMap<String, String>) without syncing the structured Field.unenforced_primary_key_position field that drives protobuf serialization and the is_unenforced_primary_key() API.

Fix

  • Added Field::sync_embedded_metadata() that re-parses well-known metadata keys (lance-schema:unenforced-primary-key, lance-schema:unenforced-primary-key:position) and updates the corresponding structured fields.
  • Call sync_embedded_metadata() after every apply_update_map() on field metadata in Transaction::build_manifest().
  • Refactored TryFrom<&ArrowField> for Field to reuse sync_embedded_metadata() instead of duplicating the parsing logic.

@github-actions github-actions bot added the bug Something isn't working label Mar 28, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 28, 2026

Codecov Report

❌ Patch coverage is 99.55157% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-core/src/datatypes/field.rs 93.75% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UpdateConfig.fieldMetadataUpdates does not set protobuf 'unenforced_primary_key' field

1 participant