Skip to content

test: add mixed-format drop column schema evolution coverage#363

Open
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:issue-258-mixed-format-drop-column
Open

test: add mixed-format drop column schema evolution coverage#363
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:issue-258-mixed-format-drop-column

Conversation

@liujiwen-up
Copy link
Copy Markdown
Contributor

Purpose

Linked issue: #258

This PR adds focused coverage for mixed-format schema evolution after DROP COLUMN.

The fixture writes old-schema data files in Parquet and ORC with a dropped column, then drops that column and writes new-schema data files in Avro. The read test verifies that Rust can scan all three formats together while exposing only the remaining table columns.

Brief change log

  • Add mixed_format_schema_evolution_drop_column to the Spark provisioning script.
  • Write old-schema Paimon data files in Parquet and ORC formats with (id, name, score).
  • Drop score, switch to Avro, and write new-schema files with (id, name).
  • Add a Rust integration test that:
    • verifies the scan plan includes parquet, orc, and avro data files;
    • reads the table through FileSystemCatalog;
    • checks that the dropped score column is not present in output batches;
    • validates decoded id and name rows across all file formats.

Tests

  • cargo fmt --all
  • cargo test -p paimon-integration-tests test_read_mixed_format_schema_evolution_drop_column --no-run
  • cargo check -p paimon-integration-tests
  • python3 -m py_compile dev/spark/provision.py
  • git diff --check -- dev/spark/provision.py crates/integration_tests/tests/read_tables.rs

Full runtime execution of the new integration test requires regenerating the Spark-provisioned warehouse with make docker-up.

API and Format

No API or persistent format changes.

Documentation

No documentation changes.

@liujiwen-up liujiwen-up force-pushed the issue-258-mixed-format-drop-column branch from 089ac48 to d6f30fe Compare June 4, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant