HIVE-29637: Incorrect Results for IS_NULL Predicate on Partition Column

[deniskuzZ]

…n Column

What changes were proposed in this pull request?

IcebergTableUtil.makeSpecFromName() — a new helper that builds a Hive-compatible partition spec map from Iceberg's PartitionData. When a partition field value is null, it maps it to an actual NULL instead of the literal string "null".

Why are the changes needed?

Queries with WHERE <partition_col> IS NULL on Iceberg tables returned incorrect results (empty result set).

Does this PR introduce any user-facing change?

No

How was this patch tested?

mvn test -Dtest=TestIcebergCliDriver -Dqfile=iceberg_isnull_partition_pruning.q

[Copilot]

Pull request overview

Fixes incorrect results for WHERE <partition_col> IS NULL on Iceberg tables by ensuring null partition values are represented as actual Java null during partition predicate evaluation, and adds a regression qtest to cover the scenario.

Changes:

• Add IcebergTableUtil.makeSpecFromName(...) to build a partition spec map from an Iceberg partition path while translating Iceberg null partition values to Java null.
• Update HiveIcebergStorageHandler#getPartitionsByExpr to use the new helper when constructing DummyPartition specs for expression evaluation.
• Add a new positive qtest (iceberg_isnull_partition_pruning.q + expected output) validating IS NULL partition pruning behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java	Adds a helper for parsing Iceberg partition paths into a spec map with correct null handling.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java	Switches partition spec construction in getPartitionsByExpr to the new helper to fix IS NULL results.
iceberg/iceberg-handler/src/test/queries/positive/iceberg_isnull_partition_pruning.q	Adds a regression query file covering WHERE str_col IS NULL on an Iceberg-partitioned table.
iceberg/iceberg-handler/src/test/results/positive/iceberg_isnull_partition_pruning.q.out	Adds the expected output for the new regression test.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zabetak]

Left some high level comments/questions cause I am not very familiar with the modified APIs.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[zabetak]

Tentatively approve assuming that we all agree that the new behavior is the one desired by all users and the old one is merely a bug.

[zabetak]

The description of hive.exec.default.partition.name property says the following:

"The default partition name in case the dynamic partition column value is null/empty string or any other values that cannot be escaped."

This code assumes that the value is null but according to the description it may be other things as well.

The change here will fix the IS [NOT] NULL predicate but may potentially change the behavior of some queries:

SELECT key, value, ds FROM pcr_t1 WHERE ds > 'A';
SELECT key, value, ds FROM pcr_t1 WHERE ds < 'A';

Do the queries return the same results before/after the changes in the PR for Iceberg and non-Iceberg tables ?
Which behavior should prevail?

[deniskuzZ]

Non-Iceberg pcr_t1

  (NULL and '' both land in ds=__HIVE_DEFAULT_PARTITION__)
  
  ┌────────────────┬─────────────────────────────────────────┬─────────────────────────────────────────┬─────────────────┐
  │     Query      │               WITHOUT fix               │                WITH fix                 │ Changed by fix? │
  ├────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼─────────────────┤
  │ ds is null     │ (none) ❌                                │ C, D → ds=NULL                          │ ✅  fixed        │
  ├────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼─────────────────┤
  │ ds is not null │ A, B                                    │ A, B                                    │ no              │
  ├────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼─────────────────┤
  │ ds = 'null'    │ B                                       │ B                                       │ no              │
  ├────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼─────────────────┤
  │ ds > 'A'       │ C, D, B → ds=__HIVE_DEFAULT_PARTITION__ │ C, D, B → ds=__HIVE_DEFAULT_PARTITION__ │ no              │
  ├────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼─────────────────┤
  │ ds < 'A'       │ A                                       │ A                                       │ no              │
  └────────────────┴─────────────────────────────────────────┴─────────────────────────────────────────┴─────────────────┘
  
  Iceberg ice_01

  (NULL → null partition; '' → real ds='')

  ┌────────────────┬─────────────┬──────────┬─────────────────┐
  │     Query      │ WITHOUT fix │ WITH fix │ Changed by fix? │
  ├────────────────┼─────────────┼──────────┼─────────────────┤
  │ ds is null     │ (none) ❌    │ C        │ ✅  fixed        │
  ├────────────────┼─────────────┼──────────┼─────────────────┤
  │ ds is not null │ A, B, D     │ A, B, D  │ no              │
  ├────────────────┼─────────────┼──────────┼─────────────────┤
  │ ds = 'null'    │ B           │ B        │ no              │
  ├────────────────┼─────────────┼──────────┼─────────────────┤
  │ ds > 'A'       │ B           │ B        │ no              │
  ├────────────────┼─────────────┼──────────┼─────────────────┤
  │ ds < 'A'       │ D, A        │ A, D     │ no (same set)   │
  └────────────────┴─────────────┴──────────┴─────────────────┘

• The fix changes exactly one query on each table type: ds is null (previously returned nothing — the bug)
• Range queries (>, <) are unchanged before/after on both table types → no regression.
• Iceberg is fully SQL-correct throughout because it never conflates NULL with ''.

PR scope is NULL semantics only.

[zabetak]

Is there something useful in the EXPLAIN output? If not we can drop those.

[deniskuzZ]

yes, filterExpr: ds is not null (type: boolean)

[zabetak]

The empty string value is another edge case that may create problems

insert into pcr_t1 partition (ds) select 'D', 'V4', '';

[zabetak]

Consider enriching the tests with:

SELECT key, value, ds FROM pcr_t1 WHERE ds > 'A';
SELECT key, value, ds FROM pcr_t1 WHERE ds < 'A';