feat: support DROP INDEX DDL

[LuciferYang]

Summary

Add support for dropping indexes on Lance tables through the Spark SQL interface:

ALTER TABLE catalog.db.table DROP INDEX index_name;

This is the counterpart to the existing CREATE INDEX and SHOW INDEXES commands, completing the index lifecycle management in lance-spark.

Design

DROP INDEX is a metadata-only operation — it removes the index entry from the dataset manifest via lance-core's dataset.dropIndex(name) API. Physical index files are not deleted; they are cleaned up by VACUUM during garbage collection.

The implementation follows the standard lance-spark SQL extension pipeline:

1. ANTLR grammar — ALTER TABLE ... DROP INDEX indexName rule + DROP token
2. AST builder — visitDropIndex in all version-specific builders (3.4, 3.5, 4.0, 4.1)
3. Logical plan — LanceDropIndex(table, indexName) with output schema (index_name, status)
4. Physical exec — LanceDropIndexExec calls dataset.dropIndex(indexName) on the driver
5. Strategy — LanceDropIndex → LanceDropIndexExec mapping with indexName.toLowerCase for consistency with CREATE INDEX

Note: The logical plan and physical exec are named LanceDropIndex / LanceDropIndexExec (not DropIndex / DropIndexExec) to avoid classpath collisions with Spark's built-in classes of the same name in spark-catalyst and spark-sql, which have different constructor signatures.

Changes

• LanceSqlExtensions.g4 — Added dropIndex grammar rule and DROP token
• LanceSqlExtensionsAstBuilder.scala (3.4, 3.5, 4.0, 4.1) — Added visitDropIndex visitor
• DropIndex.scala — New LanceDropIndex logical plan
• DropIndexExec.scala — New LanceDropIndexExec physical execution node
• LanceDataSourceV2Strategy.scala — Added LanceDropIndex → LanceDropIndexExec dispatch
• BaseAddIndexTest.java — Added testDropIndex and testDropIndexThenRecreate
• test_lance_spark.py — Added test_drop_index and test_drop_index_then_recreate integration tests
• drop-index.md — New documentation page

Test plan

Unit tests (BaseAddIndexTest.java)

• testDropIndex — Creates index, drops it, verifies schema/output, confirms index is gone
• testDropIndexThenRecreate — Full lifecycle: create → drop → recreate → verify query works
• All 9 AddIndexTest tests pass (7 existing + 2 new) across all modules (3.4, 3.5, 4.0, 4.1)
• Full test suite passes, 0 failures

Integration tests (test_lance_spark.py)

• test_drop_index — Creates BTree index, drops it via SQL, verifies output schema (index_name, status), confirms SHOW INDEXES no longer lists it
• test_drop_index_then_recreate — Full lifecycle: create → drop → recreate, verifies recreated index appears in SHOW INDEXES and queries still return correct results

[hamersaw]

Looks great thanks! Can you just add an integration test?

[LuciferYang]

Thanks for the review! Added two integration tests in docker/tests/test_lance_spark.py under TestDDLIndex:

• test_drop_index — creates a BTree index, drops it via ALTER TABLE ... DROP INDEX, verifies the output schema (index_name, status), and confirms the index no longer appears in SHOW INDEXES.
• test_drop_index_then_recreate — full lifecycle: create → drop → recreate, then verifies the recreated index exists in SHOW INDEXES and queries still work correctly.

[LuciferYang]

need to further investigate the Docker testing

[LuciferYang]

all test passed

[hamersaw]

Great add! Thanks.

[LuciferYang]

thanks @hamersaw

[LuciferYang]

By the way, I would like to mention that I will be on a 4-day holiday. During this period, I may not respond promptly to code fix suggestions.

[hamersaw]

By the way, I would like to mention that I will be on a 4-day holiday. During this period, I may not respond promptly to code fix suggestions.

Enjoy the break!