Skip to content

Core, Hive: Close FileIO in HadoopCatalog and HiveCatalog#15968

Open
dejii wants to merge 1 commit intoapache:mainfrom
dejii:fix/hadoop-hive-close-fileio
Open

Core, Hive: Close FileIO in HadoopCatalog and HiveCatalog#15968
dejii wants to merge 1 commit intoapache:mainfrom
dejii:fix/hadoop-hive-close-fileio

Conversation

@dejii
Copy link
Copy Markdown

@dejii dejii commented Apr 14, 2026

Summary

  • HadoopCatalog: add fileIO to existing closeableGroup
  • HiveCatalog: introduce CloseableGroup (replacing manual close calls) and register fileIO, keyManagementClient, and metricsReporter

Problem

All major Iceberg catalog implementations close their FileIO when catalog.close() is called — except HadoopCatalog and HiveCatalog:

For the default HadoopFileIO this is harmless — it has no close() implementation. But users who configure a non-default FileIO via io-impl (e.g. S3FileIO, GCSFileIO) will leak HTTP connection pools since close() is never called on the FileIO.

Impact

  • Default config (HadoopFileIO): No behavioral change
  • Non-default FileIO (S3FileIO, GCSFileIO): Connection pools are now properly released on catalog close

Discovered while fixing a connection pool crash in Apache Beam's IcebergIO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant