Concurrently initialize LedgerStorage to optimize startup performance.

[zhaizhibo]

Descriptions of the changes in this PR:

Optimize BookKeeper startup speed with multi-directory ledger configuration

Main Issue:

Motivation

During production upgrades, BookKeeper instances with existing data typically take 3 to 5 minutes to start up.

Changes

Parallelize the initialization of DbLedgerStorage to improve startup speed.

[zhaizhibo]

after:

[StevenLuMT]

I understand that this optimization requires providing the changes before and after the optimization? Is there any verification data available?

[hangc0276]

Create a big thread pool just for init storage instance is not a good idea.
How many disks are you configured for one bookie?

[Copilot]

Pull Request Overview

This PR optimizes the startup performance of BookKeeper by initializing DbLedgerStorage concurrently across multiple ledger directories.

• Introduces a thread pool executor with a dynamic thread count based on available processors and ledger directories.
• Uses CountDownLatch and an AtomicInteger to manage and track initialization failures across threads.

Comments suppressed due to low confidence (1)

bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java:265

• Consider adding more context to the error log (e.g., the directory index or path that failed to initialize) to ease troubleshooting.

log.error("Failed to initialize DbLedgerStorage", e);

[Copilot]

After shutting down the executor, consider awaiting its termination to ensure all tasks have completed gracefully before proceeding.

Suggested change

	storageInitExecutor.shutdown();
	storageInitExecutor.shutdown();
	try {
	if (!storageInitExecutor.awaitTermination(60, java.util.concurrent.TimeUnit.SECONDS)) {
	log.warn("storageInitExecutor did not terminate within the timeout.");
	}
	} catch (InterruptedException e) {
	Thread.currentThread().interrupt();
	log.error("Interrupted while waiting for storageInitExecutor to terminate", e);
	}

[zhaizhibo]

Create a big thread pool just for init storage instance is not a good idea. How many disks are you configured for one bookie?

The single-threaded configuration cannot fully utilize the performance of SSDs. Each SSD configured for a bookie node is set up with 4 or 8 directories, depending on the disk performance.

[gaozhangmin]

Create a big thread pool just for init storage instance is not a good idea. How many disks are you configured for one bookie?

+1