[improve][pip] PIP-461: Add queued latency metrics for offloader executors

[BewareMyPower]

Motivation

The default read threads for tiered storage is 2, when there are many concurrent read operations, many will be queued for long but there is no metric for it, so it's hard to tune the threads.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[shibd]

Maybe we can leave streamingOffload() out of scope for now, since this path does not seem to be wired into the current production offload flow yet. If you would still like to include it here, could we also note that there is an intentional delayed resubmission in BlobStoreManagedLedgerOffloader.streamingOffloadLoop() and subtract that delay from the queue-latency metric?

scheduler.chooseThread(segmentInfo)
        .schedule(() -> {
            streamingOffloadLoop(partId, dataObjectLength);
        }, 100, TimeUnit.MILLISECONDS);

Otherwise the metric may end up including this built-in 100ms delay, instead of only the actual executor queue time.

[shibd]

Maybe it could be helpful to add one more attribute here, something like operation (or caller). These executors seem to handle more than just one kind of work, so separating values such as offload, delete, open, read, and close might make the queue-latency metrics easier to interpret in practice. For example, it would be easier to tell whether the latency is coming from read traffic itself or from extra work scheduled on the same executor.