[improve][pcip] PCIP-4 Improve delayed RPC message handling in pulsar-rpc

[Denovo1998]

Motivation

Currently, Pulsar RPC encounters some issues with delayed messages (delayed RPC). The current behavior involves storing RPC requests and the responses returned by the server in the pendingRequestsMap. When the server responds, it checks the pendingRequestsMap for the corresponding RPC request. If no request is found, the response is “discarded” and not returned for user handling. We aim to perform RPC requests on a scheduled basis, especially when there are long delays, meaning the message will persist in the pendingRequestsMap for an extended period. Therefore, we need to improve the handling behavior for such delayed RPC scenarios.

Modifications

In the case of delaying the sending of RPC requests:

1. The client side no longer adds requests to the pendingRequestsMap; instead, it directly sends a delayed message to the request topic. A flag indicating that this request is a delayed message, as well as the time when the message delay expires (the time that triggers message delivery), is added to the message’s properties.
2. Upon reaching the delay time, the server side receives this message, processes it, and then sends it to the client side.
3. After the client receives the reply message, if it is a delayed message, it will directly put it into the pendingRequestsMap and directly trigger requestCallBack.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

[Denovo1998]

@liangyepianzhou @StevenLuMT @AuroraTwinkle @eolivelli PTAL!

[StevenLuMT]

Your PR is modified based on the original PCIP-2
Please submit a PR for pcip, numbered PCIP-4

[Denovo1998]

@liangyepianzhou @StevenLuMT @AuroraTwinkle @eolivelli The PCIP document has been added. PTAL!

[liangyepianzhou]

LGTM

[StevenLuMT]

Good jobs

[HQebupt]

LGTM