[Enhancement] Improve the pop revive process when reading biz messages from a remote broker

[imzs]

Before Creating the Enhancement Request

• [X] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

In RIP-32 we put forward a new proposal, slave acting master mode, as an upgrade of master-slave deployment mode. To prevent consumption stagnation of delayed message, transaction message and pop mode, RIP-32 provides two basic abilities:

• write to remote brokers (ck, delay message, op)
• read from remote brokers (biz message)

but it lacks robustness for the pop revive process when reading messages from remote.

Motivation

1. As the basis for the mechanism, let's see the definition of lower and upper api

CompletableFuture<PullResult> BrokerOutAPI.pullMessageFromSpecificBrokerAsync
CompletableFuture<Pair<GetMessageStatus, MessageExt>> EscapeBridge.getMessageAsync

PullResult & GetMessageStatus means and only means FOUND or not, but when reading from a remote broker, it's not appropriate to use the same semantics as local store, because it can't deal with unexpected errors such as RPC problems. The api invoker has no extra information to decide whether to retry or not.

2. Then talk about pop revive process, when we find a message to be revived, read biz message from local or remote and write to retry topic, but rewrite CK if read failed. In the current implementation, there will be no CK reentry because of the issue above, and even if we fix it, in some condition we'll face an endless CK rewrite because it enters into a loop of failed reading result and CK rewrite, so we need an exponential backoff and ending condition.

Describe the Solution You'd Like

1. extra information to the call chain of remote message reading
2. an exponential backoff and ending condition of CK rewrite

Describe Alternatives You've Considered

See Above

Additional Context

No response