Rectify scheduling following failures in the retry payable mode
(Planned in the second phase)
I committed a little mistake in GH-602 and didn't realize until now when I and @utkarshg6 were contemplating about something else over the code.
The payable scanner has two modes, instead of having two fully-fledged scanners for payables. I kinda hoped we'd be able to keep the ending phase of each cycle the same for both modes. It looked like a great simplification. The way it is I used a combination of traits to split the opening phase which is framed by producing the QualifiedPayablesMessage, which either contains UnpricedQualifiedPayables configured for the first or the second mode. (Technically Some() previous attempt gas price or None, as the retry payable mode or new payable respectively). This way we can deduce what mode is running up to the moment we let the qualified payables be priced by the blockchain agent (just after this one was created). Since here on, we don't have, at the moment, any feature in place to tell apart those modes.
First part
This card implies though that we do want to be able to tell even down the pipelines, as much as to the end of the scan, in the finish_scan() fn. Therefore one of the objectives of this card is to crate an enum to represent the mode and add it to all these messages in the train of BlockchainAgentWithContextMessage -> OutboundPaymentInstructions -> SentPayables.
Additionally, we want to expand the OperationOutcome enum (whose name may should be improved, perhaps to carry a reference to "payable(s)") that is a part of the struct PayableScanResult. The current variants say NewPendingPayable and Failure.
I suggest NewPendingPayable, NewPayableFailure and RetryPayableFailure newly.
And at the end, just wire things up so that when we encounter a failure and the mode being run is "retry" we should return `OperationOutcome::RetryPayableFailure and vice versa.
The second part
Inside impl Handler<SentPayables> for Accountant {}, you will inevitably be questioned by the compiler what to do with the new possibility in the OperationOutcome enum.
We want:
a) Keep OperationOutcome::NewPendingPayable unchanged.
b) OperationOutcome::NewPayableFailure is actually what now is represented by the basic OperationOutcome::Failure
c) OperationOutcome::RetryPayableFailure, though, should be followed by another schedule to the PendingPayableScanner`, using the same code which we call under a)
Add a test which describe a situation like this: Imagine we've already launched the pending payable sequence (the repetitive scanning for payables to retry), it's a pair of scanners that can run both in some cases. Here it just happened, the PendingPayableScanner ordered the retry payable scan but it failed during the payment attempt, but because it means the situation couldn't be improved if we did not send any retry payment, we still want to keep trying. Therefore, we make sure the sequence hasn't died out and schedule another attempt of the PendingPayableScanner. (Objectives for the test: You send the SentPayables message to accountant which points to a failure and also is marked as the RetryPayable mode. Assertions should be made by checking on the schedulers that they were asked to schedule the pending payable scan. See some other tests to understand; there are a couple of examples.)