Reconsider permitting interactions between manual and automatic scans

[bertllll]

Intro During the big rearchitecture of the payment system so that it accommodates various edge cases the txs can fall into and is able to resubmit them no matter what the reason was why we considered them to have failed, we took a strategical decision that in the sake of simplicity, we will disallow manual scanning for all the time when the Node starts up with the automatic scans on.

This card, on the contrary, is about changes to be made which would secure a safe coexistence of these two types enabled.

Problematics

There are a couple of small challenges that are good to know about.

a) You will do the best if the PendingPayableScanner is simply excluded ever. It will simplify further works greatly.

The reasoning behind this is that:

• It would be inconvenient to meddle with the "pending payable scan sequence" (which are possibly repeated (set apart by intervals) runs of the coupled scanners PendingPayableScanner + RetryPayableScanner, not ending until all known, administrated pending payables are found complete), because it is quite complex and carefully tuned mechanism which should work very well in the automatic-scans mode. Allowing the manual request in that time would cause a big surface of possible interactions which we want to avoid.
• After the sequence finishes, there is no use in invoking this same scanner again because we absolutely know there are no pending payables at that moment. That holds until a new run of the NewPayable scanner is performed, producing likely new payments and therefore also new pending payables to track.

b) Do not allow running the NewPayablesScanner as long as the pending payable scan sequence is in proces. Simply, it wouldn't help anything if we produced more payments before we get over those already pending. It's better to go in stages like it is designed for the automatic process.

c) You'll have face an issue which we used to ignore even in the old master branch code. It is the fact that if one scan is already running another just triggered would be always canceled. We always let the whole scan procedure of any scanner to finish unaffected by another scanner of the same type. It gains importance especially because all our scanners have procedures consisting of bidirectional communication of two Actors, requiring at least two messages to be sent between, in the PayableScanner even more than two.
That seems like a reasonable measure. However, it can cause the automatically scheduled chain of scans to break down. How? Consider that there is a scan scheduled for the time T(sch). Also remember that each scheduled scan has to end up by the setup of the next scan to be performed sometime in the future. It is an endlessly repeating operation, and so the scheduling is so prone to mishandle. Now let's think about a manually triggered scan which was requested very closely before T(sch), that is the time T(m). Because the manual scan was allowed (and could urge the payments if the user felt so), it is currently running. The time T(sch) comes up and this scheduled scan is halted since two scanners alike cannot be running while overlapping. That exposes a serious problem, though. If we simply throw the scheduled scan away and let only the manual one finish, the consequences are we aren't going to have a single scan running again afterwards unless triggered manually. This collision just demolished the automation for this particular scanner (Yes. It should be noted that the schedules are isolated form each other between the scanners. So only the scanner it happened on directly will suffer the death.).

If you decide to add a defense to this rather a less likely event, it must be one which can be easily implemented for all three scanners the user will be aware of (Payables, PendingPayables, Receivables), excluding the subclass of RetryPayableScanner which cannot be controlled directly but is always run on the request coming from the PendingPayableScanner.

It should be some sort of solution where we set a flag of a canceled scheduled scan (That can be easily recognized by checking out the response skeleton's presence), and then even a manually triggered scan (Given by the same identification method) will be allowed to schedule another scan, if this flag says true.