[processor] Check that received batches match a valid Header (or better)

[huitseeker]

The processor currently accepts any data from a counterpart, hashes it and reports the availability of the corresponding hash.

https://github.com/MystenLabs/narwhal/blob/main/worker/src/processor.rs

But since this does not involve any signature verification or deserialization, this data could be rubbish (and not correspond to any Header), and imposes an unchecked burden of storage on the worker.

We should make sure any accepted batch by a worker is matched to a valid Header (or Certificate) within a short amount of time.

See also #156 and #188. /cc @asonnino for awareness.

[asonnino]

Yes this is an issue that we always had. The solution we discussed at the time is to allow a maximum number of batches for which we don't have the header per peer; once we reach this maximum, we drop the oldest batches

[akichidis]

Following the discussions from this lightweight design doc we'll go with solution 2, meaning keeping an LRU cache for each worker in primary that will act as a coordinator to clean up stale batches