Fill forward resolution nondeterministic case

[Martin-Molinero]

Expected Behavior

• No nondeterministic behavior

Actual Behavior

• https://github.com/QuantConnect/Lean/pull/6245 helped uncover a non deterministic behavior of fill forward resolution. CustomUniverseWithBenchmarkRegressionAlgorithm is adding and removing a Minute resolution subscription while the other algorithm subscription is hour resolution. By doing so it's changing the fillForwardResolution the data stack is using back and forth between minute and hour. Because each data subscription is run ahead of time, independently, the data points already queued up might no longer respect the fillForwardResolution of the data stack at that point in time. Causing a race condition with the worker threads creating new data points and the FF resolution, with the result of the data points fluctuating.

Potential Solution

• Believe this is a rare case and ideally the solution shouldn't cause a performance overhead on the data stack as best as possible
• Dropping the data points that no longer are valid due to the changes in the fillForwardResolution the data stack is using. Determining which are those data points might be the tricky part.
• Drop all FF data points and re pass each queue by a FF enumerator
• Move the FF enumerator outside of the parallel stack -> heart breaking for performance
• ...

Reproducing the Problem

See CustomUniverseWithBenchmarkRegressionAlgorithm

System Information

N/A

Checklist

• [x] I have completely filled out this template
• [x] I have confirmed that this issue exists on the current master branch
• [x] I have confirmed that this is not a duplicate issue by searching issues

• [x] I have provided detailed steps to reproduce the issue