sled
sled copied to clipboard
lineage-driven fault injection
since all IO and cross-thread communication points are manually instrumented with failpoints and debug_delay, we can use these calls to record traces of IO and linearization points.
- run a workload generated by the normal tree quickcheck, recording the fault injection points that were crossed
- start injecting faults from the end, waking our way backwards to the beginning
- apply the failpoints fault correctness model & normal event log during execution
this greatly reduces the bug search space during fault injection testing to only injecting failures that may actually impact a particular workload
this probably does not need to pull in a solver in the style of @palvaro's molly initially to get a large amount of the benefit, but rust has some solver libraries we may be able to rely on to start doing that sort of targeted fault tree exploration.