AnthonyLloyd
Support live visualizations with Tyche, via the `OpenPBTStats` log format?
Tyche is a VS Code extension designed to help users understand what their property-based tests are doing - with nice features like live-updating graphs, and lots of explorable raw data if you want to dig deeper.
I've been collaborating with the author since late last year (as a Hypothesis maintainer), as described in our paper. Notably, this helped us find several performance problems and outright bugs in Hypothesis itself, as well as being useful for our users.
But the reason I'm opening this issue is that Tyche is not Hypothesis-specific; it's instead based on a simple json-lines format which could be emitted from any PBT library. If you want to support it from CsCheck, emitting the type, run_start, property, status, and representation is enough to enable a substantial portion of Tyche’s features.
Looks interesting. I think this can be supported by adding an extension method much like how I used to do DebugClassify (before making it an overload of Sample).
I made a start on this. @AnthonyLloyd I am not sure if the decorator approach of DebugClassify will work here - in DebugClassify you could nicely wrap the output of the initial generator, but I believe the Tyche addition should occur in the SampleActionWorker because it requires the individual generated inputs. I made some quick-and-dirty changes:
public record TycheRecord(
string type, double run_start, string property, string status, string representation,
string? status_reason, Dictionary<string, string> arguments, string? how_generated, Dictionary<string, string> features,
Dictionary<string, string>? coverage, Dictionary<string, string> timing, Dictionary<string, string> metadata);
sealed class SampleActionWorker<T>(Gen<T> gen, Action<T> assert, CountdownEvent cde, string? seed, long target, bool isIter, string propertyUnderTest) : IThreadPoolWorkItem
{
public PCG? MinPCG;
public ulong MinState;
public Size? MinSize;
public T MinT = default!;
public Exception? MinException;
public int Shrinks = -1;
public long Total = seed is null ? 0 : 1;
public long Skipped;
public ConcurrentBag<TycheRecord> TycheOutput = new ();
public void Execute()
{
var pcg = PCG.ThreadPCG;
Size? size = null;
T t = default!;
long skippedLocal = 0, totalLocal = 0;
ulong state = 0;
double timestamp = -1.0;
void RecordTycheOutput(string status)
{
var d = new Dictionary<string, string>(StringComparer.Ordinal);
TycheOutput.Add(new TycheRecord("test_case", timestamp, propertyUnderTest, status, Print(t)
,"reason",d,"testing",d,null,d,d));
}
while (true)
{
try
{
while (true)
{
if ((isIter ? Interlocked.Decrement(ref target) : target - Stopwatch.GetTimestamp()) < 0)
{
Interlocked.Add(ref Skipped, skippedLocal);
Interlocked.Add(ref Total, totalLocal);
cde.Signal();
return;
}
totalLocal++;
state = pcg.State;
t = gen.Generate(pcg, MinSize, out size);
if (MinSize is null || Size.IsLessThan(size, MinSize))
{
timestamp = ((DateTimeOffset)DateTime.Now).ToUnixTimeMilliseconds() / 1000.0;
assert(t);
RecordTycheOutput("passed");
}
else
skippedLocal++;
}
}
catch (Exception e)
{
lock (cde)
{
if (MinSize is null || Size.IsLessThan(size, MinSize))
{
MinSize = size;
MinPCG = pcg;
MinState = state;
MinT = t;
MinException = e;
Shrinks++;
}
RecordTycheOutput("failed");
}
}
}
}
public string ExceptionMessage(Func<T, string> print)
{
return SampleErrorMessage(MinPCG!.ToString(MinState), print(MinT!), Shrinks, Skipped, Total);
}
}
and then in Sample you serialize the Tycherecords. Its probably better to make it more general though so that future plugins can also more easily access generated input data.
The above implementation works, I changed the Bool_Distribution test to 100 iterations
[Fact]
public void Bool_Distribution()
{
const int frequency = 10;
var expected = Enumerable.Repeat(frequency, 2).ToArray();
Gen.Bool.Select(i => i ? 1 : 0).Array[2 * frequency]
.Select(sample => Tally(2, sample))
.Sample(actual => Check.ChiSquared(expected, actual, 10), iter: 100, time: -2);
}
which returns the attached output.
I'll make a PR if desired once I thought a little more on how to refactor SampleActionWorker and SampleActionWorkerWithMetrics (?).
@pppls Sorry, I've been away and busy when I got back. Will take a good look at the weekend.
Have given a comment on the PR. I think this could be a really good general logging extension.
I'll work on the API side for this and get it in the next release. A week or two.
Released in 4.1.0. Sorry for the delay. Thanks to you both and let me know anything further we can do.