mathnet-numerics icon indicating copy to clipboard operation
mathnet-numerics copied to clipboard

Incorrect Values for MovingStatistics StandardDeviation

Open thoblerone opened this issue 1 year ago • 0 comments

I created a simple test case to get comfortable with the MovingStatistics class and surprisingly stubled into a calculation error:

public void TryingOutMathNetStatisticsOptions()
{
    var dataList = new List<double>()
    {
        1, 1, 1, 1, 1,
        2, 2, 2, 2, 2,
        3, 3, 3, 3, 3,
        4, 4, 4, 4, 4,
        3, 3, 3, 3, 3,
        2, 2, 2, 2, 2,
        1, 1, 1, 1, 1
    };

    var statistics = new MathNet.Numerics.Statistics.MovingStatistics(5);
    var msCount = new List<double>();
    var msMean = new List<double>();
    var msStdDev = new List<double>();
    var msPopStdDev = new List<double>();

    foreach (var data in dataList)
    {
        statistics.Push(data);

        msCount.Add(statistics.Count);
        msMean.Add(statistics.Mean);
        msStdDev.Add(statistics.StandardDeviation);
        msPopStdDev.Add(statistics.PopulationStandardDeviation);

        Trace.WriteLine($"{data}\t{statistics.Count}\t{statistics.Mean}\t"+
                        $"{statistics.StandardDeviation}\t{statistics.PopulationStandardDeviation}");
    }

    for (int i = 4; i < dataList.Count; i += 5)
    {
        Assert.That(msStdDev[i], Is.Not.NaN);
        Assert.That(msPopStdDev[i], Is.Not.NaN);
    }
}

The data set was designed to be used with a window width of 5 with 5 consecutive identical values followed by the next set of 5. As both standard deviations step down to zero, for the later data the calculation behind the statistics.Push(data) call will accumulate a small rounding error in the magnitude of 1E-15 that will finally yield in a negative variance.

thoblerone avatar Oct 22 '24 13:10 thoblerone