MSL 4.1.0 Regressions - Blocks

[GallLeo]

The following models fail in result comparison. Status: 7ac79067 (2024-01-30)

• [x] Modelica.Blocks.Examples.Noise.ActuatorWithNoise - Reason: Due to #3727. Invalid reference result. - Updated reference files? @GallLeo Yes, should update reference result

• [x] Modelica.Blocks.Examples.Noise.DrydenContinuousTurbulence - Reason: Due to #3727 (in text of issue). Invalid reference result. - Updated reference files? @GallLeo Yes, should update reference result

• [x] Modelica.Blocks.Examples.Noise.NormalNoiseProperties - Reason: Due to #4151 remove comparison - Updated reference files? Reduced comparison signals in linked PR; #4297. @GallLeo please update reference result - @HansOlsson ~~please back-port #4151 to maint/4.1.x and link the PR here~~; Back-port #4297 which is done in #4416

• [x] Modelica.Blocks.Examples.Noise.UniformNoiseProperties - Reason: Due to #4151 remove comparison - Updated reference files? Reduced comparison signals in linked PR; #4297. @GallLeo please update reference result - @HansOlsson ~~please back-port #4151 to maint/4.1.x and link the PR here~~; Back-port #4297 which is done in #4416

[beutlich]

@HansOlsson Is this due to #4151?

@GallLeo Please add the appropriate labels for library and version.

[HansOlsson]

@HansOlsson Is this due to #4151?

@GallLeo Please add the appropriate labels for library and version.

I'm a bit lost. What is the error? Clicking on the links just give a bunch of green texts.

[beutlich]

Try https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/20240129232335/Modelica.Blocks.Examples.Noise.NormalNoiseProperties/CSVCompare/NormalNoiseProperties_report.html where the mean output error is the reason for result failure.

[HansOlsson]

Try https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/20240129232335/Modelica.Blocks.Examples.Noise.NormalNoiseProperties/CSVCompare/NormalNoiseProperties_report.html where the mean output error is the reason for result failure.

Ah, yes, the protected variable mu is now completely different due to PR #4151 - but y should be the same; so just reference update (or removing from comparison).

[beutlich]

Yes, removing from comparison might be good approach since protected anyway.

[GallLeo]

@GallLeo Please add the appropriate labels for library and version.

I'm a bit lost. What is the error? Clicking on the links just give a bunch of green texts.

Sorry, I removed the automatic browser links. They linked to the test case, not to the comparison report. Only the comparison links count.

[HansOlsson]

@GallLeo Please add the appropriate labels for library and version.

I'm a bit lost. What is the error? Clicking on the links just give a bunch of green texts.

Sorry, I removed the automatic browser links. They linked to the test case, not to the comparison report. Only the comparison links count.

Note that the first two seems to be reference updates, unrelated to #4151

[HansOlsson]

Possibly different random sequence for the first two.

[beutlich]

Possibly different random sequence for the first two.

This needs some more evidence as I am not aware on RNG changes.

[beutlich]

I run git bisect on Modelica.Blocks.Examples.Noise.ActuatorWithNoise and got:

a2cce87f7ab1a2a59f5263867c507e31241c23fd is the first bad commit

This points to #3727/#3728 and even @maltelenz' comment https://github.com/modelica/ModelicaStandardLibrary/issues/3727#issuecomment-763449287 where it is stated that the MSL 4.0.0 reference result is invalid.

[HansOlsson]

This was discussed at the latest MAP-Lib meeting (2024-05-14), but the ones that I should update already have a PR linked to this one - it is just waiting to be reviewed and merged.

[beutlich]

This was discussed at the latest MAP-Lib meeting (2024-05-14),

I would help if the outcome of this kind of discussion somehow gets back to the originating issues/PRs.

[casella]

@GallLeo everything should be ready for the new results generation

[casella]

The regression report looks good now on Modelica.Blocks.

@GallLeo there are three new examples in Modelica.Blocks.Examples:

1. Modelica.Blocks.Examples.DemoSignalCharacteristic
2. Modelica.Blocks.Examples.DemonstrateContinuousSignalExtrema
3. Modelica.Blocks.Examples.DemoSignalCharacteristic

We should generate new reference files for them as well, as they are currently missing

[casella]

The LTX server reports issues on two examples with random numbers:

• https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/TestcaseReports/Modelica.Blocks.Examples.Noise.ActuatorWithNoise/testcase_report.html
• https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/TestcaseReports/Modelica.Blocks.Examples.Noise.DrydenContinuousTurbulence/testcase_report.html

OpenModelica passes the verification with the latest supplied reference values successfully. Not sure what is going on

[maltelenz]

The LTX server reports issues on two examples with random numbers:

* https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/TestcaseReports/Modelica.Blocks.Examples.Noise.ActuatorWithNoise/testcase_report.html

* https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/TestcaseReports/Modelica.Blocks.Examples.Noise.DrydenContinuousTurbulence/testcase_report.html

OpenModelica passes the verification with the latest supplied reference values successfully. Not sure what is going on

To follow up my statement during the last MAP-Lib meeting, that System Modeler also gets the wrong result for DrydenContinuousTurbulence. Those wrong results turned out to be caused by an incorrect implementation of getInstanceName() in System Modeler.

[beutlich]

The LTX server reports issues on two examples with random numbers:

* https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/TestcaseReports/Modelica.Blocks.Examples.Noise.ActuatorWithNoise/testcase_report.html

* https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/TestcaseReports/Modelica.Blocks.Examples.Noise.DrydenContinuousTurbulence/testcase_report.html

OpenModelica passes the verification with the latest supplied reference values successfully. Not sure what is going on

Did you see https://github.com/modelica/ModelicaStandardLibrary/issues/4296#issuecomment-1925878832?

[GallLeo]

The regression report looks good now on Modelica.Blocks.

@GallLeo there are three new examples in Modelica.Blocks.Examples:

1. Modelica.Blocks.Examples.DemoSignalCharacteristic

2. Modelica.Blocks.Examples.DemonstrateContinuousSignalExtrema

3. Modelica.Blocks.Examples.DemoSignalCharacteristic

We should generate new reference files for them as well, as they are currently missing

DemoSignalCharacteristic was mentioned, twice. It should be instead:

1. Modelica.Blocks.Examples.DemonstrateSignalExtrema

[GallLeo]

The regression report looks good now on Modelica.Blocks. @GallLeo there are three new examples in Modelica.Blocks.Examples:

1. Modelica.Blocks.Examples.DemoSignalCharacteristic

2. Modelica.Blocks.Examples.DemonstrateContinuousSignalExtrema

3. Modelica.Blocks.Examples.DemoSignalCharacteristic

We should generate new reference files for them as well, as they are currently missing

DemoSignalCharacteristic was mentioned, twice. It should be instead:

1. Modelica.Blocks.Examples.DemonstrateSignalExtrema

New references have been pushed: https://github.com/modelica/MAP-LIB_ReferenceResults/commit/b5a180d4f6693042ed141ae273fab546680cd1da https://github.com/modelica/MAP-LIB_ReferenceResults/commit/e08df1ccfcd9d63e77a1efcaee4c2e2e2d5b7c13

[maltelenz]

As far as I can tell, all items have been resolved. Feel free to reopen if you disagree.

Closing.