ci failing with pandas version 2.2.0

[MartinRaetz]

Describe the bug Moving from pandas 2.1.x to 2.2.x lets the unit-tests fail for .mos and .epw files using DWD historical data. The newer version of pandas comes with the new version of Wetterdienst==0.73.0

To Reproduce run CI with pandas 2.2.0

Expected behavior CI not failing

Bug Fix Suggestion

• [ ] The problem occurs with variables that have to be passed-through to avoid double interpolating.
• [ ] Also get rid of all the new deprecation warnings.
• [ ] Delete the warning for new pandas version in init.py

[FWuellhorst]

related to #54 ? If so, one todo fixed by #54.

[MartinRaetz]

#54 solved deprecation warning "H" instead of "h" for hourly.

[MartinRaetz]

Out of interest, can you confirm that your environment with the latest version of pandas passes the tests? Since the newer pandas in the CI resulted in changing values in the result files.

If they pass, can you state your pandas version please? @FWuellhorst

[FWuellhorst]

I get the following tests which fail: test_output_epw and mos and test_DWD as parameterized is not found/installed. I assume this is optional . The error for epw:

Last 1000 characters don't match!
999.0,999.0,99.0
2022,12,31,20,0,?,14.8,6.0,55.5,98195.0,9999.0,9999.0,361.9,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,21,0,?,14.8,5.8,54.9,98254.2,9999.0,9999.0,363.2,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,22,0,?,15.0,5.9,54.4,98293.3,9999.0,9999.0,361.1,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,23,0,?,15.0,5.6,53.4,98326.7,9999.0,9999.0,331.8,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,24,0,?,14.6,4.8,51.6,98364.2,9999.0,9999.0,355.9,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,6.2,6.2,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
 != 999.0,999.0,99.0
2022,12,31,20,0,?,14.8,6.0,55.5,98195.0,9999.0,9999.0,361.4,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,21,0,?,14.8,5.8,54.9,98254.2,9999.0,9999.0,361.9,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,22,0,?,15.0,5.9,54.4,98293.3,9999.0,9999.0,363.2,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,23,0,?,15.0,5.6,53.4,98326.7,9999.0,9999.0,361.1,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,10.0,10.0,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0
2022,12,31,24,0,?,14.6,4.8,51.6,98364.2,9999.0,9999.0,331.8,0.0,0.0,0.0,999999.0,999999.0,999999.0,9999.0,999.0,999.0,6.2,6.2,9999.0,99999.0,9.0,999999999.0,999.0,1.0,999.0,99.0,999.0,999.0,99.0

My version is 2.2.1

[MartinRaetz]

Thanks! So this seems to be similar/equal to the problem in the CI. Knowing that, I cannot tell you, if you can trust the results if you use pandas 2.2.x.

[FWuellhorst]

Mhm, good point. It seems that the last values are wrong, somehow shifted and the last value inserted is different, if that helps:

[MartinRaetz]

My current plan of action is hoping it will fix with future versions. If not, this will require actions from our side, which I currently find hard to pinpoint.

Possibly some pandas shifting has changed. The last value is different, as this is probably f-filled.

[FWuellhorst]

Good plan. I added a warning in #55