Converting Shiny weather data with the Metadata processor: bugs and issues

[CsillaVG]

Hi Team,

There comes my concise write-up. Thanks for all the work! Best, Csilla

Shiny weather data offers four output formats, out of which two could be relevant to UMEP.

1 .CSV

At the moment, under all my os-QGIS variations, loading the csv file is not possible. I receive the error of "If invalid data was detected such as strings or other non-numrical characters, these data points could result in that the MetdataProcessor will fail to create your UMEP-formatted inputdata." and the interface remains locked (without the possibility to assign the column names to various meteorological parameters). This is most likely caused my the formatting of the file that starts with 11 lines of header lines that begin with a hash sign. Unfortunately, setting the 'Number of header lines' in the interface to 12 or higher does not resolve the issue. See attached screenshot.

Attached is a csv file for reference: Chicago2020LT.csv

2 .EPW

The epw conversion works, apart from the hard-coded year of 1985 (noted earlier). While one could replace the year in the resultant file with the help of a text editor, the problem becomes more complex when the supplied Actual Meteorological Year (AMY) is a leap year. Put simply, the current code results in duplicate datetime labels, when leap year is fed in, eventually, causing a 24-hour shift in the data following the leap day. Fixing this can be challenging to those without solid Excel or scripting skills.

Attached is an UMEP-converted epw file for reference: Chicago2020LTepw.txt

[biglimp]

Thanks Csilla, Your first point is actually a bug that I just found will fix this. The 1985 hardcode should noew be fix (in the development release).

[biglimp]

Both issues should now be fixed. Check out new development release.

[CsillaVG]

Fantastic, thank you!

[CsillaVG]

The attempt to install the development release lands in a SyntaxError: (f-string: unmatched '[')

Here is the full error message :: Couldn't load plugin 'UMEP-SuPy-QGIS3' due to an error when calling its classFactory() method

SyntaxError: f-string: unmatched '[' Traceback (most recent call last): File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 426, in _startPlugin plugins[packageName] = package.classFactory(iface) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/init.py", line 42, in classFactory from .UMEP import UMEP File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 966, in _import mod = _builtin_import(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/UMEP.py", line 68, in from .suews_database_manager.suews_database_manager import suews_database_manager File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 966, in _import mod = _builtin_import(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/suews_database_manager/suews_database_manager.py", line 56, in from .utilities.suews_SS import setup_SUEWS_SS_creator File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 966, in _import mod = _builtin_import(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/suews_database_manager/utilities/suews_SS.py", line 165 QMessageBox.information(None, 'Succesful', f'New edit {spartacus_dict['Name']}, {spartacus_dict['Origin']} added to your local database') ^^^^ SyntaxError: f-string: unmatched '['

Environment :: macOS Ventura 13.2 QGIS version: 3.40.1-Bratislava Python version: 3.11.11 UMEP 4.3.2 UMEP for processing 2.0.35 SuPy version: 2024.5.24.dev

[biglimp]

Try now.

[CsillaVG]

Unfortunately, I got the same exact error. I'll try to install QGIS into a 3.12 python environment and see if this helps with the issue, or just opens a new can of worms... It does work under Windows 11.

[biglimp]

Ok, Let us know if you find a solution for Mac.

[CsillaVG]

Installing QGIS into a Python 3.12 environment on MacOS indeed solves the above SyntaxError. The development release of UMEP, with the Metadata processor, runs. However, 1985 is still wrote into the file. This is the case with Windows as well (I forgot to check the results the other day). To double check the results, UMEPdev was downloaded and installed once again, and file conversion attempted multiple times. Still, no change regarding 1985.

Off: This set up comes with the unresolved issue of the faulthandler that becomes disabled due to conflicts between QGIS and MacOS (esp. M1), see discussions e.g. here and here. Nevertheless, one can still use QGIS and UMEP with minor caveats.

[biglimp]

I cannot replicate your error regarding 1985. I keep the year from the epw-file if I tick in the box "Keep year..." Can you send me your epw-file so that I could try it.

[CsillaVG]

Sure. Here it comes. ERA5-ERA5n_2020LTtg.zip The file was generated by Shiny Weather app. I have only added ground temperatures to the header. Attaching the original epw as well. ERA5-ERA5n_2020LT.zip

[biglimp]

Tried with your file, I got the correct year...

%iy  id  it imin   Q*      QH      QE      Qs      Qf    Wind    RH     Td     press   rain    Kdn    snow    ldown   fcld    wuh     xsmd    lai_hr  Kdiff   Kdir    Wd
2020 1 1 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.50000 89.00 7.00 100.80 0.00 0.00 -999.00 334.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 231.00
2020 1 2 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.90000 87.00 7.10 100.80 0.00 0.00 -999.00 338.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 226.00
2020 1 3 0 -999.00 -999.00 -999.00 -999.00 -999.00 1.10000 88.00 7.10 100.80 0.00 0.00 -999.00 345.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 210.00
2020 1 4 0 -999.00 -999.00 -999.00 -999.00 -999.00 1.20000 88.00 6.80 100.70 0.00 0.00 -999.00 343.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 188.00
2020 1 5 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.60000 88.00 6.60 100.70 0.00 0.00 -999.00 341.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 171.00
2020 1 6 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.50000 88.00 6.50 100.70 0.00 0.00 -999.00 338.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 160.00
2020 1 7 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.80000 90.00 6.10 100.80 0.00 0.00 -999.00 329.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 192.00
2020 1 8 0 -999.00 -999.00 -999.00 -999.00 -999.00 1.40000 89.00 6.20 100.80 0.00 0.00 -999.00 341.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 209.00
...

[CsillaVG]

It must be a midwinter magic! :) But seriously. I am still getting 1985s written into the file. Both on my MacOX and Windows machines. Attached is the screen capture from the mac. Happy holidays!

https://github.com/user-attachments/assets/aca301a2-fe55-4afe-8a5f-9b2c0b31f20d

[biglimp]

Truly strange... Try uninstall and reinstall again by physically removing the UMEP plugin from your plugin library.

In the file metdata_processor.py your lines 260-270 should look like this:

    def epw2umep(self, met_old):
        met_new = np.zeros((met_old.shape[0], 24)) - 999

        # yyyy
        if self.yyKeep == 0:
            met_new[:, 0] = 1985
            met_new[met_old.shape[0] - 1, 0] = 1986
        else:
            met_new[:, 0] = met_old[:, 0]

        # hour
...

[CsillaVG]

I did that.

1. Downloaded yet again the development version.
2. Located the installation folder.
3. Observed the UMEP-SuPy-QGIS3 folder to be completely removed then rebuilt during the un- and re-installation process.
4. Restarted QGIS between the un- and re-installations.
5. Checked the relevant section of the code: it is as it should be.

Still, I cannot make UMEP to retain the year/formatting** of the data.

NOTE: Beforehand, I have removed all additional QGIS installs, except the LT release installed in another conda environment. To make sure it doesn't interfere with the one I use, I have uninstalled there all UMEP plugins.

** There is a loop (repetition of datetime values) over February 29 in the converted 2020 epw file.

[biglimp]

I have no clue why it does not work for you. I can also test on an Ubuntu installation next week but until then I suggest you just manually change the year column. You can also try to set in two print commands on lines 266 and 268 (see below) and see where the code goes.

    def epw2umep(self, met_old):
        met_new = np.zeros((met_old.shape[0], 24)) - 999

        # yyyy
        if self.yyKeep == 0:
            met_new[:, 0] = 1985
            met_new[met_old.shape[0] - 1, 0] = 1986
            print('1985')
        else:
            met_new[:, 0] = met_old[:, 0]
            print('User spec. year')

        # hour
...

You need to have your Python console open to see the prints

[biglimp]

Cleaning up old issues. I could not replicate your error on Ubuntu. Still same issue with 1895?