UMEP icon indicating copy to clipboard operation
UMEP copied to clipboard
verified publisher icon UMEP-dev

LQf Python error

Open S9LOU opened this issue 4 years ago • 3 comments

The bug We ran the LQF processor with local data("China Shanghai") following its annual (https://umep-docs.readthedocs.io/en/latest/OtherManuals/LQF_Manual.html#lqf-manual) . However, it failed.

  1. The first bug is
    addQuotes(x): return "'" + x + "'" ： type error . Then we change the source python file to addQuotes(x): return "'" + str(x) + "'".
  2. After changing RegionalParameters.py, we got the other error message as below. Exception: The LQF database contains no weekday building cycles for: {NULL}

However, the input database contains "China_Shanghai" in both WeeklyBuildingCylcles and World table. Additionally, we have tried it on another computer with QGIS Version 3.10.13, but also got the same error. The log and the data have been attached below. We have used the LQf processor on QGIS2 version based on those data in 2018 and it succeeded. Is there anything wrong with the processor or our data?

Desktop:

  • OS: Win10
  • QGIS Version 3.16.8-Hannover (longterm release)
  • UMEP Version 3.17.3
  • SuPy Version 2021.5.26
  • Python Version:3.7.0
  • Python Path: C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\SUEWSPrepare/Modules D:/OSGeo4W64/apps/qgis-ltr/./python C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins D:/OSGeo4W64/apps/qgis-ltr/./python/plugins D:\OSGeo4W64\apps\Python37 D:\OSGeo4W64\apps\Python37\Scripts D:\OSGeo4W64\bin\python37.zip D:\OSGeo4W64\apps\Python37\DLLs D:\OSGeo4W64\apps\Python37\lib D:\OSGeo4W64\bin C:\Users\ax\AppData\Roaming\Python\Python37\site-packages D:\OSGeo4W64\apps\Python37\lib\site-packages D:\OSGeo4W64\apps\Python37\lib\site-packages\win32 D:\OSGeo4W64\apps\Python37\lib\site-packages\win32\lib D:\OSGeo4W64\apps\Python37\lib\site-packages\Pythonwin C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python

Screenshots image When press the button Run model, an error occurs below as logs.

Logs

  1. first one
TypeError: can only concatenate str (not "QVariant") to str 
Traceback (most recent call last):
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
    self.model.run()
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
    self.setupAndRun(self.config['startDates'], self.config['endDates'])
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 276, in setupAndRun
    regPar.injectAttributedOutputLayer(os.path.join(self.downscaledPath, al['file']), al['EPSG'], outFeatIds)
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 197, in injectAttributedOutputLayer
    self.extractPropertiesForCountries(con, countries)
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 210, in extractPropertiesForCountries
    attrs = "SELECT * FROM attributes WHERE id IN " + "(" + ','.join(map(addQuotes, countries)) + ") ORDER BY as_of_year ASC"
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 15, in addQuotes
    def addQuotes(x): return "'" + x + "'"
TypeError: can only concatenate str (not "QVariant") to str
  1. second one
2021-07-02T11:15:55     WARNING    Traceback (most recent call last):
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
              self.model.run()
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
              self.setupAndRun(self.config['startDates'], self.config['endDates'])
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 276, in setupAndRun
              regPar.injectAttributedOutputLayer(os.path.join(self.downscaledPath, al['file']), al['EPSG'], outFeatIds)
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 197, in injectAttributedOutputLayer
              self.extractPropertiesForCountries(con, countries)
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 218, in extractPropertiesForCountries
              raise Exception('The LQF database contains no weekday building cycles for: ' + str(diffs))
             Exception: The LQF database contains no weekday building cycles for: {NULL}

Data Since GitHub didn't support uploading our filetype, the dataset online link is: https://cloud.tsinghua.edu.cn/d/0bf54aea37da46589226/
The zip file is : input.zip

It includes the LQFDatabase_V1-1.sqlite and Shanghai Pop shapefile,

S9LOU avatar Jul 02 '21 07:07 S9LOU

Were you able to go through the London tutorial without any errors?

https://umep-docs.readthedocs.io/projects/tutorial/en/latest/Tutorials/LQF.html#lqf

biglimp avatar Jul 02 '21 07:07 biglimp

Hi, thank you for your kind reply. We could go through the London tutorial without any errors, indicating the LQf tool worked successfully. However, when we double checked our data and python code error, we found some interesting problems with the data type.

  1. Firstly, the tool couldn’t deal with the null data in our shapefiles. “TypeError: can only concatenate str (not "QVariant") to str” in above issue, is probably due to that the “admin” field of area_allocations shape file contains null value. However, we have checked the “admin” field of our data has the same datatype(Qstring) as area_allocations shape file extracted from the sample data. Therefore, we are not sure whether the null value is QVariant type in the python processing or whether the null value leads to the python error. It’s impossible for us to manually correct null type since the “admin” field is generated from the preprocess session automatically.

  • The original population data is attached below. shanghai_pop_15mar_2015_SEDAC.zip

  • The area_allocations shape file generated from pop data is also attached. Its admin field contains China_Shanghai, China_Zhaejing,China_Jiangsu as well as Null (sea). area_allocations.zip

  • The error log is attached below.

`TypeError: can only concatenate str (not "QVariant") to str 
Traceback (most recent call last):
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
    self.model.run()
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
    self.setupAndRun(self.config['startDates'], self.config['endDates'])
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 276, in setupAndRun
    regPar.injectAttributedOutputLayer(os.path.join(self.downscaledPath, al['file']), al['EPSG'], outFeatIds)
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 197, in injectAttributedOutputLayer
    self.extractPropertiesForCountries(con, countries)
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 210, in extractPropertiesForCountries
    attrs = "SELECT * FROM attributes WHERE id IN " + "(" + ','.join(map(addQuotes, countries)) + ") ORDER BY as_of_year ASC"
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 15, in addQuotes
    def addQuotes(x): return "'" + x + "'"
TypeError: can only concatenate str (not "QVariant") to str
  • Besides its admin, area_allocations shapefile’s pop field contains null, leading to a similar error. However, it also shows double type in the QGIS same with the sample data. The error log is attached below.
An error has occurred while executing Python code: 

TypeError: unsupported operand type(s) for *: 'float' and 'QVariant' 
Traceback (most recent call last):
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 143, in na_arithmetic_op
    result = expressions.evaluate(op, left, right)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\computation\expressions.py", line 233, in evaluate
    return _evaluate(op, op_str, a, b)  # type: ignore
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\computation\expressions.py", line 119, in _evaluate_numexpr
    result = _evaluate_standard(op, op_str, a, b)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\computation\expressions.py", line 68, in _evaluate_standard
    return op(a, b)
TypeError: unsupported operand type(s) for *: 'float' and 'QVariant'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
    self.model.run()
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
    self.setupAndRun(self.config['startDates'], self.config['endDates'])
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 382, in setupAndRun
    energyUse = regPar.getEnergyUse(areas.index, d_lookup.year)
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 396, in getEnergyUse
    return data['kwh_year'] * data['resPop']/data['population'] # Disaggregate energy use by building borne population
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\common.py", line 65, in new_method
    return method(self, other)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\__init__.py", line 343, in wrapper
    result = arithmetic_op(lvalues, rvalues, op)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 190, in arithmetic_op
    res_values = na_arithmetic_op(lvalues, rvalues, op)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 150, in na_arithmetic_op
    result = masked_arith_op(left, right, op)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 92, in masked_arith_op
    result[mask] = op(xrav[mask], yrav[mask])
TypeError: unsupported operand type(s) for *: 'float' and 'QVariant'
  • After we deleted those grids containing the null value of pop or admin, it worked. However, it’s impossible for us to delete all the null value in a larger area. Moreover, LQf with QGIS2 version could process exactly the same grid without any error.
  • If the LQf still can deal with the null value now? If yes, why it failed this time? If not, is there any suggested way for us to correct our data rather than simply deleted grids.
  1. Secondly, the tool requires the date of daily temperature csv file must be monotonic increasing or decreasing. Our daily temperature (2014-11-30 to 2015-11-30) have its time zone field starting from 335 to 365, and following 1 to 334.
    That’s weird since when we used to put the same daily temperature csv file into the LQf with QGIS2 version it worked. Is it still allowed to process such daily temperature file?
  • The temperature data is attached below.

dailyTemperature_2015_Shanghai.zip

  • The error log is attached below.
`An error has occurred while executing Python code: 

ValueError: index must be monotonic increasing or decreasing 
Traceback (most recent call last):
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
    self.model.run()
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
    self.setupAndRun(self.config['startDates'], self.config['endDates'])
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 386, in setupAndRun
    temperatures.append(dailyT.getTemp(d_lookup_temperature, timeStepDuration)[0]) # Mean daily temperature
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\DailyTemperature.py", line 59, in getTemp
    return self.temperature.getValueForDateTime(requestDateTime, timeBinDuration)
  File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\DataManagement\GenericAnnualSampler.py", line 384, in getValueForDateTime
    minDiff = dsts[dstDates].index.asof(localLookupTimeBin.date())
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 4356, in asof
    loc = self.get_loc(label, method="pad")
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2905, in get_loc
    indexer = self.get_indexer([key], method=method, tolerance=tolerance)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2994, in get_indexer
    indexer = self._get_fill_indexer(target, method, limit, tolerance)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 3034, in _get_fill_indexer
    indexer = self._get_fill_indexer_searchsorted(target, method, limit)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 3057, in _get_fill_indexer_searchsorted
    indexer[nonexact] = self._searchsorted_monotonic(target[nonexact], side)
  File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 5050, in _searchsorted_monotonic
    raise ValueError("index must be monotonic increasing or decreasing")
ValueError: index must be monotonic increasing or decreasing
`

S9LOU avatar Jul 06 '21 09:07 S9LOU

Does you python suggested fix work? I.e. did you try it?

suegrimmond avatar Aug 11 '21 09:08 suegrimmond