PySUS icon indicating copy to clipboard operation
PySUS copied to clipboard

UnicodeDecodeError while trying to download 'Dengue' data from SINAN

Open luabida opened this issue 3 years ago • 0 comments

Bug report

I've encountered a bug while trying to download data for 'Dengue' in 2001 to 2008.

How to reproduce

from pysus.online_data import SINAN

SINAN.download(2001, 'dengue', return_fname=True)

LOG error

>>> SINAN.download(2001, 'dengue', return_fname=True)
/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pysus/online_data/SINAN.py:62: UserWarning: Now SINAN tables are no longer split by state. Returning countrywide years
  warnings.warn("Now SINAN tables are no longer split by state. Returning countrywide years")
/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pysus/online_data/SINAN.py:87: UserWarning: Now SINAN tables are no longer split by state. Returning country table
  warnings.warn("Now SINAN tables are no longer split by state. Returning country table")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pysus/online_data/SINAN.py", line 105, in download
    filename = get_chunked_dataframe(fname, 'DBC')
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pysus/online_data/__init__.py", line 86, in get_chunked_dataframe
    df.applymap(lambda x: x.decode() if isinstance(x, bytes) else x)
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pandas/core/frame.py", line 8930, in applymap
    return self.apply(infer).__finalize__(self, "applymap")
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pandas/core/frame.py", line 8845, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pandas/core/apply.py", line 733, in apply
    return self.apply_standard()
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pandas/core/apply.py", line 857, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pandas/core/apply.py", line 873, in apply_series_generator
    results[i] = self.f(v)
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pandas/core/frame.py", line 8928, in infer
    return lib.map_infer(x.astype(object)._values, func, ignore_na=ignore_na)
  File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
  File "/home/luabida/miniconda3/envs/epigraphhub/lib/python3.9/site-packages/pysus/online_data/__init__.py", line 86, in <lambda>
    df.applymap(lambda x: x.decode() if isinstance(x, bytes) else x)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 18: invalid continuation byte

luabida avatar Oct 27 '22 19:10 luabida