TypeError: control character 'delimiter' cannot be a newline

[danial2008]

🐛 Bug Description

Traceback (most recent call last): File "D:\dev\anaconda3\envs\qlib\lib\site-packages\IPython\core\interactiveshell.py", line 3398, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in <cell line: 1> e.calendar(start_time='2010-01-01', end_time='2017-12-31', freq='day') File "D:\dev\pyqlib\qlib\qlib\data\data.py", line 90, in calendar _calendar, _calendar_index = self._get_calendar(freq, future) File "D:\dev\pyqlib\qlib\qlib\data\data.py", line 171, in _get_calendar _calendar = np.array(self.load_calendar(freq, future)) File "D:\dev\pyqlib\qlib\qlib\data\data.py", line 657, in load_calendar backend_obj = self.backend_obj(freq=freq, future=future).data File "D:\dev\pyqlib\qlib\qlib\data\storage\file_storage.py", line 129, in data H["c"][key] = self._read_calendar() File "D:\dev\pyqlib\qlib\qlib\data\storage\file_storage.py", line 111, in _read_calendar for x in np.loadtxt(fp, str, skiprows=skip_rows, max_rows=n_rows, delimiter="\n", encoding="utf-8") File "D:\dev\anaconda3\envs\qlib\lib\site-packages\numpy\lib\npyio.py", line 1301, in loadtxt arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter, File "D:\dev\anaconda3\envs\qlib\lib\site-packages\numpy\lib\npyio.py", line 1006, in _read next_arr = _load_from_filelike( TypeError: control character 'delimiter' cannot be a newline (\r or \n).

To Reproduce

Steps to reproduce the behavior: 1.download the data by get_data.py firstly 2.write a simple demo qlib.init(provider_uri="D:/dev/pyqlib/data/cn_data/")

from qlib.data import D

D.calendar(start_time='2010-01-01', end_time='2017-12-31', freq='day')[:5]

Expected Behavior

Screenshot

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information and paste them here directly.

• Qlib version: 0.8.6
• Python version: 3.8
• OS (Windows, Linux, MacOS):Windows
• Commit number (optional, please provide it if you are using the dev version):

Additional Notes

[wwwadx]

Same issue encountered!

[SunsetWolf]

This issue is caused by an upgrade of the qlib dependent library and is currently being fixed.

[Viperccc]

Same issue encountered!

[SunsetWolf]

This bug has been fixed in PR 1141.

[LiquidAmmonia]

Is this PR merged into main? I pulled the latest main branch but this bug remains.

[HuihuiWu-Microsoft]

I have the same issue with latest code.

[rmallof]

https://github.com/numpy/numpy/blob/40230640b2e5c611b0d1fe913cf3285c35848026/numpy/core/src/multiarray/textreading/readtext.c#L116

[XixuHu]

Create virtual environment using Python=3.7 and install the qlib package using the codes from https://github.com/microsoft/qlib/blob/main/examples/workflow_by_code.ipynb can run properly.

[rovedream]

同样问题

[you-n-g]

This has been fixed in the latest code https://github.com/microsoft/qlib/blob/c34051c1ced920e365f8167e2bba5302a6f08318/qlib/data/storage/file_storage.py#L109

[rmallof]

Thank you for the quick resolution - it's working for me

[lichoil]

encountered the same issue. pyqlib 0.8.6 python 3.8 os: linux

I've tried to install a new condo environment, but issue remains.

[lichoil]

[880:MainThread](2022-07-24 15:51:15,217) INFO - qlib.Initialization - [config.py:413] - default_conf: client. [880:MainThread](2022-07-24 15:51:15,219) INFO - qlib.Initialization - [init.py:74] - qlib successfully initialized based on client settings. [880:MainThread](2022-07-24 15:51:15,220) INFO - qlib.Initialization - [init.py:76] - data_path={'__DEFAULT_FREQ': PosixPath('/home/choil/.qlib/qlib_data/cn_data')} [880:MainThread](2022-07-24 15:51:15,220) INFO - qlib.workflow - [expm.py:315] - <mlflow.tracking.client.MlflowClient object at 0x7f5d10e0a940> [880:MainThread](2022-07-24 15:51:15,235) INFO - qlib.workflow - [exp.py:257] - Experiment 1 starts running ... [880:MainThread](2022-07-24 15:51:15,325) INFO - qlib.workflow - [recorder.py:293] - Recorder c36a78bf0d8d4c3398ab4654c3db3ad6 starts running under Experiment 1 ... Please install necessary libs for CatBoostModel. ModuleNotFoundError. XGBModel is skipped(optional: maybe installing xgboost can fix it). ModuleNotFoundError. PyTorch models are skipped (optional: maybe installing pytorch can fix it). [880:MainThread](2022-07-24 15:51:15,505) INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | waiting async_log Done [880:MainThread](2022-07-24 15:51:15,506) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[TypeError: control character 'delimiter' cannot be a newline (\r or \n).]. File "/home/choil/anaconda3/envs/qlib2/bin/qrun", line 8, in sys.exit(run()) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/workflow/cli.py", line 72, in run fire.Fire(workflow) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/workflow/cli.py", line 66, in workflow recorder = task_train(config.get("task"), experiment_name=experiment_name) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/model/trainer.py", line 127, in task_train _exe_task(task_config) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/model/trainer.py", line 46, in _exe_task dataset: Dataset = init_instance_by_config(task_config["dataset"], accept_types=Dataset) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/utils/init.py", line 457, in init_instance_by_config return klass(**cls_kwargs, **kwargs) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/init.py", line 115, in init self.handler: DataHandler = init_instance_by_config(handler, accept_types=DataHandler) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/utils/init.py", line 457, in init_instance_by_config return klass(**cls_kwargs, **kwargs) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/contrib/data/handler.py", line 171, in init super().init( File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 434, in init super().init(instruments, start_time, end_time, data_loader, **kwargs) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 97, in init self.setup_data() File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 570, in setup_data super().setup_data(**kwargs) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 141, in setup_data self._data = lazy_sort_index(self.data_loader.load(self.instruments, self.start_time, self.end_time)) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 135, in load { File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 136, in grp: self.load_group_df(instruments, exprs, names, start_time, end_time, grp) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 211, in load_group_df df = D.features( File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 1184, in features return DatasetD.dataset(instruments, fields, start_time, end_time, freq, inst_processors=inst_processors) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 905, in dataset instruments_d = self.get_instruments_d(instruments, freq) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 514, in get_instruments_d instruments_d = Inst.list_instruments(instruments=instruments, freq=freq, as_list=False) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 695, in list_instruments cal = Cal.calendar(freq=freq) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 90, in calendar _calendar, _calendar_index = self._get_calendar(freq, future) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 171, in _get_calendar _calendar = np.array(self.load_calendar(freq, future)) File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 657, in load_calendar backend_obj = self.backend_obj(freq=freq, future=future).data File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/storage/file_storage.py", line 129, in data H["c"][key] = self._read_calendar() File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/storage/file_storage.py", line 111, in _read_calendar for x in np.loadtxt(fp, str, skiprows=skip_rows, max_rows=n_rows, delimiter="\n", encoding="utf-8") File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1308, in loadtxt arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter, File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1006, in _read next_arr = _load_from_filelike( TypeError: control character 'delimiter' cannot be a newline (\r or \n).

[you-n-g]

The previous solution does not work for high-frequent data. It is fixed here.

https://github.com/microsoft/qlib/pull/1169/commits/54231b1ea7ffb1d91435b8262dbfd00af770a1fe

It will be merged to the main branch in the short future.

[z13670]

It works! Thank you!!!!!!!!!!

[selimarikan]

This has been fixed in the latest code

https://github.com/microsoft/qlib/blob/c34051c1ced920e365f8167e2bba5302a6f08318/qlib/data/storage/file_storage.py#L109

Confirming that manually adding this line works with Python 3.8. I have installed the library using pip and was having the same issue before.