Problems with reading mff files

[elba1212]

Description of the problem

Hi, I am trying to open data files that recorded with EGI system, with mne.io.read_raw_egi. for some recordings it works perfectly fine and for others I get an error message.

MNE version: 1.1.1 tried also 1.2.dev0 version and the error persisted. operating system: Windows 10

here is an example code and files. for sub3003 I can read the file and for sub3004 I get an error message.

Steps to reproduce

import mne

sub3003 = r"C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\samples\EL3003_SLEEP_20220221_015208.mff"
raw3003 = mne.io.read_raw_egi(sub3003)

sub3004 = r"C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\samples\EL3004_SLEEP_20220309_125216.mff"
raw3004 = mne.io.read_raw_egi(sub3004)

Link to data

subject 3003: https://www.dropbox.com/s/1n4miy31643wcm4/EL3003_SLEEP_20220221_015208.mff.zip?dl=0

subject 3004: https://www.dropbox.com/s/8ydahvu84dhv6f6/EL3004_SLEEP_20220309_125216.mff.zip?dl=0

Expected results

read the mff file smoothly

Actual results

Error message:

''' sub3004 = r"C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\EEG\EL3004\EL3004_SLEEP_20220309_125216.mff" raw = mne.io.read_raw_egi(sub3004) Reading EGI MFF Header from C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\EEG\EL3004\EL3004_SLEEP_20220309_125216.mff... Traceback (most recent call last):

File "C:\Users\paltiele.WISMAIN\AppData\Local\Temp\ipykernel_12524\2134038754.py", line 2, in raw = mne.io.read_raw_egi(sub3004)

File "", line 12, in read_raw_egi

File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egi.py", line 157, in read_raw_egi exclude, preload, channel_naming, verbose)

File "", line 12, in _read_raw_egi_mff

File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 390, in _read_raw_egi_mff preload, channel_naming, verbose)

File "", line 12, in init

File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 404, in init egi_info = _read_header(input_fname)

File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 212, in _read_header mff_hdr = _read_mff_header(input_fname)

File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 93, in _read_mff_header list(epochs['last_samps'])))

RuntimeError: EGI epoch first/last samps could not be parsed: [0] [26586364] '''

Additional information

Platform: Windows-10-10.0.19041-SP0 Python: 3.7.13 (default, Mar 28 2022, 08:03:21) [MSC v.1916 64 bit (AMD64)] Executable: C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\python.exe CPU: Intel64 Family 6 Model 140 Stepping 1, GenuineIntel: 8 cores Memory: 31.7 GB

mne: 1.2.dev0 numpy: 1.21.6 {MKL 2022.1-Product with 4 threads} scipy: 1.7.3 matplotlib: 3.5.3 {backend=module://matplotlib_inline.backend_inline}

sklearn: 1.0.2 numba: 0.56.2 nibabel: Not found nilearn: Not found dipy: Not found openmeeg: Not found cupy: Not found pandas: 1.3.5 pyvista: Not found pyvistaqt: Not found ipyvtklink: Not found vtk: Not found qtpy: Not found ipympl: Not found pyqtgraph: Not found pooch: v1.6.0

mne_bids: Not found mne_nirs: Not found mne_features: Not found mne_qt_browser: Not found mne_connectivity: Not found mne_icalabel: Not found

[welcome[bot]]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[scott-huberty]

Hi @elba1212 - can you zip the .mff files before you upload them to dropbox? Do you have any smaller files that you can share and which also cause the issue? The files you shared appear to be so large that dropbox won't even let me download them.

EDIT: It looks like the signal.bin file (this is the primary data file) inside the .mff directory, is 25g!

[scott-huberty]

@elba1212 I also just wanted to highlight that the patient video stream is included in your file. If you are not allowed to disclose patient information you can remove the .mov files inside the .mff directory (which may also slightly decrease your overall file size).

Thanks!

[larsoner]

One of those .bin files is 26.5GB for example -- I'm guessing this as a 32-bit int problem. I should be able to check this week hopefully

[elba1212]

Hi @scott-huberty and @larsoner, Thank you for your reply!

I tried replicating the error with some of my smaller files, but it only replicates with my large files. (when I try to read the smaller files for some of them, I get a different error: IndexError when it reads the event channel).

I removed the videos, zipped the mff files, and updated the links in the original message as you suggested. Thanks, Ella

[drammock]

@scott-huberty @larsoner there is an off-by-one error in subject 3004:

ipdb> p n_samps_epochs
26586364
ipdb> p n_samps_block
26586365

I can't access subject 3003, Dropbox says "zip files are supported but something went wrong"

[larsoner]

We have an off-by-one-bug check elsewhere:

https://github.com/mne-tools/mne-python/blob/706aea1eb723801c43ae5fc5f68bbb7b53184c29/mne/io/egi/egimff.py#L559-L563

I wonder if we need to introduce one here, too.

[elba1212]

I corrected the "3003" link; it works now. Anyway, this is the file that I can successfully read.

If you need more examples of problematic files, tell me.