SpikeInterface
Error when loading template_metrics extension
I've been getting some errors when computing/loading extensions from a SortingAnalyzer containig a template_metrics extension. I calculate extensions from a dict as follows:
phy_exts = {"random_spikes": {},
"waveforms": {},
"templates":{},
"noise_levels": {},
"principal_components": {"n_components":5, "mode":"by_channel_local"},
"template_similarity": {"method": "cosine_similarity"},
"spike_amplitudes": {},
"quality_metrics": {},
"template_metrics": {}}
analyzer_obj.compute(phy_exts)
After successfully computing waveforms, fitting PCAs and projecting waveforms, I get these errors:
c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1924: UserWarning: Found no run_info file for template_metrics, extension should be re-computed.
warnings.warn(f"Found no run_info file for {self.extension_name}, extension should be re-computed.")
c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1931: UserWarning: Found no run_info file for template_metrics, extension should be re-computed.
warnings.warn(f"Found no run_info file for {self.extension_name}, extension should be re-computed.")
c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:2004: UserWarning: Found no data for template_metrics, extension should be re-computed.
warnings.warn(f"Found no data for {self.extension_name}, extension should be re-computed.")
After which it will proceed to calculate spike_amplitudes successfully but then hang indefinitely.
I couldn't manually delete the folder with the waveforms extension, so code seemed to have hang there.
Hello, Thanks for posting the issue. I can't reproduce the error. Here was my attempt to reproduce:
import spikeinterface.full as si
rec, sort = si.generate_ground_truth_recording()
analyzer_obj = si.create_sorting_analyzer(recording=rec, sorting=sort, format="binary_folder", folder="test_analyzer")
phy_exts = {"random_spikes": {},
"waveforms": {},
"templates":{},
"noise_levels": {},
"principal_components": {"n_components":5, "mode":"by_channel_local"},
"template_similarity": {"method": "cosine_similarity"},
"spike_amplitudes": {},
"quality_metrics": {},
"template_metrics": {}}
analyzer_obj.compute(phy_exts)
analyzer_2 = si.load_sorting_analyzer("test_analyzer")
analyzer_2.compute(phy_exts)
I get the warning, but the spike amplitudes are successfully computed, then it computes the pc_metrics. It'd be good if I could reproduce your problem. Do you know which extensions were successfully computed for the analyzer that you're loading?
I'll look into the warning, which seems like it shouldn't be there.
Hey Chris are you on windows? Maybe I should test. There can some folder release issues with Windows that doesn't happen on Mac/Linux (which I think you mostly test from). I'm not free until tomorrow, but happy to also troubleshoot then unless you have a windows computer to work on :)
If I disable computing of both template_metrics and quality_metrics, code runs fine. If I disable only template_metrics, code still hangs indefinitely but I get this error at some point:
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\qualitymetrics\misc_metrics.py:908](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/qualitymetrics/misc_metrics.py:908): UserWarning: Some units have too few spikes : amplitude_cutoff is set to NaN
warnings.warn(f"Some units have too few spikes : amplitude_cutoff is set to NaN")
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\numpy\core\_methods.py:206](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/numpy/core/_methods.py:206): RuntimeWarning: Degrees of freedom <= 0 for slice
ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\numpy\core\_methods.py:163](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/numpy/core/_methods.py:163): RuntimeWarning: invalid value encountered in divide
arrmean = um.true_divide(arrmean, div, out=arrmean,
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\numpy\core\_methods.py:198](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/numpy/core/_methods.py:198): RuntimeWarning: invalid value encountered in divide
ret = ret.dtype.type(ret / rcount)
Provided cone snippet throws warnings but indeed runs until pc_metric calculation, but is interrupted by the following error:
---------------------------------------------------------------------------
PermissionError Traceback (most recent call last)
Cell In[1], [line 19](vscode-notebook-cell:?execution_count=1&line=19)
[16](vscode-notebook-cell:?execution_count=1&line=16) analyzer_obj.compute(phy_exts)
[18](vscode-notebook-cell:?execution_count=1&line=18) analyzer_2 = si.load_sorting_analyzer("test_analyzer")
---> [19](vscode-notebook-cell:?execution_count=1&line=19) analyzer_2.compute(phy_exts)
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1328, in SortingAnalyzer.compute(self, input, save, extension_params, verbose, **kwargs)
[1326](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1326) params_, job_kwargs = split_job_kwargs(kwargs)
[1327](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1327) assert len(params_) == 0, "Too many arguments for SortingAnalyzer.compute_several_extensions()"
-> [1328](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1328) self.compute_several_extensions(extensions=input, save=save, verbose=verbose, **job_kwargs)
[1329](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1329) elif isinstance(input, list):
[1330](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1330) params_, job_kwargs = split_job_kwargs(kwargs)
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1443, in SortingAnalyzer.compute_several_extensions(self, extensions, save, verbose, **job_kwargs)
[1441](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1441) for extension_name in sorted_extensions.keys():
[1442](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1442) for child in _get_children_dependencies(extension_name):
-> [1443](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1443) self.delete_extension(child)
[1445](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1445) extensions_with_pipeline = {}
[1446](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1446) extensions_without_pipeline = {}
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1616, in SortingAnalyzer.delete_extension(self, extension_name)
[1613](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1613) if self.format != "memory" and self.has_extension(extension_name):
[1614](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1614) # need a reload to reset the folder
[1615](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1615) ext = self.load_extension(extension_name)
-> [1616](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1616) ext.delete()
[1618](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1618) # remove from dict
[1619](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1619) self.extensions.pop(extension_name, None)
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:2306, in AnalyzerExtension.delete(self)
[2302](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2302) def delete(self):
[2303](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2303) """
[2304](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2304) Delete the extension from the folder or zarr and from the dict.
[2305](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2305) """
-> [2306](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2306) self._delete_extension_folder()
[2307](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2307) self.params = None
[2308](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2308) self.run_info = self._default_run_info_dict()
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:2292, in AnalyzerExtension._delete_extension_folder(self)
[2290](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2290) extension_folder = self._get_binary_extension_folder()
[2291](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2291) if extension_folder.is_dir():
-> [2292](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2292) shutil.rmtree(extension_folder)
[2294](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2294) elif self.format == "zarr":
[2295](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2295) import zarr
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\shutil.py:787, in rmtree(path, ignore_errors, onerror, dir_fd)
[785](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:785) # can't continue even if onerror hook returns
[786](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:786) return
--> [787](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:787) return _rmtree_unsafe(path, onerror)
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\shutil.py:634, in _rmtree_unsafe(path, onerror)
[632](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:632) os.unlink(fullname)
[633](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:633) except OSError:
--> [634](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:634) onerror(os.unlink, fullname, sys.exc_info())
[635](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:635) try:
[636](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:636) os.rmdir(path)
File c:\Users\systemses\Anaconda3\envs\si_env\Lib\shutil.py:632, in _rmtree_unsafe(path, onerror)
[630](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:630) else:
[631](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:631) try:
--> [632](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:632) os.unlink(fullname)
[633](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:633) except OSError:
[634](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:634) onerror(os.unlink, fullname, sys.exc_info())
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test_analyzer\\extensions\\waveforms\\waveforms.npy'
I managed to debug the errors, I think! This could be the cause of the hanging, or maybe not. I would appreciate it if you could try out the newly linked PR #3478 - thanks!
I couldn't manually delete the folder with the waveforms extension, so code seemed to have hang there.
See #3348 — maybe the same issue? (Not that we have a solution yet, but in case you want to follow along.) And I'm able to compute quality + template metrics tho with no issues even tho I do usually run into the waveform folder issue — and that's on Linux — so I don't think that's the issue here.
I can also pick this up in a couple days to test more on Windows. (I'll probably test Chris's script to see if we can recreate it in a controlled fashion and if not then we may need to ask for your dataset @jonpedros ) As for your issue @jonahpearl I think we re-ping Sam on that issue to see if he has some bandwidth to play with it a bit more.