pySCENIC
pySCENIC copied to clipboard
aertslab
[BUG] pyscenic ctx error
Describe the bug I am trying to run the ctx step to get the regulon.csv result but I always got the Error. I am stuck here for over one week. Really hope someone can help me. Thanks!!
Steps to reproduce the behavior
- Command run when the error occurred:
(pyscenic_new) C:\Users\shunn\>pyscenic ctx adjacencies.csv \
"mm9-500bp-upstream-7species.mc9nr.genes_vs_motifs.rankings.feather" "mm9-tss-centered-10kb-7species.mc9nr.genes_vs_motifs.rankings.feather" \
--annotations_fname "motifs-v9-nr.mgi-m0.001-o0.0.tbl.txt" \
--expression_mtx_fname "rat.expr_matrix_t.csv" \
--output regulons.csv \
--num_workers 6
- Error encountered:
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
2025-06-23 14:18:58,702 - pyscenic.cli.pyscenic - INFO - Creating modules.
2025-06-23 14:18:59,332 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2025-06-23 14:21:41,887 - pyscenic.utils - INFO - Calculating Pearson correlations.
2025-06-23 14:21:42,250 - pyscenic.utils - WARNING - Note on correlation calculation: the default behaviour for calculating the correlations has changed after pySCENIC verion 0.9.16. Previously, the default was to calculate the correlation between a TF and target gene using only cells with non-zero expression values (mask_dropouts=True). The current default is now to use all cells to match the behavior of the R verision of SCENIC. The original settings can be retained by setting 'rho_mask_dropouts=True' in the modules_from_adjacencies function, or '--mask_dropouts' from the CLI.
Dropout masking is currently set to [False].
2025-06-23 14:27:26,534 - pyscenic.utils - INFO - Creating modules.
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\utils.py:244: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
df = adjacencies.groupby(by=COLUMN_NAME_TARGET).apply(
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\utils.py:244: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
df = adjacencies.groupby(by=COLUMN_NAME_TARGET).apply(
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\utils.py:244: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
df = adjacencies.groupby(by=COLUMN_NAME_TARGET).apply(
2025-06-23 14:28:20,487 - pyscenic.cli.pyscenic - INFO - Loading databases.
2025-06-23 14:28:20,928 - pyscenic.cli.pyscenic - INFO - Calculating regulons.
2025-06-23 14:28:20,928 - pyscenic.prune - INFO - Using 6 workers.
2025-06-23 14:28:20,928 - pyscenic.prune - INFO - Using 6 workers.
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\ctxcore\__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound, get_distribution
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Anaconda3\envs\pyscenic_new\lib\multiprocessing\spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
TypeError: BaseProcess._bootstrap() takes 1 positional argument but 2 were given
Traceback (most recent call last):
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\multiprocessing_on_dill\connection.py", line 301, in _recv_bytes
ov, err = _winapi.ReadFile(self._handle, bsize,
BrokenPipeError: [WinError 109] 管道已结束。
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Anaconda3\envs\pyscenic_new\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Anaconda3\envs\pyscenic_new\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\Anaconda3\envs\pyscenic_new\Scripts\pyscenic.exe\__main__.py", line 7, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\cli\pyscenic.py", line 721, in main
args.func(args)
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\cli\pyscenic.py", line 244, in prune_targets_command
df_motifs = calc_func(
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\prune.py", line 424, in prune2df
return _distributed_calc(
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\prune.py", line 247, in _distributed_calc
fnames = [recv.recv() for recv in receivers]
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\prune.py", line 247, in <listcomp>
fnames = [recv.recv() for recv in receivers]
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\multiprocessing_on_dill\connection.py", line 250, in recv
buf = self._recv_bytes()
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\multiprocessing_on_dill\connection.py", line 321, in _recv_bytes
raise EOFError
EOFError
Then I tried:
(pyscenic_new) C:\Users\shunn\>pyscenic ctx adjacencies.csv \
"mm9-500bp-upstream-7species.mc9nr.genes_vs_motifs.rankings.feather" "mm9-tss-centered-10kb-7species.mc9nr.genes_vs_motifs.rankings.feather" \
--annotations_fname "motifs-v9-nr.mgi-m0.001-o0.0.tbl.txt" \
--expression_mtx_fname "rat.expr_matrix_t.csv" \
--output regulons.csv \
--num_workers 6\
--mode dask_multiprocessing
I got the error:
2025-06-23 14:17:09,558 - pyscenic.utils - INFO - Creating modules.
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\utils.py:244: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
df = adjacencies.groupby(by=COLUMN_NAME_TARGET).apply(
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\utils.py:244: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
df = adjacencies.groupby(by=COLUMN_NAME_TARGET).apply(
D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\utils.py:244: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
df = adjacencies.groupby(by=COLUMN_NAME_TARGET).apply(
2025-06-23 14:18:00,234 - pyscenic.cli.pyscenic - INFO - Loading databases.
2025-06-23 14:18:00,710 - pyscenic.cli.pyscenic - INFO - Calculating regulons.
Traceback (most recent call last):
File "D:\Anaconda3\envs\pyscenic_new\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Anaconda3\envs\pyscenic_new\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\Anaconda3\envs\pyscenic_new\Scripts\pyscenic.exe\__main__.py", line 7, in <module>
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\cli\pyscenic.py", line 721, in main
args.func(args)
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\cli\pyscenic.py", line 244, in prune_targets_command
df_motifs = calc_func(
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\prune.py", line 424, in prune2df
return _distributed_calc(
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\prune.py", line 351, in _distributed_calc
return create_graph().compute(
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\pyscenic\prune.py", line 340, in create_graph
return aggregate_func(
File "D:\Anaconda3\envs\pyscenic_new\lib\site-packages\dask\dataframe\dask_expr\io\_delayed.py", line 122, in from_delayed
if len(dfs) == 0:
TypeError: object of type 'generator' has no len()
Please complete the following information:
- pySCENIC version: 0.12.1+8.gd2309fe
- Installation method: Conda and Git
- Run environment: CLI
- Package versions:
# packages in environment at D:\Anaconda3\envs\pyscenic_new:
#
# Name Version Build Channel
aiohappyeyeballs 2.6.1 pypi_0 pypi
aiohttp 3.12.13 pypi_0 pypi
aiosignal 1.3.2 pypi_0 pypi
arboreto 0.1.6 pypi_0 pypi
asttokens 3.0.0 pyhd8ed1ab_1 conda-forge
async-timeout 5.0.1 pypi_0 pypi
attrs 25.3.0 pypi_0 pypi
bokeh 3.7.3 pypi_0 pypi
boltons 25.0.0 pypi_0 pypi
bzip2 1.0.8 h2466b09_7 conda-forge
ca-certificates 2025.6.15 h4c7d964_0 conda-forge
certifi 2025.6.15 pypi_0 pypi
charset-normalizer 3.4.2 pypi_0 pypi
click 8.2.1 pypi_0 pypi
cloudpickle 3.1.1 pypi_0 pypi
colorama 0.4.6 pyhd8ed1ab_1 conda-forge
comm 0.2.2 pyhd8ed1ab_1 conda-forge
contourpy 1.3.2 pypi_0 pypi
cpython 3.10.18 py310hd8ed1ab_0 conda-forge
ctxcore 0.2.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
cytoolz 1.0.1 pypi_0 pypi
dask 2025.5.1 pypi_0 pypi
debugpy 1.8.14 py310h9e98ed7_0 conda-forge
decorator 5.2.1 pyhd8ed1ab_0 conda-forge
dill 0.4.0 pypi_0 pypi
distributed 2025.5.1 pypi_0 pypi
exceptiongroup 1.3.0 pyhd8ed1ab_0 conda-forge
executing 2.2.0 pyhd8ed1ab_0 conda-forge
fonttools 4.58.4 pypi_0 pypi
frozendict 2.4.6 pypi_0 pypi
frozenlist 1.7.0 pypi_0 pypi
fsspec 2025.5.1 pypi_0 pypi
h5py 3.14.0 pypi_0 pypi
idna 3.10 pypi_0 pypi
importlib-metadata 8.7.0 pyhe01879c_1 conda-forge
interlap 0.2.7 pypi_0 pypi
ipykernel 6.29.5 pyh4bbf305_0 conda-forge
ipython 8.37.0 pyha7b4d00_0 conda-forge
jedi 0.19.2 pyhd8ed1ab_1 conda-forge
jinja2 3.1.6 pypi_0 pypi
joblib 1.5.1 pypi_0 pypi
jupyter_client 8.6.3 pyhd8ed1ab_1 conda-forge
jupyter_core 5.8.1 pyh5737063_0 conda-forge
kiwisolver 1.4.8 pypi_0 pypi
krb5 1.21.3 hdf4eb48_0 conda-forge
libexpat 2.7.0 he0c23c2_0 conda-forge
libffi 3.4.6 h537db12_1 conda-forge
liblzma 5.8.1 h2466b09_2 conda-forge
libsodium 1.0.20 hc70643c_0 conda-forge
libsqlite 3.50.1 hf5d6505_6 conda-forge
libzlib 1.3.1 h2466b09_2 conda-forge
llvmlite 0.44.0 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
loompy 3.0.8 pypi_0 pypi
lz4 4.4.4 pypi_0 pypi
markupsafe 3.0.2 pypi_0 pypi
matplotlib 3.10.3 pypi_0 pypi
matplotlib-inline 0.1.7 pyhd8ed1ab_1 conda-forge
msgpack 1.1.1 pypi_0 pypi
multidict 6.5.0 pypi_0 pypi
multiprocessing-on-dill 3.5.0a4 pypi_0 pypi
narwhals 1.44.0 pypi_0 pypi
nest-asyncio 1.6.0 pyhd8ed1ab_1 conda-forge
networkx 3.4.2 pypi_0 pypi
numba 0.61.2 pypi_0 pypi
numexpr 2.11.0 pypi_0 pypi
numpy 2.2.6 pypi_0 pypi
numpy-groupies 0.11.3 pypi_0 pypi
openssl 3.5.0 ha4e3fda_1 conda-forge
packaging 25.0 pyh29332c3_1 conda-forge
pandas 2.3.0 pypi_0 pypi
parso 0.8.4 pyhd8ed1ab_1 conda-forge
partd 1.4.2 pypi_0 pypi
pickleshare 0.7.5 pyhd8ed1ab_1004 conda-forge
pillow 11.2.1 pypi_0 pypi
pip 25.1.1 pyh8b19718_0 conda-forge
platformdirs 4.3.8 pyhe01879c_0 conda-forge
prompt-toolkit 3.0.51 pyha770c72_0 conda-forge
propcache 0.3.2 pypi_0 pypi
psutil 7.0.0 py310ha8f682b_0 conda-forge
pure_eval 0.2.3 pyhd8ed1ab_1 conda-forge
pyarrow 20.0.0 pypi_0 pypi
pygments 2.19.2 pyhd8ed1ab_0 conda-forge
pynndescent 0.5.13 pypi_0 pypi
pyparsing 3.2.3 pypi_0 pypi
pyscenic 0.12.1+8.gd2309fe pypi_0 pypi
python 3.10.18 h8c5b53a_0_cpython conda-forge
python-dateutil 2.9.0.post0 pyhff2d567_1 conda-forge
python_abi 3.10 7_cp310 conda-forge
pytz 2025.2 pypi_0 pypi
pywin32 307 py310h9e98ed7_3 conda-forge
pyyaml 6.0.2 pypi_0 pypi
pyzmq 27.0.0 py310h656833d_0 conda-forge
requests 2.32.4 pypi_0 pypi
scikit-learn 1.7.0 pypi_0 pypi
scipy 1.15.3 pypi_0 pypi
seaborn 0.13.2 pypi_0 pypi
setuptools 80.9.0 pyhff2d567_0 conda-forge
six 1.17.0 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pypi_0 pypi
stack_data 0.6.3 pyhd8ed1ab_1 conda-forge
tblib 3.1.0 pypi_0 pypi
threadpoolctl 3.6.0 pypi_0 pypi
tk 8.6.13 h2c6b04d_2 conda-forge
toolz 1.0.0 pypi_0 pypi
tornado 6.5.1 py310ha8f682b_0 conda-forge
tqdm 4.67.1 pypi_0 pypi
traitlets 5.14.3 pyhd8ed1ab_1 conda-forge
typing_extensions 4.14.0 pyhe01879c_0 conda-forge
tzdata 2025.2 pypi_0 pypi
ucrt 10.0.22621.0 h57928b3_1 conda-forge
umap-learn 0.5.7 pypi_0 pypi
urllib3 2.5.0 pypi_0 pypi
vc 14.3 h41ae7f8_26 conda-forge
vc14_runtime 14.44.35208 h818238b_26 conda-forge
wcwidth 0.2.13 pyhd8ed1ab_1 conda-forge
wheel 0.45.1 pyhd8ed1ab_1 conda-forge
xyzservices 2025.4.0 pypi_0 pypi
yarl 1.20.1 pypi_0 pypi
zeromq 4.3.5 ha9f60a1_7 conda-forge
zict 3.0.0 pypi_0 pypi
zipp 3.23.0 pyhd8ed1ab_0 conda-forge
Perhaps it's related to the dask and pandas versions. My normal environment for running Pyscenic is set up like this:
conda create -y -n pyscenic python=3.10 dask-core=2023.10.0
pip install --no-deps git+https://github.com/aertslab/pySCENIC.git
pip install git+https://github.com/DmitryUlyanov/Multicore-TSNE.git
pip install numpy==1.23.0 pandas==1.3.5 contourpy==1.0.6 numexpr==2.8.4 bokeh==2.4.3 numba==0.58.1 tbb==2022.1.0 scikit-learn==1.6.1 natsort==8.0.2 scipy==1.9.3 llvmlite==0.41.1 arboreto==0.1.6 anndata==0.7.8 distributed==2023.10.0 dask==2023.10.0 decorator==4.3.0 scanpy==1.8.2 tqdm==4.62.3 networkx==2.8.8 umap-learn==0.3.10 diptest==0.8.2 interlap boltons==21.0.0 statsmodels==0.13.5 loompy==3.0.8
Perhaps you could give it a try.
