ctlab
GADMA slow for 5 population model
Hello,
I'm attempting to run a 5 population model using the Moments engine with SMAC_BO_combination Bayesian optimization. GADMA has been running for 50 hours but still reports No models yet. I'm running this on a cluster compute node with 8 cores and 640 GB of RAM.
Is this long runtime normal for a larger model? Or is there something I can do to help speed things up?
Custom model file:
import moments
import numpy as np
def model_func(params, ns):
"""
Five population demographic history for the species complex
populations:
A - Acin Elk
B - Acin upper Tennessee
C - Aana
D - Amay Cumberland basin
E - Amay Big South Fork
All pops have constant population size.
Order of splits in this model are based on prior phylogenetic results
Ancestral population was split (t1 + t2) time ago into two new
populations: AB and CDE. Next, CDE population was split into C and DE.
Then AB then split into two populations, A and B. Finally, DE splits
into two populations, D and E. There is symmetrical migration between pops A, B,
C, D, and E after the final split. But there is no migration between
ancestral populations.
:param nAB: Size of ancestral population AB after earliest split
:param nDCE: Size of ancestral population DCE after earliest split
:param nA: Size of extant population A
:param nB: Size of extant population B
:param nC: Size of extant population C
:param nDE: Size of ancestral population DE
:param nD: Size of extant population D
:param nE: Size of extant population E
:params mAB, mAC, mAD, mAE, mBC, mBD, mBE, mCD, mCE, mDE: migration rates between each pair of extant populations
:param t1: Time between ancestral population split to C - DE split
:param t2: Time between C - DE split and A - B split
:param t3: Time between A - B split and D - E split
:param t4: Time between D - E split to the present
"""
nAB, nDCE, nA, nB, nC, nDE, nD, nE, mAB, mAC, mAD, mAE, mBC, mBD, mBE, mCD, mCE, mDE, t1, t2, t3, t4 = params
sts = moments.LinearSystem_1D.steady_state_1D(sum(ns))
fs = moments.Spectrum(sts)
# first split: AB - CDE
fs = fs.split(idx = 0, n0 = ns[0] + ns[3], n1 = ns[1] + ns[2] + ns[4])
fs.integrate(Npop=[nAB, nDCE], tf=t1)
# second split: C - DE
fs = fs.split(idx = 1, n0 = ns[1], n1 = ns[2] + ns[4])
fs.integrate(Npop=[nAB, nC, nDE], tf=t2)
# third split: A - B
fs = fs.split(idx = 0, n0 = ns[0], n1 = ns[3])
fs.integrate(Npop=[nA, nC, nDE, nB], tf=t3)
# fourth split: D - E
fs = fs.split(idx = 2, n0 = ns[2], n1 = ns[4])
# Full 5x5 migration matrix
mig_mat = [
[0, mAC, mAD, mAB, mAE],
[mAC, 0, mCD, mBC, mCE],
[mAD, mCD, 0, mBD, mDE],
[mAB, mBC, mBD, 0, mBE],
[mAE, mCE, mDE, mBE, 0 ]
]
fs.integrate(Npop=[nA, nC, nD, nB, nE], tf=t4, m=mig_mat)
return fs(base)
Parameters file:
Input file: easySFS/output/dadi/A-C-D-B-E.sfs
Output directory: results/model_5pops_01_quick
Custom filename: models/model_5pops_01.py
Num init const: 2
Global optimizer: SMAC_BO_combination
Global maxeval: 200
Number of repeats: 8
Number of processes: 8
Linked SNP's: False
Engine: moments
# NEED TO REFINE THIS
Mutation rate: 2.35e-08
# NEED TO FIND REFERENCE FOR THIS
Time for generation: 2.0
# Effective length of the sequence used to build the SFS data.
# This should be used together with the Mutation rate and can be replaced
# by the Theta0 setting.
# Default: None
# https://gadma.readthedocs.io/en/latest/faq.html#how-can-i-get-effective-length-of-sequence-l
# Assume total length of sequence that was used for SFS building is equal to Nseq.
# From this data total number of X SNP’s were received.
# But not all of them were used for SFS: some of them (e.g. total number of Y SNP’s) were filtered out.
# Then we should count filtered SNP’s and take L value the following way:
# L = (X - Y) / X * Nseq
# in our case:
# X = 171832 [total number of variable sites in m80p _stats.txt file]
# 16823 SNPs in the m80p unlinked VCF, so
# Y = 155009
# so L = ((171832 - 155009) / 171832) * 3995963) = 391220
Sequence length: 391220
# Engine that will draw demographic model plots.
# Can be moments or demes.
# Default: moments
Model plot engine: demes
Units of time in drawing: thousand years
Thanks for your help, Dan
Hi @dmacguigan,
Unfortunately, it can take a lot of time. The duration depends on your cluster and on the sample size of your data - the bigger SFS is, the longer it will take. Five populations may be extremely slow.
There is one way to check the progress. For each run X there is an output folder named by the number of this run gadma_output/X. There you can find a file eval_fie, which stores all log-likelihood evaluations. The number of lines in this file corresponds to the number of evaluations, and the target number is 200 (as you set it by Global maxeval). Once run reaches 200 - it finishes.
Another thing you might be interested in this eval_file is that each evaluation has its time (the last value in the line). Therefore, you can get an estimation of the remaining time. Note that this time can be different for different evaluations and runs.
I hope that helps.
The only thing you can do to fast things up is to downproject your SFS, infer parameters, and then run an additional GADMA resume run with the original SFS to adjust parameters. If you would like to give it a try, I can help you with that.
Best regards, Ekaterina
Thanks for the quick response @noscode! It looks like my optimization only got through about 24 evaluations in 72 hours, so my current approach is definitely not viable.
I'd be very interested to try your down projection strategy. I'm already down projecting my SFS a bit with easySFS.py --proj 14,44,26,20,44 -a. Do you have a suggestion for a target sample size for the initial parameterization run?
I would suggest you downproject SFS to 10 per population. You can just specify Projections: 10,10,10,10,10 in your parameters file. It will still take some time, but will be significantly faster than before.
Once the run is finished, you can run another GADMA run with the following options:
# take best models from previuous launch
Resume from: output_directory_of_first_run
Only models: True
# Use original projections
Projections: 14,44,26,20,44
# Do not run global optimization, just local one as we are already close to the answer
Global maxeval: 1
The part might take even more time than the first, as likelihood evaluations are much more time-expensive, but I think that should be a good way to try it.
Thanks for the suggestion! Unfortunately, I encountered the following errors when using Projections: 10,10,10,10,10 in my params file. Strangely, it seems to work fine if I used easySFS to reproject my SFS down to 10 per population.
Any troubleshooting suggestions?
Full log file here: GADMA_model_5pops_01_quick_small.log
Some errors from the log:
+ Fri May 23 09:52:39 EDT 2025 job GADMA_model_5pops_01 started in mThM.q with jobID=8130826 on compute-76-07
Data reading
Number of populations: 5
Projections: [10, 10, 10, 10, 10]
Population labels: ['A', 'C', 'D', 'B', 'E']
Outgroup: False
--Successful data reading (0.10 s)--
--Successful arguments parsing--
Parameters of launch are saved in output directory: /pool/public/genomics/macguigand/Allohistium/ddRAD/analyses/GADMA/moments/results/model_5pops_01_quick_small/params_file
All output is saved in output directory: /pool/public/genomics/macguigand/Allohistium/ddRAD/analyses/GADMA/moments/results/model_5pops_01_quick_small/GADMA.log
--Start pipeline--
Run launch number 1
Run launch number 2
Run launch number 6
Run launch number 8
Run launch number 7
Run launch number 4
Run launch number 3
Run launch number 5
[000:01:00]
No models yet
UserWarning: Setting `Input file` is renamed in 2 version of GADMA to `Input data`. It is successfully read. (/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/cli/settings_storage.py:1020)
UserWarning: Custom model was set, engine to draw models will be the same as for evaluations: moments (/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/cli/settings_storage.py:1520)
WARNING:Inference:Model is < 0 where data is not masked.
WARNING:Inference:Number of affected entries is 5. Sum of data in those entries is 0.447372:
.
.
.
Run 8 failed due to following exception:
Traceback (most recent call last):
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/core/core.py", line 37, in job
obj.run(settings.get_optimizers_init_kwargs())
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/core/core_run.py", line 620, in run
result = self.run_without_increase(initial_kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/core/core_run.py", line 442, in run_without_increase
result = optimizer.optimize(f, variables, **opt_kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/optimizers/combinations.py", line 164, in optimize
global_result = self.global_optimizer.optimize(
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/optimizers/global_optimizer.py", line 277, in optimize
return super(GlobalOptimizer, self).optimize(
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/optimizers/optimizer.py", line 572, in optimize
self._optimize(f=f_in_opt,
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 993, in _optimize
kernel_name1, message1 = choose_kernel_if_needed(
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 72, in choose_kernel_if_needed
Y[~np.isfinite(Y)] = np.max(Y[np.isfinite(Y)])
File "<__array_function__ internals>", line 180, in amax
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2791, in amax
return _wrapreduction(a, np.maximum, 'max', axis, None, out,
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity
Hi,
Have you tried to run it the second time? I think I know why this error occurred, but it might be just a very rare case. Anyway, if easySFS' downprojecting works better - you can use smaller SFS and then switch Input data to bigger SFS.
Best regards, Ekaterina
If you update GADMA to the last version, it should print a proper values for Nanc instead of [WARNING Nanc < 0!].
Thank's for the advice @noscode. I ended up just using the easySFS downprojection. Unfortunately, I'm having a hard time upgrading to the latest version of GADMA in my conda environment. Lots of conflicting packages.
On a different note, is there a way to plot my best model after a run is complete? I encountered the following error during my run.
failed to draw model due to the following exception: module 'moments.Plotting' has no attribute 'plot_5d_comp_Poisson'
Ideally, I'd like to plot the models with demes, but I don't see a demes file produced by GADMA.
(gadma_env) [macguigand@login02 model_5pops_01_quick_small]$ ls -la
total 3364
drwxrwsr-x 10 macguigand macguigand 4096 May 28 21:25 .
drwxrwsr-x 7 macguigand macguigand 4096 May 29 16:40 ..
drwxrwsr-x 2 macguigand macguigand 4096 May 29 02:32 1
drwxrwsr-x 2 macguigand macguigand 4096 May 29 03:01 2
drwxrwsr-x 2 macguigand macguigand 4096 May 29 02:23 3
drwxrwsr-x 2 macguigand macguigand 4096 May 29 06:04 4
drwxrwsr-x 2 macguigand macguigand 4096 May 29 17:57 5
drwxrwsr-x 2 macguigand macguigand 4096 May 29 01:11 6
drwxrwsr-x 2 macguigand macguigand 4096 May 29 00:00 7
drwxrwsr-x 2 macguigand macguigand 4096 May 28 21:24 8
-rw-rw-r-- 1 macguigand macguigand 3355532 May 29 18:07 GADMA.log
-rw-rw-r-- 1 macguigand macguigand 2309 May 29 18:07 best_aic_model_moments_code.py
-rw-rw-r-- 1 macguigand macguigand 2309 May 29 18:05 best_logLL_model_moments_code.py
-rw-rw-r-- 1 macguigand macguigand 4719 May 27 17:55 extra_params_file
-rw-rw-r-- 1 macguigand macguigand 11380 May 27 17:55 params_file
(gadma_env) [macguigand@login02 model_5pops_01_quick_small]$ ls -la 1/
total 1308
drwxrwsr-x 2 macguigand macguigand 4096 May 29 02:32 .
drwxrwsr-x 10 macguigand macguigand 4096 May 28 21:25 ..
-rw-rw-r-- 1 macguigand macguigand 439144 May 29 02:28 GADMA_GA.log
-rw-rw-r-- 1 macguigand macguigand 2309 May 29 02:32 current_best_aic_model_moments_code.py
-rw-rw-r-- 1 macguigand macguigand 2309 May 29 02:30 current_best_logLL_model_moments_code.py
-rw-rw-r-- 1 macguigand macguigand 376656 May 29 02:28 eval_file
-rw-rw-r-- 1 macguigand macguigand 2309 May 29 02:32 final_best_logLL_model_moments_code.py
-rw-rw-r-- 1 macguigand macguigand 483498 May 29 02:28 save_file
Hi @dmacguigan,
I am afraid the only way to get some pictures right away is to run final_best_logLL_model_moments_code.py - it will draw plot of the final history using moments. If you want to have demes plot, you will have to implement demes' yaml file and then draw it using demesdraw package. Unfortunately, GADMA is not able to transform moments model into demes.
Best regards, Ekaterina
Oh so GADMA is only able to plot dadi results with demes? I guess I missed that in the documentation.
GADMA plots demes for models specified by structure only. If I am not mistaken, dadi can generate demes files for its models, but it is not part of GADMA, but part of dadi. I would be glad to include demes generation for all models in GADMA, unfortunately, it is not that easy.
Best regards, Ekaterina
Hello @noscode,
Wanted to loop back on my plotting problems. I've tried to draw a plot using the best_logLL_model_moments_code.py script, but I'm getting the following error.
(gadma_env) [macguigand@compute-64-16 model_5pops_01_small_long]$ python best_logLL_model_moments_code.py
Model log likelihood (LL(model, data)): -5511.838680458968
Optimal value of theta: 1632.2582473857115
Size of ancestral population: 461529
Traceback (most recent call last):
File "best_logLL_model_moments_code.py", line 36, in <module>
moments.ModelPlot.plot_model(gen_mod,
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/moments/ModelPlot.py", line 235, in plot_model
ax.grid(b=True, which="major", axis="y", color=gridline_color)
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 3196, in grid
self.yaxis.grid(visible, which=which, **kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/matplotlib/axis.py", line 1660, in grid
self.set_tick_params(which='major', **gridkw)
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/matplotlib/axis.py", line 932, in set_tick_params
kwtrans = self._translate_tick_params(kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env/lib/python3.8/site-packages/matplotlib/axis.py", line 1076, in _translate_tick_params
raise ValueError(
ValueError: keyword grid_b is not recognized; valid keywords are ['size', 'width', 'color', 'tickdir', 'pad', 'labelsize', 'labelcolor', 'zorder', 'gridOn', 'tick1On', 'tick2On', 'label1On', 'label2On', 'length', 'direction', 'left', 'bottom', 'right', 'top', 'labelleft', 'labelbottom', 'labelright', 'labeltop', 'labelrotation', 'grid_agg_filter', 'grid_alpha', 'grid_animated', 'grid_antialiased', 'grid_clip_box', 'grid_clip_on', 'grid_clip_path', 'grid_color', 'grid_dash_capstyle', 'grid_dash_joinstyle', 'grid_dashes', 'grid_data', 'grid_drawstyle', 'grid_figure', 'grid_fillstyle', 'grid_gapcolor', 'grid_gid', 'grid_in_layout', 'grid_label', 'grid_linestyle', 'grid_linewidth', 'grid_marker', 'grid_markeredgecolor', 'grid_markeredgewidth', 'grid_markerfacecolor', 'grid_markerfacecoloralt', 'grid_markersize', 'grid_markevery', 'grid_mouseover', 'grid_path_effects', 'grid_picker', 'grid_pickradius', 'grid_rasterized', 'grid_sketch_params', 'grid_snap', 'grid_solid_capstyle', 'grid_solid_joinstyle', 'grid_transform', 'grid_url', 'grid_visible', 'grid_xdata', 'grid_ydata', 'grid_zorder', 'grid_aa', 'grid_c', 'grid_ds', 'grid_ls', 'grid_lw', 'grid_mec', 'grid_mew', 'grid_mfc', 'grid_mfcalt', 'grid_ms']
Details of my conda environment. Like I mentioned before, I'm having trouble installing the latest version of GADMA with conda, so stuck on 2.0.0 for now.
(gadma_env) [macguigand@compute-64-16 model_5pops_01_small_long]$ conda list
# packages in environment at /home/macguigand/software/anaconda3/envs/gadma_env:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
apricot-select 0.6.1 pyhd8ed1ab_0 conda-forge
asciitree 0.3.3 py_2 conda-forge
attrs 24.2.0 pyh71513ae_0 conda-forge
binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge
binutils_linux-64 2.36 hf3e587d_1 conda-forge
blas 1.0 openblas
blas-devel 3.9.0 20_linux64_openblas conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
bottleneck 1.3.7 py38ha9d4c09_0
brotli 1.1.0 hb9d3cd8_2 conda-forge
brotli-bin 1.1.0 hb9d3cd8_2 conda-forge
brotli-python 1.0.9 py38hfa26641_8 conda-forge
bzip2 1.0.8 h5eee18b_6
c-ares 1.34.5 hb9d3cd8_0 conda-forge
ca-certificates 2025.4.26 hbd8a1cb_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge
cffi 1.17.0 py38heb5c249_0 conda-forge
charset-normalizer 3.4.0 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
cloudpickle 3.1.0 pyhd8ed1ab_1 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
configspace 0.4.21 py38h26c90d9_1 conda-forge
contourpy 1.1.1 py38h7f3f72f_1 conda-forge
cpython 3.8.20 py38hd8ed1ab_2 conda-forge
cuda-cudart 12.4.127 h99ab3db_0
cuda-cudart_linux-64 12.4.127 hd681fbe_0
cuda-nvrtc 12.4.127 h99ab3db_1
cuda-nvtx 12.4.127 h6a678d5_1
cuda-version 12.4 hbda6634_3
cudnn 9.1.1.17 cuda12_1
cycler 0.12.1 pyhd8ed1ab_0 conda-forge
cython 3.0.11 py38h6d02427_0 conda-forge
cytoolz 0.12.3 py38h01eb140_0 conda-forge
dadi 2.3.0 py38hcaaa2bb_2 conda-forge
dask 2023.3.0 pyhd8ed1ab_0 conda-forge
dask-core 2023.3.0 pyhd8ed1ab_0 conda-forge
demes 0.2.3 pyhd8ed1ab_0 conda-forge
demesdraw 0.4.0 pyhd8ed1ab_0 conda-forge
distributed 2023.3.0 pyhd8ed1ab_0 conda-forge
emcee 3.1.6 pyhd8ed1ab_0 conda-forge
fasteners 0.17.3 pyhd8ed1ab_0 conda-forge
filelock 3.16.1 pyhd8ed1ab_0 conda-forge
fonttools 4.53.1 py38h2019614_0 conda-forge
freetype 2.13.3 ha770c72_1 conda-forge
fsspec 2024.10.0 pyhff2d567_0 conda-forge
gadma 2.0.0 pyhdfd78af_0 bioconda
gcc_impl_linux-64 11.1.0 h6b5115b_8 conda-forge
gcc_linux-64 11.1.0 h97fdae6_1 conda-forge
gmp 6.3.0 h6a678d5_0
gmpy2 2.1.5 py38h6a1700d_1 conda-forge
gxx_impl_linux-64 11.1.0 h6b5115b_8 conda-forge
gxx_linux-64 11.1.0 h33c4e4b_1 conda-forge
h2 4.1.0 pyhd8ed1ab_0 conda-forge
h5py 3.9.0 nompi_py38hc4a6f91_101 conda-forge
hdf5 1.14.1 nompi_h4f84152_100 conda-forge
hpack 4.0.0 pyh9f0ad1d_0 conda-forge
hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge
idna 3.10 pyhd8ed1ab_0 conda-forge
importlib-metadata 8.5.0 pyha770c72_0 conda-forge
importlib-resources 6.4.5 pyhd8ed1ab_0 conda-forge
importlib_metadata 8.5.0 hd3eb1b0_0
importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge
jinja2 3.1.4 pyhd8ed1ab_0 conda-forge
joblib 1.4.2 pyhd8ed1ab_0 conda-forge
kernel-headers_linux-64 3.10.0 he073ed8_18 conda-forge
kiwisolver 1.4.5 py38h7f3f72f_1 conda-forge
krb5 1.21.3 h143b758_0
lazy_import 0.2.2 pyhd8ed1ab_0 conda-forge
lcms2 2.17 h717163a_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 4.0.0 h0aef613_1 conda-forge
libabseil 20240116.2 cxx17_he02047a_1 conda-forge
libaec 1.1.3 h6a678d5_0
libblas 3.9.0 20_linux64_openblas conda-forge
libbrotlicommon 1.1.0 hb9d3cd8_2 conda-forge
libbrotlidec 1.1.0 hb9d3cd8_2 conda-forge
libbrotlienc 1.1.0 hb9d3cd8_2 conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libcublas 12.4.5.8 h99ab3db_1
libcufft 11.2.1.3 h99ab3db_1
libcurand 10.3.5.147 h99ab3db_1
libcurl 8.12.1 h332b0f4_0 conda-forge
libcusolver 11.6.1.9 h99ab3db_1
libcusparse 12.3.1.170 h99ab3db_1
libdeflate 1.22 hb9d3cd8_0 conda-forge
libedit 3.1.20230828 h5eee18b_0
libev 4.33 hd590300_2 conda-forge
libffi 3.4.4 h6a678d5_1
libfreetype 2.13.3 ha770c72_1 conda-forge
libfreetype6 2.13.3 h48d6fc4_1 conda-forge
libgcc 15.1.0 h767d61c_2 conda-forge
libgcc-devel_linux-64 11.1.0 h80e7780_8 conda-forge
libgcc-ng 15.1.0 h69a702a_2 conda-forge
libgfortran 15.1.0 h69a702a_2 conda-forge
libgfortran-ng 15.1.0 h69a702a_2 conda-forge
libgfortran5 15.1.0 hcea5267_2 conda-forge
libgomp 15.1.0 h767d61c_2 conda-forge
libjpeg-turbo 3.1.0 hb9d3cd8_0 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
liblapacke 3.9.0 20_linux64_openblas conda-forge
libllvm14 14.0.6 hcd5def8_4 conda-forge
libmagma 2.8.0 h0af6554_0 conda-forge
libmagma_sparse 2.8.0 h0af6554_0 conda-forge
libnghttp2 1.64.0 h161d5f1_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libnvjitlink 12.4.127 h99ab3db_1
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libpng 1.6.47 h943b412_0 conda-forge
libprotobuf 4.25.3 hd5b35b9_1 conda-forge
libsanitizer 11.1.0 h56837e0_8 conda-forge
libsqlite 3.49.2 hee588c1_0 conda-forge
libssh2 1.11.1 hcf80075_0 conda-forge
libstdcxx 15.1.0 h8f9b012_2 conda-forge
libstdcxx-devel_linux-64 11.1.0 h80e7780_8 conda-forge
libstdcxx-ng 15.1.0 h4852527_2 conda-forge
libtiff 4.7.0 he137b08_1 conda-forge
libtorch 2.5.1 gpu_cuda124_h7d9372d_202
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.50.0 hb9d3cd8_0 conda-forge
libwebp-base 1.5.0 h851e524_0 conda-forge
libxcb 1.17.0 h8a09558_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
llvmlite 0.41.1 py38h94a1851_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.3 py38hdcd8cb4_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
magma 2.8.0 h51420fd_0 conda-forge
markupsafe 2.1.5 py38h01eb140_0 conda-forge
matplotlib-base 3.7.2 py38h1128e8f_0
moments 1.1.16 py38h24c8ff8_3 bioconda
more-itertools 10.5.0 pyhd8ed1ab_0 conda-forge
mpc 1.3.1 h24ddda3_1 conda-forge
mpfr 4.2.1 h90cbb55_3 conda-forge
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.8 py38hea7755e_0 conda-forge
munkres 1.0.7 py_1 bioconda
nccl 2.21.5.1 ha515578_0
ncurses 6.5 h2d0b736_3 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
nlopt 2.7.1 h6a678d5_0
nomkl 1.0 h5ca1d4c_0 conda-forge
nose 1.3.7 py_1006 conda-forge
numba 0.58.1 py38h6a678d5_0
numcodecs 0.12.1 py38h854fd01_1 conda-forge
numexpr 2.8.4 py38hb2af0cf_101 conda-forge
numpy 1.24.4 pypi_0 pypi
numpy-base 1.22.3 py38hb8be1f0_0
openblas 0.3.25 pthreads_h7a3da1a_0 conda-forge
openjpeg 2.5.3 h5fbd93e_0 conda-forge
openssl 3.5.0 h7b32b05_1 conda-forge
packaging 25.0 pyh29332c3_1 conda-forge
pandas 2.0.3 py38h01efb38_1 conda-forge
partd 1.4.1 pyhd8ed1ab_0 conda-forge
patsy 0.5.6 pyhd8ed1ab_0 conda-forge
pillow 10.4.0 py38h2bc05a7_0 conda-forge
pip 24.3.1 pyh8b19718_0 conda-forge
platformdirs 4.3.6 pyhd8ed1ab_0 conda-forge
pomegranate 1.0.0 pyhd8ed1ab_1 conda-forge
pooch 1.8.2 pyhd8ed1ab_0 conda-forge
psutil 6.0.0 py38hfb59056_0 conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
pyaml 20.4.0 pyhd3eb1b0_0
pybind11-abi 4 hd8ed1ab_3 conda-forge
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pynisher 1.0.10 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 py38h06a4308_0
pyrfr 0.9.0 py38h7f3f72f_1 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.8.20 h4a871b0_2_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge
python_abi 3.8 7_cp38 conda-forge
pytorch 2.3.0 cpu_py38h1ce4368_1
pytz 2024.2 pyhd8ed1ab_0 conda-forge
pywin32-on-windows 0.1.0 pyh1179c8e_3 conda-forge
pyyaml 6.0.2 py38h2019614_0 conda-forge
qhull 2020.2 h4bd325d_2 conda-forge
readline 8.2 h5eee18b_0
requests 2.32.3 pyhd8ed1ab_0 conda-forge
ruamel.yaml 0.16.12 py38h0a891b7_3 conda-forge
ruamel.yaml.clib 0.2.8 py38h01eb140_0 conda-forge
scikit-allel 1.3.6 py38h01efb38_0 conda-forge
scikit-learn 1.1.2 py38h0b08f9b_0 conda-forge
scikit-optimize 0.10.2 pyhd8ed1ab_0 conda-forge
scipy 1.10.1 pypi_0 pypi
seaborn 0.12.2 py38h06a4308_0
seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge
setuptools 75.3.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sleef 3.8 h1b44611_0 conda-forge
smac 0.13.1 pyhd8ed1ab_1 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sqlite 3.49.2 h9eae976_0 conda-forge
statsmodels 0.14.0 py38ha9d4c09_0
sympy 1.13.3 pyh2585a3b_104 conda-forge
sysroot_linux-64 2.17 h0157908_18 conda-forge
tbb 2021.8.0 hdb19cb5_0
tblib 3.0.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
toolz 1.0.0 pyhd8ed1ab_0 conda-forge
tornado 6.4.1 py38hfb59056_0 conda-forge
tqdm 4.67.1 pyhd8ed1ab_0 conda-forge
typing_extensions 4.12.2 pyha770c72_0 conda-forge
tzdata 2025b h04d1e81_0
unicodedata2 15.1.0 py38h01eb140_0 conda-forge
urllib3 2.2.3 pyhd8ed1ab_0 conda-forge
wheel 0.45.1 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge
xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge
xz 5.6.4 h5eee18b_1
yaml 0.2.5 h7f98852_2 conda-forge
zarr 2.17.1 pyhd8ed1ab_0 conda-forge
zict 3.0.0 pyhd8ed1ab_0 conda-forge
zipp 3.21.0 pyhd8ed1ab_0 conda-forge
zlib 1.3.1 hb9d3cd8_2 conda-forge
zstandard 0.23.0 py38h62bed22_0 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
Dear @dmacguigan,
Pictures for moments might be very picky on matplotlib version. I have checked my tests and seems I have the following version requirement: "matplotlib<3.5". Could you please try to install it the following way and let me know if moments plotting issue will be resolved?
Best regards, Ekaterina
Hi @noscode,
I managed to install the latest version of GADMA with pip. However, now I'm encountering a new error.
AttributeError: module 'numpy' has no attribute 'int'
Should I be using a different version of numpy? I have 1.24.4 installed.
My current environment:
(gadma_env_latest) [macguigand@compute-64-16 moments]$ pip list
Package Version
------------------- -----------
attrs 25.3.0
bayesmark 0.0.8
click 8.1.8
cloudpickle 3.1.1
ConfigSpace 0.4.21
contourpy 1.1.1
cycler 0.12.1
Cython 3.1.1
dadi 2.4.0
dask 2023.5.0
demes 0.2.3
demesdraw 0.4.0
distributed 2023.5.0
fonttools 4.57.0
fsspec 2025.3.0
gadma 2.0.3
gitdb 4.0.12
GitPython 3.1.44
h5py 3.10.0
importlib_metadata 8.5.0
importlib_resources 6.4.5
Jinja2 3.1.6
joblib 1.4.2
kiwisolver 1.4.7
lazy_import 0.2.2
locket 1.0.0
MarkupSafe 2.1.5
matplotlib 3.7.5
moments-popgen 1.3.1
more-itertools 10.5.0
mpmath 1.3.0
msgpack 1.1.0
nlopt 2.7.1
numpy 1.24.4
packaging 25.0
pandas 2.0.3
partd 1.4.1
pathvalidate 3.2.1
pillow 10.4.0
pip 24.3.1
psutil 7.0.0
pyaml 25.5.0
pynisher 1.0.10
pyparsing 3.1.4
pyrfr 0.9.0
python-dateutil 2.9.0.post0
pytz 2025.2
PyYAML 6.0.2
ruamel.yaml 0.16.12
ruamel.yaml.clib 0.2.8
scikit-allel 1.3.7
scikit-learn 1.1.2
scikit-optimize 0.10.2
scipy 1.6.0
setuptools 75.3.0
six 1.17.0
smac 0.13.1
smmap 5.0.2
sortedcontainers 2.4.0
tblib 3.0.0
threadpoolctl 3.5.0
toolz 1.0.0
tornado 6.4.2
typing_extensions 4.13.2
tzdata 2025.2
urllib3 2.2.3
wheel 0.45.1
xarray 2023.1.0
zict 3.0.0
zipp 3.20.2
I am so sorry you have to deal with all these issues! Yes, seems that you need numpy<1.20.0 to avoid such an error. I can double-check what particular version of numpy will work, if you still have a problem.
Thanks! That fixed the numpy isssue, but now another error rears its ugly head.
Traceback (most recent call last):
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/core/core.py", line 38, in job
obj.run(settings.get_optimizers_init_kwargs())
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/core/core_run.py", line 620, in run
result = self.run_without_increase(initial_kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/core/core_run.py", line 442, in run_without_increase
result = optimizer.optimize(f, variables, **opt_kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/combinations.py", line 164, in optimize
global_result = self.global_optimizer.optimize(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/global_optimizer.py", line 277, in optimize
return super(GlobalOptimizer, self).optimize(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/optimizer.py", line 572, in optimize
self._optimize(f=f_in_opt,
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 993, in _optimize
kernel_name1, message1 = choose_kernel_if_needed(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 73, in choose_kernel_if_needed
optimizer.kernel_name = get_best_kernel(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/utils/utils.py", line 704, in get_best_kernel
scores = get_loo_scores_for_kernels(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/utils/utils.py", line 683, in get_loo_scores_for_kernels
score = get_loo_score_for_optimizer(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/utils/utils.py", line 661, in get_loo_score_for_optimizer
gp = optimizer.get_model(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 606, in get_model
return SMACGaussianProcess(smac_gp)
NameError: name 'SMACGaussianProcess' is not defined
Perhaps this line is the culprit?
My GADMA enviornment:
(gadma_env_latest) [macguigand@login02 ~]$ pip list
Package Version
------------------- -----------
attrs 25.3.0
bayesmark 0.0.8
click 8.1.8
cloudpickle 3.1.1
ConfigSpace 0.4.21
contourpy 1.1.1
cycler 0.12.1
Cython 3.1.1
dadi 2.4.0
dask 2023.5.0
demes 0.2.3
demesdraw 0.1.3
distributed 2023.5.0
fonttools 4.57.0
fsspec 2025.3.0
gadma 2.0.3
gitdb 4.0.12
GitPython 3.1.44
h5py 3.10.0
importlib_metadata 8.5.0
importlib_resources 6.4.5
Jinja2 3.1.6
joblib 1.4.2
kiwisolver 1.4.7
lazy_import 0.2.2
locket 1.0.0
MarkupSafe 2.1.5
matplotlib 3.3.4
moments-popgen 1.3.1
more-itertools 10.5.0
mpmath 1.3.0
msgpack 1.1.0
nlopt 2.7.1
numpy 1.19.5
packaging 25.0
pandas 1.2.5
partd 1.4.1
pathvalidate 3.2.1
pillow 10.4.0
pip 24.3.1
psutil 7.0.0
pyaml 25.5.0
pynisher 1.0.10
pyparsing 3.1.4
pyrfr 0.9.0
python-dateutil 2.9.0.post0
pytz 2025.2
PyYAML 6.0.2
ruamel.yaml 0.16.12
ruamel.yaml.clib 0.2.8
scikit-allel 1.3.7
scikit-learn 1.1.2
scikit-optimize 0.9.0
scipy 1.6.0
setuptools 75.3.0
six 1.17.0
smac 0.13.1
smmap 5.0.2
sortedcontainers 2.4.0
tblib 3.0.0
threadpoolctl 3.5.0
toolz 1.0.0
tornado 6.4.2
typing_extensions 4.13.2
tzdata 2025.2
urllib3 2.2.3
wheel 0.45.1
xarray 0.17.0
zict 3.0.0
zipp 3.20.2
```
Oh, that is totally my fault, but it can be easily fixed. For some reason, this line is commented out, but it should not be (or at least not the whole line).
I can push an update to pip with this issue fixed, or you can add the following line at the beginning of the gadma/optimizers/bayesian_optimization.py file on your local copy of GADMA:
from .gaussian_process import SMACGaussianProcess
What option is better for you?
I edited my local copy of the script, seems to be running ok now. Will keep you posted. Thanks again for the quick responses!
Another error. Suggestions?
Traceback (most recent call last):
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/core/core.py", line 38, in job
obj.run(settings.get_optimizers_init_kwargs())
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/core/core_run.py", line 620, in run
result = self.run_without_increase(initial_kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/core/core_run.py", line 442, in run_without_increase
result = optimizer.optimize(f, variables, **opt_kwargs)
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/combinations.py", line 164, in optimize
global_result = self.global_optimizer.optimize(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/global_optimizer.py", line 277, in optimize
return super(GlobalOptimizer, self).optimize(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/optimizer.py", line 572, in optimize
self._optimize(f=f_in_opt,
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 1027, in _optimize
iter_callback(x_best, y_best, X_init, Y_init, message=message)
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/optimizer.py", line 554, in iter_callback
self.write_report(
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/optimizer.py", line 392, in write_report
self._write_report_to_stream(variables, run_info, stream)
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 961, in _write_report_to_stream
GPyOptBayesianOptimizer._write_report_to_stream(
NameError: name 'GPyOptBayesianOptimizer' is not defined
In the same file gadma/optimizers/bayesian_optimization.py you can uncomment class GPyOptBayesianOptimizer (lines 86-311) - then it should be solved.
I really appreciate everything you are doing, I recently slightly updated code for Bayesian optimization and commented out things I decided are insufficient. Seems I was wrong, but at least I just commented them out, not deleted.
Still not quite there. New error:
+ Fri Jun 6 09:13:51 EDT 2025 job GADMA_model_5pops_01 started in lThC.q with jobID=8149300 on compute-65-11
Traceback (most recent call last):
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/bin/gadma", line 5, in <module>
from gadma.core import main
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/__init__.py", line 113, in <module>
from .optimizers import get_local_optimizer, all_local_optimizers # NOQA
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/__init__.py", line 12, in <module>
from .bayesian_optimization import SMACBayesianOptimizer, SMACSquirrelOptimizer # NOQA
File "/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/optimizers/bayesian_optimization.py", line 309, in <module>
if GPyOpt_available:
NameError: name 'GPyOpt_available' is not defined
And if uncomment the line from .. import GPyOpt_available, GPyOpt in bayesian_optimization.py, I get another error:
ImportError: cannot import name 'GPyOpt_available' from partially initialized module 'gadma' (most likely due to a circular import) (/home/macguigand/software/anaconda3/envs/gadma_env_latest/lib/python3.8/site-packages/gadma/__init__.py)
Great, looks like it's running now!
On a different note, I have a question about the parameter units. The numbers I'm getting for time parameters (t1-4) seem extremely low if they are in years, as the GADMA log suggests.
[006:35:02]
All best by log-likelihood models
Number log-likelihood AIC score Model Units
Run 17 -6024.86 12093.73 [Nanc = 810042] (nAB=1.70228, nDCE=0.6455, nA=0.25516, nB=0.15695, nC=0.20901, nDE=0.12869, nD=0.0374, nE=0.01002, mAB=1.41323, mAC=0.48424, mAD=0.04536, mAE=0.00075, mBC=0.00108, mBD=1.14226, mBE=6.93572, mCD=0.62923, mCE=2.22793, mDE=0.0487, t1=0.22015, t2=0.30057, t3=0.12961, t4=0.0056) matern52_PI (theta = 2864.82) physical, time in years
Also, what are the units for the population sizes and migration rates?
I'm using the model and parameters from my first post in this thread.
Also, I should note that I'm still getting a plotting error:
Run 17 warning: failed to draw model due to the following exception: module 'moments.Plotting' has no attribute 'plot_5d_comp_Poisson'.
Hello @noscode,
Apologies, I found the documentation page describing parameter units (https://gadma.readthedocs.io/en/latest/user_manual/units.html).
It appears my results are being reported in genetic units, even though I set Relative parameters to FALSE and specified a mutation rate and sequence length.
I have follow up question about the migration rate parameters. I see that dividing the migration parameters by 2 * Nref gets me a "proportion of migrants in population per generation". If I want to convert that to number of migrants per generation, would I simply multiply by the "receiving" population size estimate?
Thanks, Dan
Hi @dmacguigan,
Sorry for slow reply. Regarding the plotting error: unfortunately, moments cannot draw comparison of model and data site frequency spectra for four and five populations. GADMA does not know about that and tries to call a function, but it does not exist.
Genetic units. Yes, your parameters are in genetic units. It is wrong for GADMA to write that time and units are physical - I will note that as a bug and fix it. Unfortunately, you will have to translate units yourself. I decided that GADMA should not translate units for manually specified models automatically. Those models are black-box functions for GADMA, and one cannot be sure that parameter 'n_something' actually refers to a population size. It is still good that you specified sequence length and mutation rate as you can see an ancestral population size in the output and use it for the translation. Please let me know if I should help you with the translation.
Regarding migration - yes, you are correct.
Best, Ekaterina
Hello @noscode,
Thank you for the detailed clarification!
I was able to finish the first down-sampled run with my data. I'm now ready to start the final local evaluation with my larger SFS.
Just to clarify, I should only need 1 repeat for this final optimization, right? Or would you still recommend running multiple repeats?
Global maxeval: 1
Number of repeats: 1
Number of processes: 1
Hi @dmacguigan,
No, you have to use the same number of repeats: GADMA needs to "resume" each repeat from the previous run by running local search. Just use the same number of repeats and same number of processes.