PyDDM
PyDDM copied to clipboard
mwshinn
Multithreading-related issues
Dear pyddm developers! Thank you for developing this excellent package. But I ran into a problem and don't know if it's a bug. Refer to your example (monkey) to fit the leaked-DDM. But if it's dealing with multiple participant loops, it won't be able to import NumPy, even if I re- "import numpy as np" before each participant fit. I'm not sure if this has something to do with multithreading. Looking forward to your answers!
If I “pyddm.set_N_cpus (1)” rather than >1 after importing numpy, there won't be any problems
Hi Hyphcha,
To help me help you, please read this about how to submit a bug report: https://pyddm.readthedocs.io/en/latest/contact.html
import pandas
import numpy as np
import pyddm
import pyddm.plot
from pyddm import Sample
with open("roitman_rts.csv", "r") as f:
df_rt = pandas.read_csv(f)
pyddm.set_N_cpus(2) # if only 1, it works well
# Two monkeys
df_rt = df_rt[df_rt["rt"] > .1]
df_rt = df_rt[df_rt["rt"] < 1.65]
conditions = ["coh", "monkey", "trgchoice"]
model_leak = pyddm.gddm(
drift=lambda driftcoh,leak,coh,x : driftcoh*coh - leak*x,
bound=lambda bound_base,invtau,t : bound_base * np.exp(-t*invtau),
nondecision="ndtime",
parameters={"driftcoh": (-20,20),
"leak": (-5, 5),
"bound_base": (.5, 10),
"ndtime": (0, .5),
"invtau": (.1, 10)},
conditions=["coh"])
for monkey in df_rt["monkey"].unique():
# import numpy as np
filtered_data = df_rt[(df_rt["monkey"] == monkey)]
roitman_sample = Sample.from_pandas_dataframe(filtered_data, rt_column_name="rt", choice_column_name="correct")
model_leak.fit(sample=roitman_sample, verbose=False)
model_leak.show()
print("Parameters:", model_leak.parameters())
Hi! dear max, that's my code. the error is NameError: name 'np' is not defined Just as your example [https://pyddm.readthedocs.io/en/stable/quickstart.html] but two monkeys in loop.
Hi Hyphcha,
This is still not all the information requested. Please go through the list on that page carefully and make sure to include everything. E.g., I don't think this is a minimal example, and the full error output is not included. Additionally it includes model fitting, see if you can eliminate this while still maintaining the error.
---------------------------------------------------------------------------
RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\multiprocess\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\multiprocess\pool.py", line 48, in mapstar
return list(map(*args))
^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\paranoid\decorators.py", line 115, in _decorated
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\model.py", line 620, in solve
return self.solve_numerical_implicit(conditions=conditions, return_evolution=return_evolution, force_python=force_python)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\paranoid\decorators.py", line 115, in _decorated
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\model.py", line 981, in solve_numerical_implicit
return self.solve_numerical(method="implicit", conditions=conditions, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\paranoid\decorators.py", line 115, in _decorated
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\model.py", line 795, in solve_numerical
return self.solve_numerical_c(conditions=conditions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\model.py", line 704, in solve_numerical_c
drift = np.asarray(get_drift(x=self.x_domain(conditions=conditions), t=0, conditions=conditions))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\paranoid\decorators.py", line 115, in _decorated
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\model.py", line 409, in x_domain
B = max([self.get_dependence("bound").get_bound(t=t, conditions=conditions) for t in self.t_domain()])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\functions.py", line 981, in get_bound
return bound(**{v: getattr(self, v) for v in _required_parameters_bound}, **{v: conditions[v] for v in _required_conditions_bound}, **extras)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ZhaoPenggeng\AppData\Local\Temp\ipykernel_5296\1368485515.py", line 14, in <lambda>
NameError: name 'np' is not defined
"""
The above exception was the direct cause of the following exception:
NameError Traceback (most recent call last)
Cell In[2], line 25
23 filtered_data = df_rt[(df_rt["monkey"] == monkey)]
24 roitman_sample = Sample.from_pandas_dataframe(filtered_data, rt_column_name="rt", choice_column_name="correct")
---> 25 model_leak.fit(sample=roitman_sample, verbose=False)
26 model_leak.show()
27 print("Parameters:", model_leak.parameters())
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\model.py:269, in Model.fit(self, sample, fitparams, fitting_method, lossfunction, verify, method, verbose)
267 if lossfunction is None:
268 lossfunction = LossLikelihood
--> 269 fit_adjust_model(sample=sample, model=self, fitparams=fitparams,
270 fitting_method=fitting_method,
271 lossfunction=lossfunction, verify=verify,
272 method=method, verbose=verbose)
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\functions.py:377, in fit_adjust_model(sample, model, fitparams, fitting_method, lossfunction, verify, method, verbose)
375 if "disp" not in fitparams.keys():
376 fitparams["disp"] = verbose
--> 377 x_fit = differential_evolution(_fit_model, constraints, **fitparams)
378 elif fitting_method == "hillclimb":
379 x_fit = evolution_strategy(_fit_model, x_0, **fitparams)
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\scipy\_lib\_util.py:440, in _transition_to_rng.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
433 message = (
434 "The NumPy global RNG was seeded by calling "
435 f"`np.random.seed`. Beginning in {end_version}, this "
436 "function will no longer use the global RNG."
437 ) + cmn_msg
438 warnings.warn(message, FutureWarning, stacklevel=2)
--> 440 return fun(*args, **kwargs)
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\scipy\optimize\_differentialevolution.py:501, in differential_evolution(func, bounds, args, strategy, maxiter, popsize, tol, mutation, recombination, rng, callback, disp, polish, init, atol, updating, workers, constraints, x0, integrality, vectorized)
484 # using a context manager means that any created Pool objects are
485 # cleared up.
486 with DifferentialEvolutionSolver(func, bounds, args=args,
487 strategy=strategy,
488 maxiter=maxiter,
(...)
499 integrality=integrality,
500 vectorized=vectorized) as solver:
--> 501 ret = solver.solve()
503 return ret
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\scipy\optimize\_differentialevolution.py:1176, in DifferentialEvolutionSolver.solve(self)
1171 self.feasible, self.constraint_violation = (
1172 self._calculate_population_feasibilities(self.population))
1174 # only work out population energies for feasible solutions
1175 self.population_energies[self.feasible] = (
-> 1176 self._calculate_population_energies(
1177 self.population[self.feasible]))
1179 self._promote_lowest_energy()
1181 # do the optimization.
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\scipy\optimize\_differentialevolution.py:1337, in DifferentialEvolutionSolver._calculate_population_energies(self, population)
1335 parameters_pop = self._scale_parameters(population)
1336 try:
-> 1337 calc_energies = list(
1338 self._mapwrapper(self.func, parameters_pop[0:S])
1339 )
1340 calc_energies = np.squeeze(calc_energies)
1341 except (TypeError, ValueError) as e:
1342 # wrong number of arguments for _mapwrapper
1343 # or wrong length returned from the mapper
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\scipy\_lib\_util.py:657, in _FunctionWrapper.__call__(self, x)
656 def __call__(self, x):
--> 657 return self.f(x, *self.args)
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\functions.py:359, in fit_adjust_model.<locals>._fit_model(xs)
357 x = p.minval
358 s(m, x)
--> 359 lossf = lf.loss(m)
360 if verbose:
361 _logger.info(repr(m) + " loss="+ str(lossf))
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\paranoid\decorators.py:115, in _wrap.<locals>._decorated(*args, **kwargs)
112 def _decorated(*args, **kwargs):
113 # Skip verification if paranoid is disabled.
114 if Settings.get("enabled", function=func) == False:
--> 115 return func(*args, **kwargs)
116 # We only run this function once for performance reasons, and
117 # then pass it as an argument to each check function.
118 sig = inspect.Signature.from_callable(func)
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\models\loss.py:166, in LossLikelihood.loss(self, model)
161 @accepts(Self, Model)
162 @returns(Number)
163 @requires("model.dt == self.dt and model.T_dur == self.T_dur")
164 def loss(self, model):
165 assert model.dt == self.dt and model.T_dur == self.T_dur
--> 166 sols = self.cache_by_conditions(model)
167 loglikelihood = 0
168 for k in sols.keys():
169 # nans come from negative values in the pdfs, which in
170 # turn come from the dx parameter being set too low. This
(...)
175 # an exception may be the better way to handle this to
176 # make sure it doesn't go unnoticed.
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\models\loss.py:93, in LossFunction.cache_by_conditions(self, model)
79 """Solve the model for all relevant conditions.
80
81 Solve `model` for each combination of conditions found within the
(...)
90
91 """
92 from ..functions import solve_all_conditions
---> 93 return solve_all_conditions(model, sample=self.sample, method=self.method)
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\pyddm\functions.py:539, in solve_all_conditions(model, sample, condition_combinations, method)
535 if paranoid_settings.get('enabled') is False:
536 # The *2 makes sure that this runs on all subprocesses,
537 # since you can't broadcast commands to all processes
538 _parallel_pool.map(lambda x : paranoid_settings.set(enabled=False), [None]*_parallel_pool.n_cpus*2)
--> 539 sols = _parallel_pool.map(meth, conds, chunksize=1)
540 for c,s in zip(conds, sols):
541 cache[frozenset(c.items())] = s
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\multiprocess\pool.py:367, in Pool.map(self, func, iterable, chunksize)
362 def map(self, func, iterable, chunksize=None):
363 '''
364 Apply `func` to each element in `iterable`, collecting the results
365 in a list that is returned.
366 '''
--> 367 return self._map_async(func, iterable, mapstar, chunksize).get()
File d:\Tools\Miniconda\envs\PyDDM\Lib\site-packages\multiprocess\pool.py:774, in ApplyResult.get(self, timeout)
772 return self._value
773 else:
--> 774 raise self._value
NameError: name 'np' is not defined
The above is the complete error message. my pyddm=0.8.1 (newest), python=3.12.9, numpy=1.26.4 I'm sorry I'm a new GitHub user and I'm not very familiar with submitting bug reports.
Again, please follow ALL of the instructions on the link I sent, including condensing your code down to the minimal possible example that still shows the bug, trying to reproduce while eliminating fitting, etc.
The most streamlined code is already above. It's just a small change to your example. I've also uploaded it to my github repository.