BatchRunnerMP

[TabernaA]

Describe the bug Running the BatchRunnerMP I initially got the pickle error: https://github.com/projectmesa/mesa/issues/787

I downloaded the last update and manage to solve the error. Yet my BatchrunnerMp never starts

batch_run = BatchRunnerMP(KSModel , nr_processes = 2, variable_parameters= variable_params, fixed_parameters=fixed_params, iterations=iterations, max_steps=max_steps, model_reporters = model_reporters, display_progress=True)

batch_run.run_all()

Everytimre I run it I do not get any error but does not even display the progresses. if I stop the kernel it is always at this point:

in run_all for params, model_data in self.pool.imap_unordered(self.run_wrapper, run_iter_args):

File "C:\Users\tabernaa\Anaconda3\lib\multiprocessing\pool.py", line 856, in next self._cond.wait(timeout)

File "C:\Users\tabernaa\Anaconda3\lib\threading.py", line 302, in wait waiter.acquire()

KeyboardInterrupt

Thanks in advance for your help Have a nice day A

[tpike3]

@TabernaA Run BatchRunnerMP with one processor (nr_processes=1), this allows for debugging in BatchRunnerMP as it guarantees to the processor talking to your screen will be the one that errors.

[TabernaA]

@tpike3 So basically with one processor works, but it does not with >1. I tried to change laptop and there it works so it is very strange.

[TabernaA]

Thanks a lot for the reply!

[tpike3]

@TabernaA very interesting, do the laptops have different operating systems or are they running different versions of Mesa, python etc? If they have different versions of multiprocessing that could also be a big reason as python made some significant changes to it.

Let me know if you get the chance.

[TabernaA]

@tpike3 So basically the one that did not work with nr_processor > 1 was my laptop (Widows 10, Intel Core i5-8250u CPU @ 1.69Ghz 8 logical processors).

The one where it works vey well is a server (Windows server 2012 R2, CPU E5-2659 v3 2.30Ghz, 2 processors but it got 40 CPU with the CPU count).

I am quite new in programming in general so understanding what is happening is beyond my reach now. Still with exactly the same code it worked with the server and not in the laptop.

[tpike3]

@TabernaA Sorry for the delay on this. Did you trying running your laptop with one processor (nr_processor =1) and if so did you get an error message?

Could you tell me your Mesa and multiprocessing version?

Multiprocessing is a constant challenge, but happy to struggle together.

[TabernaA]

@tpike3 I have to re-frame this. Sometimes it works sometime it does not. I am investigating, but it is very strange how sometimes it does not start and sometimes just after closing and reopening it works. Yet, in my laptop even with the multiprocessor is very slow, without adding much speed with respect to the individual run. The opposite in the server where with the multiprocessor I reduced drastically the time. I am starting to think that, in my laptop, even if the CPU count gives me multiple processors there is only actually one working? It is very strange. I am now using MESA versione 0.8.8.1. I was using the multiprocessor version I downloaded from here but now I am using the same as the MES version since I got the same error.

Yes it is a struggle, I am already very happy it works in the server, it cut me so much time. Also an important side note (let me know if I should open a related issue or it is okay here): I can't reproduce my multiprocessing run twice. For instance: I do my multiprocessing run in the server and get some results. If I restart the kernel and run it again I get different results. I have randomness elements from numpy and random, yet I set the seed at the begining. Indeed, If I do the same process with a single run I manage to get the same results once I restart the Kernel. The problem comes out when I do a batch run, then I get different results.

Is there a way to get the same results even with multiprocessing? Thanks for the support and have a nice day A

[tpike3]

@TabernaA Sorry for the delay and those are some tough questions. For randomness you have to use model.random throughout, that is the only way to use the model seed reusing random or using numpy.random will add another random element and mess up your results.

To see if your processers are all being used I would look at your Task Manager and performance and you can see the number of processors being employed

For the working sometimes and not, I am not sure, but sounds like there is a condition that only occurs occasionally (e.g. two agents occasionally meeting that are incompatible)...those problems are always difficult

[tpike3]

@TabernaA were you able to resolve your issues?

[TabernaA]

Hi, I changed laptop (from Windows to Mac), and I did not work on the model for a bit. I came back on the multiprocessing todays and I am now having this problem:

https://github.com/projectmesa/mesa/issues/952

I understood it was a Windows and not mac problem. So basically I have the opposite. The same code runs just fine in windows server with 40 multiprocessors but in my mac I get this problem. I tried to place

        if __name__ == '__main__':
            freeze_support()
            ...

I tried in different places but I still keep receiving a constant loop text. The problem is related to the fact that i moved from Spyder to Sublime text. Interestingly if i adjust the code to run back to spyder (on the same Mac) it works without errors. Any hints on why sublime creates this problem? Thanks in advance A

[tpike3]

@TabernaA Wow that is an interesting one, just due to the specificity I do not think there is anyway I can recreate it. With Spyder providing more features than sublime I think there is two options:

1. There is an issue not related to multiprocessing that occurs the way sublime and spyder handle basic functions similar to this overflow . I would test in sublime running 1 processor to see.
2. It is directly related to how spyder and sublime handle multiprocessing and so for sublime specifically there is probably some specific code you need. For this I would try and get just a basic multiprocessing code to work in sublime.

Hope this helps, I am curious on what the issue is but trying this two approaches should help identify it. Let me know.

[tpike3]

@TabernaA Have you been able to resolve this? Let me know, otherwise I will assume the issue is resolved.

[Isabelle-EY]

Hi! I have the same issue. BatchrunnerMP works fine for one processor, but not for multiple. I am using mesa 0.9.0 and multiprocessing 2.6.2.1. Any suggestions?

[rht]

BatchrunnerMP is deprecated. You should use batch_run. You can import it with `from mesa.batchrunner import batch_run. But we are releasing Mesa 1.0.0 this month, which lets you do

import mesa
...
# somewhere in the code
mesa.batch_run(...)

[Isabelle-EY]

Thanks!!

[rht]

The issue (even on batch_run) and its solution have been well documented in https://mesa.readthedocs.io/en/stable/useful-snippets/snippets.html#using-multi-process-batch-run-on-windows. Closing.