projectmesa
Remove all library requirements
Mesa currently ships as a "batteries included" package with requirements for all possible functionality provided. While this is nice for new users, concerns have been raised that mesa is unsuitable for integration into larger projects due to its (possibly unused) dependencies. Upon further inspection all currently required packages are optional.
-
click,cookiecutteronly used formesacommand -
jupyternot used at all -
networkxonly implicitly used by NetworkGrid -
numpyonly used for ContinuousSpace -
pandasonly used for DataFrame output of the DataCollector -
tornadoonly used for Visualization -
tqdmonly used for batch runner
Those requirements could be put under some extra_requires in the setup.py file and then users could still install mesa with pip install mesa[all] to have everything included.
Steps for this issue would include:
- [ ] Update setup.py (decide on name for "all")
- [ ] Lazy load all dependencies with helpful error messages if it fails
- [ ] Update documentation
But before working on this issue we should decide whether this is a good idea or not
Credits to @rht for starting this discussion in #614
- Small dependencies that do not affect the "closure size" and runtime much could be included by default.
click,cookiecutter,tqdmare small and fast to load. - I'd argue that
visualizationis an essential component (MASON has it by default).
Tangent: maybe there should be a meta issue tracking feature parity with MASON (source: https://github.com/eclab/mason for easy access). This would be informative for people who are considering to migrate from MASON.
My sense is that the median Mesa user will want the 'batteries included' installation with all the dependencies. Instead of asking users to specify pip install mesa[all] to get that version, could we specify pip install mesa[min] (or similar) to install the dependency-free version?
The [] syntax is only for optional dependencies via extra_requires. The only way possible is to have a package mesa-core that mesa the suite inherits from.
Yes as rht says this is not possible. A sideways idea would be to create a conda package. I think a lot of new (and probably also more experienced) python users are now using Anaconda. And there we could deploy a 'batteries included' package and keep a minimal base version at pip. That is
conda install mesa
would equal to
pip install mesa[all]
- Small dependencies that do not affect the "closure size" and runtime much could be included by default.
click,cookiecutter,tqdmare small and fast to load.
That would put us in the place to decide what are "small" and "large" dependencies.
- I'd argue that
visualizationis an essential component (MASON has it by default).
It's about making it optional, not removing it. There are a lot of other ways to visualize the results (especially if your doing research and want to create publication ready figures)
And there we could deploy a 'batteries included' package and keep a minimal base version at pip.
There would be confusing and non-standard since they are meant to be different installation methods of a package. Additionally, what if a conda package depends on the minimal version?
I don't know of a standard regarding this (if there is one, I agree we should comply), but its not unusual for the dependencies to differ. The goals of pip and conda are different. You can compare the dependencies for numpy or matplotlib for example (sorry I can't find a list for either of those, but confirmed it by testing locally).
And conda packages can only depend on other conda packages. Even if, how could that be a problem in that direction? The only source of error I can imagine is install the minimal version with pip and then install a conda package depending on the maximal/conda version of mesa. But then the worst case scenario is receiving an error message that tells users they have to install a dependency.
The
[]syntax is only for optional dependencies viaextra_requires. The only way possible is to have a packagemesa-corethatmesathe suite inherits from.
Which probably would be the cleanest solution and would also answer #134 . But I don't see the resources for this split nor the immediate need. I think there would be hardly anyone to only use the core components. And for those use cases it is easy enough to fork the repo and adapt
The difference between scipy in conda and pip is that the former has openblas and cython as requirements (https://github.com/conda-forge/scipy-feedstock/blob/master/recipe/meta.yaml), and in pip, openblas is optional. The pip version also has a mechanism not to update numpy if possible (https://github.com/scipy/scipy/blob/master/setup.py#L419-L428). Anyone would have the exact same functioning scipy regardless of the installation method, with performance being the only difference.
One use case of the minimal version is abcEconomics (https://github.com/ab-ce/abce) -- we are planning to reshape the scheduler, the agent verbs, etc (maybe later logging as well but this is too complicated) to be consistent with Mesa / to use Mesa as a basis (cc: @DavoudTaghawiNejad). One plus point of this is that I will port abcE's multi-core scheduling (via multiprocessing) to Mesa. I'm sure there are other projects that may benefit from this deduplication of the ABM layer / API.
there is always the pip mesa --no-dependency option if you don't want to install it without dependencies.
there is always the pip mesa --no-dependency option if you don't want to install it without dependencies.
That's for manual install, but there is no such option when being specified in requirements.txt. Nor in Pipfile, toml file (I forgot the filename for this) or other requirements.txt successors.
I think I have a clear use case for this. I am running mesa in the browser using pyscript (which in turn relies on pyodide). In pyscript, it is possible to import pure Python packages from PyPI without any problems. However, mesa does not import, and this appears to be due to the tornado dependency. To solve this, I needed to make a local copy of mesa and remove the things I didn’t need. However, this is of course not a desirable solution.
My problem would be solved if the tornado based visualization was an extra. As far as I understand, the mesa version 2+ is anyway moving away from that visualization style, so maybe it would be quite acceptable for most users to specify mesa[tornado] if they wanted that visualization?
My problem would be solved if the tornado based visualization was an extra. As far as I understand, the mesa version 2+ is anyway moving away from that visualization style, so maybe it would be quite acceptable for most users to specify mesa[tornado] if they wanted that visualization?
Looking at your repo, it seems that you don't require the visualization component at all, even for the new Solara-based one.
I think you could do fine with pip install --no-deps mesa separately from pip install -r requirements.txt, because there is no way to specify --no-deps for 1 package in requirements.txt
Since import mesa will by default perform imports from the Tornado stuff, so I think a workaround would be to from mesa.space import xinstead.
Since
import mesawill by default perform imports from the Tornado stuff, so I think a workaround would be tofrom mesa.space import xinstead.
Interesting observation, maybe we could wrap the visualization import in a try...except block so it only gets imported when the package is installed.
Speaking of which, @jakobaxelsson, your project reminds me of https://github.com/mlc-ai/web-llm, recently featured in latent.space, where you could use it to run 70B Llama 2 in the web browser.
Thank you so much for your advice. The suggested workaround helped me solve the problem. Although it makes my code slightly less clear in a few places, it is still far, far better than having to maintain a local copy of mesa.
In the end, what I needed to do was the following. Since pyscript does not seem to have an option to install packages without dependencies, I had to fall back on the underlying pyodide package install, using micropip.install(...). This function handles that:
async def import_mesa():
import micropip
await micropip.install("mesa", deps = False)
try:
import mesa
except:
pass
Initially, I did not have the try: … except: …. However, this leads to exceptions when trying to do e.g. from mesa.agent import agent as MesaAgent. After adding the try-clause, it works (but I cannot explain why this was needed, and it was only by accident that I discovered that this would work.)
Although I can certainly live with this workaround, I still think it might be a good idea to provide optional extras when installing mesa, to be able to have more flexibility for users who only need a smaller part of this large library. It is also not a very intrusive change. For existing users, they would need to do a one time change in their requirements.txt changing mesa to mesa[all] the next time they upgrade to a new version. For new users, it doesn’t make a difference. Probably, it would be possible to give reasonable error messages if a user tries to use a part of the library that has not been installed. (But again, my issue is solved, so this is just my humble opinion about a useful future extension to mesa.)
from mesa.agent import agent as MesaAgent shouldn't error. What was the error message you encountered?
Regarding with mesa[all], I'd say it is much more common for people to install the whole thing. I'd consider mesa-core/mesa-minimal instead.
I get ModuleNotFoundError: No module named 'tornado'. You can easily replicate this if you go to https://pyodide.org/en/stable/console.html and type in the following lines:
>>> import micropip
>>> await micropip.install("mesa", deps = False)
>>> await micropip.install("pandas", deps = False)
>>> await micropip.install("networkx", deps = False)
>>> from mesa.model import Model as MesaModel
However, if you then evaluate the last line once more, there is no error.
It could be that https://github.com/projectmesa/mesa/blob/main/mesa/init.py is executed even when you do from mesa.model import Model as MesaModel.
According to https://discuss.python.org/t/help-packaging-optional-application-features-using-extras/14074/4, it is not possible to remove requirements from default requirements. As such, mesa[minimal] can't be done.
A short term solution would be, as @Corvince said, to wrap imports inside a try-except.
As such,
mesa[minimal]can't be done.
But a separate mesa-core/mesa-minimal should do.
Would it be an option to change https://github.com/projectmesa/mesa/blob/main/mesa/init.py so that each separate import statement is wrapped in a try-except clause? In that way, mesa would initiate whatever it can, and there could be a warning message for those imports that fail? (Just a quick thought, there may be consequences that I don't fully understand...)
Would it be an option to change https://github.com/projectmesa/mesa/blob/main/mesa/init.py so that each separate import statement is wrapped in a try-except clause? In that way, mesa would initiate whatever it can, and there could be a warning message for those imports that fail? (Just a quick thought, there may be consequences that I don't fully understand...)
Yes this is what I was thinking as well. Only that right now we only have to do this for the visualization, the rest is part of this repo so should be available.
Maybe you want to submit a PR with the change?