Conda build?

[adam2392]

Is your feature request related to a problem? Please describe.

I'm on a M1 Mac and trying to install graspologic. However, the pip install is pretty buggy for me in general, and so conda usually is the way to go to install things with numba/cython/etc. dependencies. Is there the possibility of having a conda release as well as a pypi release?

However, when installing, I get the following error:

(mnex64) adam2392@Adams-MBP-2 graspologic % pip install graspologic
Requirement already satisfied: graspologic in /Users/adam2392/miniforge3/lib/python3.9/site-packages/graspologic-0.3.0-py3.9.egg (0.3.0)
Collecting anytree>=2.8.0
  Using cached anytree-2.8.0-py2.py3-none-any.whl (41 kB)
Collecting beartype>=0.7.1
  Using cached beartype-0.8.0-py3-none-any.whl (455 kB)
Collecting gensim<=3.9.0,>=3.8.0
  Using cached gensim-3.8.3.tar.gz (23.4 MB)
Collecting graspologic
  Using cached graspologic-0.2.0.tar.gz (5.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
  Using cached graspologic-0.1.0.tar.gz (5.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
ERROR: Cannot install graspologic==0.1.0, graspologic==0.2.0 and graspologic==0.3.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    graspologic 0.3.0 depends on graspologic-native>=1.0.0
    graspologic 0.2.0 depends on graspologic-native
    graspologic 0.1.0 depends on graspologic-native

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Describe the solution you'd like

I'm not exactly sure what the problem is, but if there is a conda build then that would alleviate this issue I think.

[daxpryce]

@adam2392 So I know for sure we aren't publishing graspologic-native for the m1 - at the time I created the build matrix for it, there were no github workers that were for macos on M1.

If that has changed I'm more than happy to configure those builds, otherwise you're going to have to build graspologic native manually until we can get those into a normal CICD flow.

As for the rest of it; I have no idea what conda entails. Our pip list is pretty straight forward so I don't see why it couldn't be a conda publish, I just don't know the first thing about it at all (I have the reverse issue, where nothing ever worked for me in conda and always worked with pip - bizarre, amirite?)

I'm happy to work through it with you though, I just don't know the first thing about it or getting it to work locally. If you're interested in diving a bit deeper on it I'll try to make things work from our end!

[daxpryce]

There's actually a chance I could get a graspologic-native build for m1 on the github runners to publish, but I have no way of testing it. If I were to do that in a snapshot, could I ask you to try to install it and see if it works?

[bdpedigo]

@adam2392 seems like what @daxpryce suggests is worth figuring out regardless.

As far as your original request goes, I'm open to setting up a conda-forge feedstock once this gets fixed, but first it would require that all of our dependencies are also on conda-forge. This means we'd need graspologic-native as well as hyppo (cc @sampan501) on there. I'm not sure what would be involved for graspologic-native as its not really a python package but we could look into it if there's interest (and if hyppo wants to get on there too).

[bdpedigo]

Also, as an immediate fix, I'm guessing you could just clone the repo and install locally, ignoring dependencies (i.e. just not installing graspologic-native). As long as you don't actually need graspologic.partition.leiden everything else might work fine

[daxpryce]

layouts uses it too, so if you don't touch leiden or layouts you are fine

[rajpratyush]

I think we had discussion of whether should we have CI/CD pipeline for graspologic #650

[dokato]

I'm trying to install graspologic on my Mac M1, pip doesn't work atm, so I tried to install locally as @bdpedigo advised, but it ends with an error:

UPDATING build/lib.macosx-11.0-arm64-3.9/matplotlib/_version.py
set build/lib.macosx-11.0-arm64-3.9/matplotlib/_version.py to '3.3.0'
error: Setup script exited with error: Failed to download FreeType. Please download one of ['https://downloads.sourceforge.net/project/freetype/freetype2/2.6.1/freetype-2.6.1.tar.gz', 'https://download.savannah.gnu.org/releases/freetype/freetype-2.6.1.tar.gz'] and extract it into build/freetype-2.6.1 at the top-level of the source repository.

even though I did what error message avised, which is I put downloaded freetype to the build directory:

$ ls build 
bdist.macosx-11.0-arm64 freetype-2.6.1          lib

EDIT:

Installation works with --no-deps, but yet it still complains about the lack of hyppo while importing.

Any timeline for ARM architecture compatibility?

[bdpedigo]

hi @dokato, sorry you are having trouble with this. I can't speak to ARM timeline, @daxpryce?

what happens if you install hyppo (and all other dependencies besides graspologic-native) manually, and then do the --no-deps install? AFAIK graspologic-native was the only issue here.

[daxpryce]

hi @dokato

I'm still looking for candidates to test whether these m1 builds even work. If you're willing to try out a pre-release of graspologic-native I can try to get a build together, but it won't be until next week some time. I'm not even sure I can do the build yet because Github actions don't have a dedicated m1 based server to build for, but there are some rumors that I can do it on one of the amd64 runners for m1. Since I had no committed tester, I didn't bother putting it together since it was unclear it would even work and no way to verify it's success. I'm loathe to publish something and then have to yank it after leaving it sit out there for months untested until it finally fails.

If you can't commit to trying some builds out, you can at least feel free to build graspologic-native locally; the instructions @ https://github.com/microsoft/graspologic-native#building should get you an install built locally.

And, of course, if anyone out there has "packaging an sdist for later building by maturin and rust" experience, I'd love some help in making an sdist available with a compile-on-target-machine path for any architecture not supported by the github action runners we have available

[dokato]

@daxpryce sure, happy to help and test that!

Meanwhile, I created a local whl for python v 3.9 and works like a charm (minus hyppo).

[daxpryce]

Fantastic, I'll create another build pipeline specifically for m1 builds and we'll publish them just to github for now (so you'll have to do manual download / install to test vs. using pip or conda-pointing-at-pypi). Can you send me an email @ [email protected] so we can run through things over email instead of tickets? I will report back once we have some results and we can let this ticket go into stasis in the meantime

[bdpedigo]

saw this, not sure if helpful https://github.blog/changelog/2022-08-09-github-actions-self-hosted-runners-now-support-apple-m1-hardware/

[johandahlberg]

I'm wondering if there has been any progress on this issue. We are packaging our application pixelator using conda, and since graspologic is an important dependency for us we would be keen to see a conda package for graspologic. Is there anything I can do to help make such a package available?

[bdpedigo]

hi @johandahlberg - I'm totally open to revisiting this, but I don't have the bandwidth to do it myself. I assume you are talking about adding a conda-forge feedstock. if so, perhaps this is all that is needed? https://github.com/conda-forge/staged-recipes?tab=readme-ov-file#grayskull---recipe-generator-for-python-packages-on-pypi

to make this sustainable I'd also want some automated way of maintaining/updating this, but i know very little about how this works. i cant tell if the conda-forge feedstock basically does this already. but perhaps you know more about maintaining conda-forge packages?

[bdpedigo]

p.s. i am on an M2 mac now and have not had any of the issues with apple silicon hardware that I believe was holding this up in the past somehow, so here's hoping that's true for everyone else...

[johandahlberg]

I haven't done it before, but the way I understand it is that once there is a conda-forge feedstock the process should be very simple (if not fully automatic) as long as your dependencies do not change to much. I have a work in progress recipe up here now: https://github.com/johandahlberg/staged-recipes/tree/graspologic-recipe and I'd be happy finish that up, add you as a maintainer, and try to get it into conda-forge.

Right now I am waiting to get this merged https://github.com/microsoft/graspologic-native/pull/39 - or if that is not possible I will have to look into some alternative approach.

[bdpedigo]

let me get in touch with the maintainer of that package - otherwise your plan sounds good

[johandahlberg]

Thanks! I'll try to move that along and let you know how I progress.

[johandahlberg]

This is now a conda package available from conda-forge: https://anaconda.org/conda-forge/graspologic

[bdpedigo]

thank you for your work on this @johandahlberg, appreciate it!