Non-top level namespace packages in `pip install -e .` mode

[kimbar]

Setup

Let's consider a package with following structure:

site-packages
└┬ house + __init__.py
 ├─ walls.py
 └┬ corridor
  ├┬ livingroom + __init__.py
  │├─ couch.py
  │└─ table.py
  └┬ kitchen + __init__.py
   └ oven.py

The corridor is a namespace package, and its inhabitants live in different distribution packages. Hence, we have a total of three setup.py files in the source code: for house, for livingroom and for kitchen. It is relatively simple to configure it all for a final installation in site-packages.

The problem arises when we want to install all three with pip install -e . for development. There is this helpful sample-namespace-packages repo where you can test different ways of installing namespace packages, but this sample assumes that your namespace is a top-level package in site-packages. This is not the case here.

After installation of the example above, pip will create appropriate *.egg-link and easy-install.pth files in site-packages. The PEP 420 uses a concept of a "parent path" of a namespace package, which is sys.path if the namespace package is top-level. And the sys.path is properly populated from easy-install.pth as far as I can tell.

In our example however, the "parent path" is house.__path__ which I cannot convince to include source directories of kitchen and livingroom (they land in sys.path all alright, but they are useless there as far as I can imagine.)

In the end I resorted to an ugly hack, and I populate the "parent path" in house/__init__.py directly from easy-install.pth. It works, and I didn't had to write any custom import loaders, but it is a really cumbersome piece of code that deploys only because it rises an ImportError outside of the pip install -e . situation (I can share the exact solution if anyone is interested).

The problem

Is there a way to convince vanilla importlib and/or setuptools to properly fill the parent path of corridor (which is house.__path__) when (sub)packages are installed for development with pip install -e .?

[merwok]

This should be reported to pip, if it’s still relevant.

To determine if it is: re-test with native namespace package only + recent pip version (with standards-based develop installs).

[abravalheri]

I think you meant packages. Is there a way to convince vanilla importlib and/or setuptools to properly fill the parent path of corridor

Have you tried using a src-layout without __init__.py at the namespace root (native namespaces from PEP 420)? That is probably the easiest way. Anything else it is likely to be fragile...

You may also have to omit the __init__.py all the way up in some of the distributions (I think you can have one of them with an __init__.py, but not all of them, but I am not sure. Best to test).

[merwok]

You may also have to omit the __init__.py all the way up in some of the distributions

I think you meant packages, and I don’t think that is needed. someroot can be a regular package and someroot.things a namespace package.

[abravalheri]

I think you meant packages, I don’t think that is needed. someroot can be a regular package and someroot.things a namespace package.

No I meant distribution, as in a wheel file.

Imagine that you have distribution1 which contains:

package
|- __init__.py
`- nested_namespace
   `- mod_from_dist1.py

Now if you have distribution2 which also have an init.py under package:

package
|- __init__.py
`- nested_namespace
  `- mod_from_dist2.py

Sure, mode_from_dist1.py and mod_from_dist2.py can coexist inside the nested namespace, it is a namespace after all. But if both wheels have __init__.py at the top-level, what will pip do?

It could consider that both distributions provide the same top-level package... With that assumption, it could decide to uninstall completely the first distribution before installing the second... I don't remember exactly what happens here, that is why I suggested to test first.

(OK this discussion is more relevant for regular installs, but the same package layout may also influence the way import machinery finds the nested namespace)

[merwok]

Right, that will cause issues. Doesn’t flask do it with its namespace flask.ext for plugins? Could look at how it does.