Feature request: Statically link against ``libncurses``

[christian-krieg]

Hey there! First of all: Thanks for this great package!

TL;DR:

I propose to statically link gnureadline not only against readline, but also against ncurses. This would enable usage in environments where ncurses is not available, making this Stand-alone GNU readline module even more stand-alone.

Detailed request

I am using gnureadline with a standalone Python build which is linked against libedit only, on a Linux system with minimal support for shared libraries. Unfortunately, it fails on import, because it couldn't find libtinfo.so.6 (provided by ncurses):

>>> import gnureadline
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libtinfo.so.6: cannot open shared object file: No such file or directory

To verify, I built gnureadline and checked it's shared-object dependencies. Indeed, gnureadline depends on libtinfo.so.6:

$ git checkout main
$ rm -rfv build/ gnureadline.egg-info readline wheels
$ python3 -m pip wheel --wheel-dir=wheels .
$ ldd build/lib.linux-x86_64-cpython-39/gnureadline.cpython-39-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007ffdf1e39000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f1b5610e000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1b560ec000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1b55f18000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1b561ca000)

So I went on to make gnureadline statically linking against ncurses in https://github.com/christian-krieg/python-gnureadline/tree/feature/static-ncurses. With this, the dependency on libtinfo.so.6 goes away:

$ git checkout feature/static-ncurses
$ rm -rfv build/ dist/ gnureadline.egg-info readline wheels
$ python3 -m pip wheel --verbose --wheel-dir=wheels .
$ ldd build/lib.linux-x86_64-cpython-39/gnureadline.cpython-39-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007ffc172e8000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb1e3250000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb1e307c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb1e332a000)

After installing on the target machine, importing gnureadline works as expected:

>>> import gnureadline
>>>

Discussing a potential pull request

My intention is to submit a PR; before submitting, I would like to request guidance on the following issues:

1. How could statically linking against libncurses impact users of gnureadline? Would such a move be transparent to them? What would be potential side effects?
2. Should static linking against libncurses be optional? If so, what would be the preferred way of specifying such an option in the build system? Would there be a simple way to manage (Python) distribution packages with different build options?
3. When working on the fix, I noticed conditional code to check for MacOS build systems. As I am not a Mac user, I would not be able to test the build process and the correctness of the built module's functional behavior. Would there be someone who could test the PR on Mac?
4. I added an ncurses directory, holding a source archive and a build script build-ncurses.sh. To increase expressiveness in presence of an additional library, I adapted rl/build.sh, and created a new rl/build-readline.sh. Please let me know whether such a file name change is desirable. Of course, I am very open to port my changes back to rl/build.sh.

Thank you very much again for this great package!

[ludwigschwardt]

Thanks for your effort and kind words, Christian!

This is an interesting idea. You seem to have quite a tricky system (and I've seen many weird ones in the past thanks to gnureadline!).

Have you tried the precompiled manylinux wheels on PyPI? They tend to have all libraries statically included, via auditwheel. I can understand that you would need to compile from source on an old distro or with older Pythons.

[christian-krieg]

Hi @ludwigschwardt, many thanks for your kind response! I had a look at the precompiled manylinux wheels on PyPI already; Although I generated above example code/output using Python 3.9, I would need a precompiled wheel for Python 3.12, which isn't available yet.

Thank you very much for pointing out auditwheel. I haven't been aware of it.

Anyway, I feel that my use case may be very limited, which may be too much risk for other users to break their systems when switching to a gnureadline that is statically linked to ncurses. Probably an independent fork may be better the better option than incorporating this change into your gnureadline package? What is your opinion on this?

[ludwigschwardt]

I hope to make a Python 3.12 wheel soon... I've been working on a new branch and the other good development is that there are now Apple M1 runners readily available to make wheels for those platforms too. Your request is extra fuel to the fire :-)

[ludwigschwardt]

Hi Christian,

I had a look at python-build-standalone again and now understand better what it's about.

This section in their docs makes me pause though:

libedit/libreadline link against a curses library, most likely ncurses. And ncurses has tie-ins with a terminal database. This is a thorny situation, as terminal databases can be difficult to distribute because end-users often want software to respect their terminal databases. But for that to work, ncurses needs to be compiled in a way that respects the user’s environment. On macOS, we use the system libedit and libncurses, which is typically provided in /usr/lib. On Linux, we build libedit and ncurses from source and statically link against their respective libraries. Project releases before 2023 linked against readline on Linux.

While libreadline is relatively straightforward to build from scratch, libncurses sounds like a cross-platform hassle... Python-build-standalone still uses the shared ncurses library on macOS (I suspect, since it is well hidden these days 😂), so this feature could potentially be restricted to Linux systems.

It's a can of worms I'd rather not open... But please go ahead on your side 😁 I'll at least update the docs to mention that it's not quite stand-alone.

[ludwigschwardt]

This is not considered at the moment.