Add option to filter files from folders installed using install_subdir

[lithomas1]

Being able to use install_subdir is a nice feature to avoid having to list individual files with install_sources in the meson build file.

However, sometimes there are files(e.g. test data), .DS_Store folder on macOS, that you don't want to include, and there's no good way to exclude them currently, without resorting to install_sources.

An easy way to fix this is to have a config file(similar in syntax to maybe .gitignore or MANIFEST.in), where you can exclude files.

It's also probably best for this logic to live in meson-python, since the only time you'd really care is when your building a wheel.

[FFY00]

I think this is something that may make more sense to implement in Meson directly.

@eli-schwartz, thoughts?

[dnicolodi]

Most definitely something that would need to be implemented in Meson. The last time I discussed this with @eli-schwartz there wasn't much excitement about adding this feature, but it was not rejected either.

[rgommers]

MANIFEST.in has always been a footgun, so I'm also not too enthusiastic about this.

However, I believe this can already be done, using .gitattributes. At least, that excludes them from an sdist, so if you build a wheel via an sdist they will also be excluded. See for example https://github.com/scipy/scipy/blob/main/.gitattributes

[dnicolodi]

The idea I had was to add an install_package() method to the python_installation object. This method would mirror install_sources() but take a folder as an argument and would recursively install all *.py files in the folder as long as the folder contains a __init__.py file. This is somehow similar to the setuptools packages configuration option. All files that are not python source files would need to be still installed via other mechanisms. The only constraint is that, to follow Meson's philosophy, the list of files to install should be computed at configure time, like install_subdir() does. I could look into implementing this, but improving other aspects of Meson and meson-python has higher priority. After all, keeping a list of the files to be installed is not such a daunting task and, in the worst case, it can be easily automated.

[lithomas1]

The idea I had was to add an install_package() method to the python_installation object. This method would mirror install_sources() but take a folder as an argument and would recursively install all *.py files in the folder as long as the folder contains a __init__.py file. This is somehow similar to the setuptools packages configuration option. All files that are not python source files would need to be still installed via other mechanisms. The only constraint is that, to follow Meson's philosophy, the list of files to install should be computed at configure time, like install_subdir() does. I could look into implementing this, but improving other aspects of Meson and meson-python has higher priority. After all, keeping a list of the files to be installed is not such a daunting task and, in the worst case, it can be easily automated.

I am not too big a fan of adding another install_* method, especially to the python_installation object. I don't think this problem is Python-specific, and for the most part install_subdir does its job well enough.

The problematic files tend to either live together in one folder(e.g. __pycache__, .DS_STORE, data files for tests), or are mostly single, so install_subdir() could just take in a list of files/folders to exclude.

[dnicolodi]

install_subdir() already has the capability to exclude files, see the exclude_files argument. It however does not accept a pattern to match file names and it never will, globbing is against the principles on which Meson is built. Also, Meson is designed to build artifacts without polluting the sources folder, thus it does not make sense for it to have a mechanism to exclude temporary files and other "garbage" from the install.

[lithomas1]

install_subdir() already has the capability to exclude files, see the exclude_files argument. It however does not accept a pattern to match file names and it never will, globbing is against the principles on which Meson is built.

Thanks for the pointer. I'll go check it out.

[dnicolodi]

I don't think there is anything actionable on the meson-python side here. Meson already has the exclude_files and exclude_directories arguments to install_files thus the functionality is there, just there isn't the possibility to use patterns to match the files to exclude. This functionality however needs to be implemented in Meson. Closing for now.