Allow site packages to install header files

[FFY00]

BPO	44445
Nosy	@jaraco, @tarekziade, @merwok, @encukou, @stefanor, @mitya57, @zooba, @dstufft, @pablogsal, @FFY00
PRs	Clarify that distutils.(plat)include is for CPython's headers #29578

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-06-17.15:03:28.410>
labels = ['3.10']
title = 'Add `site-include` install scheme path in sysconfig'
updated_at = <Date 2022-03-21.11:45:31.504>
user = 'https://github.com/FFY00'

bugs.python.org fields:

activity = <Date 2022-03-21.11:45:31.504>
actor = 'mitya57'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-06-17.15:03:28.410>
creator = 'FFY00'
dependencies = []
files = []
hgrepos = []
issue_num = 44445
keywords = ['patch']
message_count = 3.0
messages = ['396004', '396012', '399362']
nosy_count = 10.0
nosy_names = ['jaraco', 'tarek', 'eric.araujo', 'petr.viktorin', 'stefanor', 'mitya57', 'steve.dower', 'dstufft', 'pablogsal', 'FFY00']
pr_nums = ['29578']
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue44445'
versions = ['Python 3.10']

[FFY00]

During the distutils deprecation, we have identified that there is no good replacement for the distutils headers install path. This path defines where packages are supposed to install headers on the system.

The setuptools equivalent would be include/platinclude, but these paths depend on installed_base/installed_platbase, which makes them the same on virtual environments. For this reason, they are not very suitable as a headers replacement -- if a package installs to include on a virtual environment, those files will be available everywhere.

{
    ...
    'include': '{installed_base}/include/python{py_version_short}{abiflags}',
    'platinclude': '{installed_platbase}/include/python{py_version_short}{abiflags}',
}

I propose introducing two new paths, site-include and site-platinclude, as follows:

{
    ...
    'include': '{installed_base}/include/python{py_version_short}{abiflags}',
    'platinclude': '{installed_platbase}/include/python{py_version_short}{abiflags}',
    'site-include': '{base}/include/python{py_version_short}{abiflags}-site',
    'site-platinclude': '{platbase}/include/python{py_version_short}{abiflags}-site',
}

This would make them different paths on virtual environments and would allow us to install header files there instead of include/platinclude.

Does anyone have a better idea? Or is there perhaps something I have missed?

---

Hopefully, this could go into Python 3.10, so that users can easily replace distutils usages with sysconfig. I understand if that may be unlikely.

---

Relevant links:
https://discuss.python.org/t/clarification-on-a-wheels-header-data/9305
pypa/pip#9617

[zooba]

distutils.sysconfig doesn't expose the headers path either, it's only there as a default value for the install command (Lib/distutils/command/install.py).

So it doesn't seem unreasonable to provide a recommendation on where to put shared header files and let installers do their own calculation. If an installer wants to install into _another_ environment, it can't rely on sysconfig anyway, so we need the spec as well as any implementation. And if we don't have the spec then people will have to reverse-engineer the implementation anyway, so we may as well start with the spec.

Now, whether we actually need or want packages dumping all their headers in one directory is a different question. At least on some platforms they'll also need to import libraries too, and tools like Cython have different files yet again. Many existing project keep these files inside their package and offer the path on request (e.g. pybind11), so perhaps we actually want to standardise pkg-config-like metadata instead?

(Also crossposting back to the discuss thread)

[FFY00]

distutils will install the headers to {base}/include/python{py_version_short}{abiflags}/{dist_name}, and I think probably the best option would be to do the same. So, I think we could add a site-include scheme like I described and have the installers append a directory with the distribution name to it.

I think it would be reasonable to put all headers in a shared directory, site-include, given that we make sure the installers put them in a subdirectory for each distribution.

About standardizing pkg-config-like metadata, I don't think that's a great idea, as it would not replace the mechanisms you mentioned. It might replace some, but not all, as there could always be a use-case that needs more information or interaction than what our standard may provide.

Does this make sense to you?

[FFY00]

@zooba friendly ping

[zooba]

We're discussing standardising something that CPython does not use (headers path for installed packages). This isn't the place for it - it should be in a distutils-sig post on Discourse, where the people who maintain package builders and extension module compilers can participate.

[FFY00]

I opened https://discuss.python.org/t/deprecating-the-headers-wheel-data-key/23712, which would remove the need for this.