Implement equivalent to `pip.locations.distutils_scheme` in sysconfig

[sylvaincorlay]

BPO	26955
Nosy	@ned-deily, @merwok, @dstufft, @tdsmith

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-05-04.16:17:44.660>
labels = ['type-feature', 'library', '3.10']
title = 'Implement equivalent to `pip.locations.distutils_scheme` in sysconfig'
updated_at = <Date 2020-10-22.21:18:37.358>
user = 'https://bugs.python.org/sylvaincorlay'

bugs.python.org fields:

activity = <Date 2020-10-22.21:18:37.358>
actor = 'eric.araujo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-05-04.16:17:44.660>
creator = 'sylvain.corlay'
dependencies = []
files = []
hgrepos = []
issue_num = 26955
keywords = []
message_count = 7.0
messages = ['264836', '266052', '266059', '266224', '267888', '267956', '379350']
nosy_count = 5.0
nosy_names = ['ned.deily', 'eric.araujo', 'dstufft', 'sylvain.corlay', 'tdsmith']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26955'
versions = ['Python 3.10']

[sylvaincorlay]

When installing a python package that has headers, these headers are usually installed under the main python include directory, which can be retrieved with sysconfig.get_path('include') or directly referred to as 'include' when setting the include directories of an extension module.

However, on some systems like OS X, headers for extension modules are not located in under the python include directory (/usr/local/Cellar/pythonX/X.Y.Z/Frameworks/Python.framework/Versions/X.Y/include/pythonX.Y) but in /usr/local/include/pythonX.Y.

Is there a generic way to find the location where headers are installed in a python install?

pip.locations implements distutils_scheme which seems to be returning the right thing, but it seems to be a bit overkill to require pip. On the other side, no path returned by sysconfig corresponds to /usr/local/include/pythonX.Y

[ned-deily]

"When installing a python package that has headers, these headers are usually installed under the main python include directory". Can you give some specific examples of packages and platforms where that is the case? Just taking a quick look at some packages I happen to have installed, like lxml and cffi using a vanilla pip with a vanilla Unix-y Python 3.5.1, their header files get installed in the site-packages directory by default (using pip), not the main Python include directory. And various distributors of Python have their own preferences where to install things, for example, the locations you cite are those used by Homebrew on OS X and differ from other OS X Pythons. I'm not sure that there is a general solution as things stand now; I think there is probably a lot of variability among packages (e.g. their setup.py) and Python distributions.

[sylvaincorlay]

Ned, that is because these packages (lxml, cffi) have header files as package_data, instead of headers. This is why they are being copied into site-packages. People willing to include them must either rely on knowledge of their location, or a python functions returning if (like numpy's get_include)

Distutils provides a standard way of distributing headers. These are passed to the setup function via the headers keyword argument, which triggers the install_headers command.

[ned-deily]

Sylvain, OK. I suggest you bring the matter up on the distutils-sig mailing list (if you haven't already) because that's where discussions and decisions are made about features for Python packaging including distutils. Otherwise, the issue is likely to languish here. Good luck!

https://mail.python.org/mailman/listinfo/distutils-sig

[tdsmith]

As a Homebrew maintainer I'm happy to consider improving Homebrew's configuration if someone can point me to an extant package that uses this mechanism.

[sylvaincorlay]

We use it in the pybind11 project. I am not sure that homebrew does anything wrong.

It is just that it is the only case I am aware of where the install_headers command does not install headers in a subdirectory of sysconfig.get_path('include'). On the other hand, pip.locations.distutil_scheme provides the location used by install_headers.

[merwok]

If added, this should be in sysconfig, not distutils.