A more unixy layout for cpython on windows

[isuruf]

I'm proposing some layout changes for the upcoming 3.15 release

    'nt': {
        'stdlib': '{installed_base}/Lib',
        'platstdlib': '{base}/Lib',
        'purelib': '{base}/Lib/site-packages',
        'platlib': '{base}/Lib/site-packages',
        'include': '{installed_base}/Include',
        'platinclude': '{installed_base}/Include',
        'scripts': '{base}/Scripts',
        'data': '{base}',
        },

by adding the new scheme

    'nt_conda': {
        'stdlib': '{installed_base}/lib/{implementation_lower}{py_version_short}{abi_thread}',
        'platstdlib': '{base}/lib/{implementation_lower}{py_version_short}{abi_thread}',
        'purelib': '{base}/lib/{implementation_lower}/site-packages',
        'platlib': '{base}/lib/{implementation_lower}/site-packages',
        'include': '{installed_base}/include/{implementation_lower}',
        'platinclude': '{installed_base}/include/{implementation_lower}',
        'scripts': '{base}/bin',  # maybe keep it Scripts for now and then add support to conda analogous to `python_site_packages_path`
        'data': '{base}',
        },

and then making nt_conda the default layout. This is what debian does to have dist-packages and site-packages split.

[isuruf]

cc @mgorny @jaimergp @zooba

[zooba]

FYI, CPython think platlibdir on Windows is DLLs, so the proposed scheme would be a significant change from what it currently is.

Other than that, I assume the benefit here is either getting to spend a lot of time porting existing code new code doesn't have to concern itself with layout as much, or it becomes easier to co-install different runtimes (apart from python[w].exe being the sole entrypoint). In the latter case, there's definitely more work and changes you'll need to make, and I'm not sure the former is worth Conda doing unilaterally, but perhaps you've seen more issues here than I have?

[mgorny]

Won't this break compatibility with pip and similar tools that can currently treat conda environments as venvs?

[isuruf]

FYI, CPython think platlibdir on Windows is DLLs, so the proposed scheme would be a significant change from what it currently is.

Thanks. I've removed platlibdir to lib.

The benefit is to lower the special casing of the python layout in conda environments. For eg: $PREFIX/include has the python internal headers there, so we can't put headers of other packages there. Therefore we have a special $PREFIX/Library which serves as a prefix for every other package.

Won't this break compatibility with pip and similar tools that can currently treat conda environments as venvs?

I don't think so. Do you have some particular example I can check out?

[mgorny]

I don't think so. Do you have some particular example I can check out?

I'm thinking of creating a conda environment, installing some packages via conda, then installing more packages via pip install ... and uv pip install ....

[isuruf]

pip takes the install locations from sysconfig, so if we fix that, then it should be fine.

[jaimergp]

This will probably require some updates in places that assume (and hardcode) the presence of Python in certain places. For example, in conda:

• https://github.com/conda/conda/blob/c35ac19fec86eaeee718bac41696301b8fcd84be/conda/common/path/__init__.py#L151
• https://github.com/conda/conda/blob/c35ac19fec86eaeee718bac41696301b8fcd84be/conda/common/path/python.py#L60-L67
• https://github.com/conda/conda/blob/c35ac19fec86eaeee718bac41696301b8fcd84be/conda/common/path/python.py#L52-L57

Not that this should deter us from homogenising it, but just saying that downstream will need a generous heads up to accommodate for this change.

[jaimergp]

I assume Pixi and Mamba will also need to handle this, pinging @wolfv and @SylvainCorlay for awareness.

[isuruf]

1. In original post I mentioned we can keep the old way for scripts for backwards compat.
2. conda, pixi, mamba support python_site_packages_path, so that code path is not used.
3. Not changed. So not a concern.

[jaimergp]

This was brought up in the Apr 29th community call. Conclusion was to draft a CFEP-like document where we formalize the decisions and motivations behind this initiative, even if it's just to leave it here in the issue description. This can be done in parallel with the exploratory work in the PR itself.