| Store | Cart

Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

From: Nick Coghlan <ncog...@gmail.com>
Sun, 30 Nov 2014 01:47:16 +1000
On 29 November 2014 at 01:51, Antoine Pitrou <soli...@pitrou.net> wrote:
> On Sat, 29 Nov 2014 01:27:44 +1000> Nick Coghlan <ncog...@gmail.com> wrote:>> >>> > Is this not going to be a slippery slope?>>>> Only if folks publish Linux binaries themselves, and that's still a>> bad idea (for the same reason publishing distro binaries is already a>> rare thing for people to do).>> Well, let's not make this a matter of ideology. Everyone knows it's a> bad idea to publish binaries, yet it's often better than nothing,> especially if the software is tedious to compile.

It's not a matter of ideology, but a matter of practicality. Debian
stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow
enough (and have strong enough ABI compatibility guarantees) to be
practical for ISVs to target with prebuilt binaries. Beyond that, the
rate of development and breadth of target environments in the Linux
world tends to make providing prebuilt binaries in *any* format
difficult (unless you statically link everything you need, which works
fine until you decide you want a particular package to depend on
binaries provided by another one by linking to them dynamically).

Regardless of target environment, though, being able to prebuild wheel
files in a way that clearly indicates the platform restricted nature
of the end result is a useful tool for system integrators. That
benefit applies regardless of whether the builds are happening in the
context of Linux distro development, or in the context of maintaining
a particular set of infrastructure services.

>> > How many binary packages will package authors have to provide to cover>> > people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied>> > by three or four Python versions is already a lot of binaries to>> > build...>>>> I'd still advise against folks posting Linux wheels on PyPI, just as>> they tend not to post RPM or deb files. This is so we can provide>> wheels at the distro level (or build them internally) without creating>> vast amounts of confusion.>> So do we (software authors) have to wait for that mythical "we" who are> going to build binaries in time for our packages?

If the compatibility tagging issue can be resolved, I don't think
there should be any restrictions against uploading Linux wheels that
avoid the generic Linux platform tags. I just doubt it will make sense
for most developers to worry about it, just as most of them don't
worry about providing RPMs or Debian packages.

> Case in point: can I ask you (the mythical "we") to build packages for> all major distros (including supported LTS releases), and the four most> recent Python versions, of the following piece of software:> https://github.com/numba/llvmlite

No, that would be a service provided by the as yet hypothetical PyPI
build farm. If/when that happens, it will need to have a way of
tagging Linux wheels appropriately, though.

Nearer term (and what prompted me to start this thread), the Fedora
Environments & Stacks working group is investigating providing
prebuilt wheel files for the Fedora ecosystem, and potentially for
EPEL as well (see
https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement
for the broader context of that effort). For other ecosystems, you'll
have to ask participants in those ecosystems.

>> This isn't really about that - it's about having a way to>> tackle it at the distro level, without introducing significant>> potential for confusion on end user systems>> I'm not sure I understand: distros provide their own packages, they> don't (shouldn't) blindly pull binary wheels from PyPI. Why would they> depend on the wheel tagging format?

We don't plan to blindly pull anything from PyPI - we're looking at
the feasibility of publishing already reviewed software in ecosystem
native formats (with the two pilot projects focusing on Java JAR files
and Python wheel files).

When I last mentioned that idea here, Marcus pointed out that doing
that with the generic "linux_x86_64" compatibility tag on the wheel
filenames would be problematic, as there'd be nothing preventing
anyone from pulling them down onto inappropriate systems, with no
obvious trace of the Fedora or EPEL specific assumptions that went
into building them.

While that's a valid concern, I also don't want to go invent our own
custom compatibility tagging convention just for Fedora & EPEL, but
rather work within the limits of what upstream Python packaging
natively supports. At the moment there is no such tagging system,
which is why I'm interested in pursuing improvements to the definition
of the platform tag.

However, after thinking further about the situation with EPEL (where
we'd likely want a single set of wheel files to cover not only an
entire major release series, but also downstream RHEL derivatives like
CentOS), the possibility of sharing wheel files between the Fedora &
Software Collections Python builds, and the point Matthias raised
about the limitations of the current platform tag when it comes to
multiarch support on Debian, I'm back to considering the idea of
making it possible to override the default platform tag with something
more appropriate (I think Daniel may have been the first one to
suggest that?). The task of defining what the appropriate platform tag
overrides should be would then fall back on the Linux distros as part
of declaring our ABI compatibility expectations.

Slightly tangentially, I'm also now wondering if we might also be able
to address the SOABI problem on Python 2 with an appropriate
configuration system design.

Consider if the following could be included in the "pyvenv.cfg" file
in a virtual environment:

[compatibility]
python=cp27,cp2,py2
abi=cp27mu
platform=linux_x86_64_epel_6

Or for a Python 3 virtual environment:

[compatibility]
python=cp34,cp3,py3
abi=cp34m,abi3
platform=linux_x86_64_epel_6

If present, these pyvenv.cfg settings would override the normal PEP
425 compatibility tag calculations (I'm OK with the idea of *needing*
to be in a virtual environment to gain the power to configure these
tags). Note that this would *only* affect the tags used in filenames,
and when searching for wheel files. It would still be up

I'm not sold on the technical details yet (consider the above to be
just a sketch of the idea), but it seems to me that having those
virtual environment specific overrides defined in a tool independent
way, and updating bdist_wheel to read them when creating wheel files,
and pip to read them when deciding which wheel files to consider as
candidates for installation, would likely address the use case,
without locking us in to anything too specific in the absence of
suitable data.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia
_______________________________________________
Distutils-SIG maillist  -  Dist...@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Recent Messages in this Thread
Nick Coghlan Nov 28, 2014 06:03 am
Antoine Pitrou Nov 28, 2014 08:19 am
Nick Coghlan Nov 28, 2014 03:27 pm
Antoine Pitrou Nov 28, 2014 03:51 pm
Nick Coghlan Nov 29, 2014 03:47 pm
Antoine Pitrou Nov 29, 2014 04:10 pm
Nick Coghlan Nov 29, 2014 05:02 pm
Donald Stufft Nov 29, 2014 05:40 pm
Antoine Pitrou Nov 29, 2014 05:41 pm
Tres Seaver Nov 29, 2014 05:39 pm
Marius Gedminas Nov 28, 2014 08:40 am
Matthias Klose Nov 28, 2014 03:31 pm
Nick Coghlan Nov 28, 2014 03:38 pm
Marcus Smith Dec 01, 2014 01:41 am
Olivier Grisel Dec 02, 2014 10:57 am
Nick Coghlan Dec 02, 2014 11:56 am
Messages in this thread