| Store | Cart

Re: [Distutils] Working toward Linux wheel support

From: Nick Coghlan <ncog...@gmail.com>
Fri, 17 Jul 2015 18:22:27 +1000
On 17 July 2015 at 03:41, Nate Coraor <n...@bx.psu.edu> wrote:
> Hi all,>> I've recently been working on adding SOABI support for Python 2.x and other> pieces needed to get wheels w/ C extensions for Linux working. Here's the> work for wheels:>>     https://bitbucket.org/pypa/wheel/pull-request/54/>> Based on that, I've added support for those wheels to pip here:>>     https://github.com/natefoo/pip/tree/linux-wheels>> As mentioned in the wheels PR, there are some questions and decisions made> that I need guidance on:>> - On Linux, the distro name/version (as determined by> platform.linux_distribution()) will be appended to the platform string, e.g.> linux_x86_64_ubuntu_14_04. This is going to be necessary to make a> reasonable attempt at wheel compatibility in PyPI. But this may violate PEP> 425.

I think it's going beyond it in a useful way, though. At the moment,
the "linux_x86_64" platform tag *under*specifies the platform - a
binary extension built on Ubuntu 14.04 with default settings may not
work on CentOS 7, for example.

Adding in the precise distro name and version number changes that to
*over*specification, but I now think we can address that through
configuration settings on the installer side that allow the
specification of "compatible platforms". That way a derived
distribution could add the corresponding upstream distribution's
platform tag and their users would be able to install the relevant
wheel files by default.

Rather than putting the Linux specific platform tag derivation logic
directly in the tools, though, what if we claimed a file under the
"/etc/python" subtree and used it to tell the tools what platform tags
to use? For example, we could put the settings relating to package
tags into "/etc/python/binary-compatibility.cfg" and allow that to be
overridden on a per-virtualenv basis with a binary-compatibility.cfg
file within the virtualenv.

For example, we could have a section where for a given platform, we
overrode both the build and install tags appropriately. For RHEL 7.1,
that may look like:

    [linux_x86_64]
    build=rhel_7_1
    install=rhel_7_0,rhel_7_1,centos_7_1406,centos_7_1503

Using JSON rather than an ini-style format would also work:

    {
      "linux_x86_64": {
        "build": "rhel_7_1",
        "install": ["rhel_7_0", "rhel_7_1", "centos_7_1406", "centos_7_1503"]
     }
    }

The reason I like this approach is that it leaves the definition of
ABI compatibility in the hands of the distros, but also makes it safe
to publish Linux wheel files on PyPI (just not with the generic
linux_x86_64 platform tag).

> - By default, wheels will be built using the most specific platform> information. In practice, I build our wheels[1] using Debian Squeeze in> Docker and therefore they should work on most currently "supported" Linuxes,> but allowing such wheels to PyPI could still be dangerous because forward> compatibility is not always guaranteed (e.g. if a SO version/name changes,> or a C lib API method changes in a non-backward compatible way but the SO> version/name does not change). That said, I'd be happy to make a much more> generalized version of our docker-build[2] system that'd allow package> authors to easily/rapidly build distro/version-specific wheels for many of> the popular Linux distros. We can assume that a wheel built on a vanilla> install of e.g. Ubuntu 14.04 will work on any other installation of 14.04> (this is what the distro vendors promise, anyway).

Right, if we break ABI within a release, that's our fault (putting on
my distro developer hat), and folks will rightly yell at us for it. I
was previously wary of this approach due to the "what about derived
distributions?" problem, but realised recently that a config file that
explicitly lists known binary compatible platforms should suffice for
that. There's only a handful of systems folks are likely want to
prebuild wheels for (Debian, Ubuntu, Fedora, CentOS/RHEL, openSuse),
and a configuration file based system allows ABI compatible derived
distros to be handled as if they were their parent.

> - I attempt to set the SOABI if the SOABI config var is unset, this is for> Python 2, but will also be done even on Python 3. Maybe that is the wrong> decision (or maybe SOABI is guaranteed to be set on Python 3).

Python 3 should always set it, but if it's not present for some
reason, deriving it makes sense.

> - Do any other implementations define SOABI? PyPy does not, I did not test> others. What should we do with these?

The implementation identifier is also included in the compatibility
tags, so setting that in addition to the platform ABI tag when a wheel
contains binary extensions should suffice.

> Because the project I work for[3] relies heavily on large number of> packages, some of which have complicated build-time dependencies, we have> always provided them as eggs and monkeypatched platform support back in to> pkg_resources. Now that the PyPA has settled on wheels as the preferred> binary packaging format, I am pretty heavily motivated to do the work to> work out all the issues with this implementation.

Thank you!

Regards,
Nick.

-- 
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia
_______________________________________________
Distutils-SIG maillist  -  Dist...@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Recent Messages in this Thread
Nate Coraor Jul 16, 2015 05:41 pm
Nick Coghlan Sep 06, 2015 11:32 pm
Nick Coghlan Sep 05, 2015 02:56 am
Marcus Smith Sep 07, 2015 04:11 am
Nate Coraor Sep 03, 2015 05:22 pm
Nick Coghlan Sep 07, 2015 04:26 am
Daniel Holth Aug 20, 2015 07:22 pm
Donald Stufft Aug 20, 2015 07:25 pm
Donald Stufft Sep 02, 2015 11:45 pm
Daniel Holth Sep 03, 2015 12:15 pm
Nate Coraor Aug 24, 2015 03:03 pm
Wes Turner Aug 24, 2015 05:51 pm
Nick Coghlan Jul 17, 2015 08:22 am
Chris Barker Jul 17, 2015 03:36 pm
Antoine Pitrou Jul 17, 2015 03:46 pm
Chris Barker Jul 17, 2015 03:53 pm
Andrea Bedini Jul 18, 2015 07:00 am
Tres Seaver Jul 21, 2015 03:25 am
Leonardo Rochael Almeida Jul 21, 2015 03:07 pm
Marcus Smith Jul 17, 2015 04:50 pm
Olivier Grisel Jul 17, 2015 06:34 pm
Daniel Holth Jul 17, 2015 08:18 pm
Chris Barker - NOAA Federal Jul 18, 2015 01:13 am
Daniel Holth Jul 18, 2015 02:11 am
Paul Moore Jul 18, 2015 11:51 am
Nick Coghlan Jul 20, 2015 05:50 am
Chris Barker Jul 20, 2015 05:37 pm
Paul Moore Jul 20, 2015 06:37 pm
Nick Coghlan Jul 27, 2015 02:19 pm
Nate Coraor Jul 27, 2015 07:07 pm
Oscar Benjamin Jul 21, 2015 04:38 pm
Chris Barker Jul 24, 2015 06:52 pm
Oscar Benjamin Jul 28, 2015 03:02 pm
Wes Turner Jul 28, 2015 04:21 pm
Nate Coraor Aug 12, 2015 08:21 pm
Robert Collins Aug 12, 2015 11:49 pm
Nathaniel Smith Aug 13, 2015 01:05 am
Nate Coraor Aug 13, 2015 02:07 pm
Leonardo Rochael Almeida Aug 13, 2015 07:30 pm
Wes Turner Aug 13, 2015 07:43 pm
Nathaniel Smith Aug 14, 2015 01:47 am
Wes Turner Aug 14, 2015 01:50 am
Nathaniel Smith Aug 14, 2015 02:33 am
Wes Turner Aug 14, 2015 02:41 am
Leonardo Rochael Almeida Sep 08, 2015 07:18 pm
Donald Stufft Sep 08, 2015 07:22 pm
Leonardo Rochael Almeida Sep 08, 2015 07:39 pm
Nathaniel Smith Aug 14, 2015 01:25 am
Robert Collins Aug 14, 2015 01:31 am
Wes Turner Aug 14, 2015 01:38 am
Robert Collins Aug 14, 2015 01:44 am
Wes Turner Aug 14, 2015 01:44 am
Nathaniel Smith Aug 14, 2015 02:14 am
Wes Turner Aug 14, 2015 02:24 am
Robert Collins Aug 14, 2015 02:27 am
Nathaniel Smith Aug 14, 2015 07:38 am
David Cournapeau Aug 13, 2015 05:52 pm
Nathaniel Smith Aug 14, 2015 04:07 am
Chris Barker Aug 14, 2015 04:04 pm
David Cournapeau Aug 14, 2015 04:20 pm
Chris Barker Aug 14, 2015 04:00 pm
Leonardo Rochael Almeida Jul 20, 2015 01:42 am
Nick Coghlan Jul 20, 2015 06:00 am
Chris Barker Jul 20, 2015 05:39 pm
Marcus Smith Sep 06, 2015 04:09 pm
Nick Coghlan Sep 05, 2015 08:35 am
Nick Coghlan Sep 05, 2015 06:44 am
Nick Coghlan Sep 05, 2015 06:43 am
Nathaniel Smith Sep 05, 2015 06:46 am
Donald Stufft Sep 05, 2015 03:06 am
Wes Turner Sep 08, 2015 07:14 pm
Daniel Holth Sep 08, 2015 07:32 pm
Nathaniel Smith Sep 09, 2015 11:49 pm
Nate Coraor Sep 21, 2015 03:33 pm
Nate Coraor Sep 09, 2015 03:06 pm
Donald Stufft Sep 08, 2015 06:33 pm
Donald Stufft Sep 07, 2015 04:02 pm
Marcus Smith Sep 07, 2015 05:51 pm
Wes Turner Sep 07, 2015 10:16 pm
Nate Coraor Sep 03, 2015 02:04 pm
Daniel Holth Sep 03, 2015 01:56 pm
Antoine Pitrou Aug 20, 2015 07:51 pm
Nate Coraor Aug 20, 2015 07:40 pm
Donald Stufft Aug 20, 2015 07:19 pm
Antoine Pitrou Aug 20, 2015 07:14 pm
Steve Dower Aug 14, 2015 04:17 pm
Daniel Holth Aug 20, 2015 06:38 pm
Chris Barker Aug 14, 2015 08:16 pm
Alexander Walters Aug 14, 2015 10:32 pm
Nate Coraor Aug 20, 2015 06:26 pm
Nick Coghlan Sep 05, 2015 02:12 am
Daniel Holth Sep 01, 2015 01:57 pm
Wes Turner Aug 26, 2015 01:42 am
Nate Coraor Aug 27, 2015 07:21 pm
Ben Finney Sep 06, 2015 11:42 pm
Messages in this thread