Re: [Distutils] Working toward Linux wheel support

Hi,

On 17 July 2015 at 05:22, Nick Coghlan <ncog...@gmail.com> wrote:

> On 17 July 2015 at 03:41, Nate Coraor <n...@bx.psu.edu> wrote:> > [...]> >> > As mentioned in the wheels PR, there are some questions and decisions> made> > that I need guidance on:> >> > - On Linux, the distro name/version (as determined by> > platform.linux_distribution()) will be appended to the platform string,> e.g.> > linux_x86_64_ubuntu_14_04. This is going to be necessary to make a> > reasonable attempt at wheel compatibility in PyPI. But this may violate> PEP> > 425.>> I think it's going beyond it in a useful way, though. At the moment,> the "linux_x86_64" platform tag *under*specifies the platform - a> binary extension built on Ubuntu 14.04 with default settings may not> work on CentOS 7, for example.>> Adding in the precise distro name and version number changes that to> *over*specification, but I now think we can address that through> configuration settings on the installer side that allow the> specification of "compatible platforms". That way a derived> distribution could add the corresponding upstream distribution's> platform tag and their users would be able to install the relevant> wheel files by default.> [...]


The definition of "acceptable platform tags" should list the platforms in
order of preference (for example, some of the backward compatible past
releases of a linux distro, in reverse order), so that if multiple
acceptable wheels are present the closest one is selected.

As some other have mentioned, this doesn't solve the problem of system
dependencies. I.e.: a perfectly compiled lxml wheel for
linux_x86_64_ubuntu_14_04, installed into Ubuntu 14.04, will still fail to
work if libxml2 and libxslt1.1 debian packages are not installed (among
others).

Worse is that pip will gladly install such package, and the failure will
happen as a potentially cryptic error message payload to an ImportError
that doesn't really make it clear what needs to be done to make the package
actually work.

To solve this problem, so far we've only been able to come up with two
extremes:

 - Have the libraries contain enough metadata in their source form that we
can generate true system packages from them (this doesn't really help the
virtualenv case)
 - Carry all the dependencies. Either by static linking, or by including
all dynamic libraries in the wheel, or by becoming something like Conda
where we package even non Python projects.

As a further step that could be taken on top of Nate's proposed PR, but
avoiding the extremes above, I like Daniel's idea of "specifying the full
library names [...] à-lá RPM". Combine it with the specification of
abstract locations, and we could have wheels declare something like.

 - lxml wheel for linux_x86_64_ubuntu_14_04:
   - extdeps:
      - <dynlibdir>/libc.so.6
      - <dynlibdir>/libm.so.6
      - <dynlibdir>/libxml2.so.2
      - <dynlibdir>/libexslt.so.0

This also makes it possible to have wheels depend on stuff other than
libraries, for example binaries or data files (imagine a lightweight
version of pytz that didn't have to carry its own timezones, and depended
on the host system to keep them updated). As long as we have a proper
abstract location to anchor the files, we can express these dependencies
without hardcoding paths as they were on the build machine.

It even opens the possibility that some of these external dependencies
could be provided on a per-virtualenv basis, instead of globally.

Pip could then (optionally?) check the existence of these external
dependencies before allowing installation of the wheel, increasing the
likelihood that it will work once installed.

This same way of expressing external dependencies could be extended to
source packages themselves. For example the `setup()` (or whatever
successor we end up with) for a PIL source package could express dependency
on '<include>/png.h'.

Or, what's more likely these days, a dependency on
'<bindir>/libpng12-config', which when run prints the correct invocations
of gcc flags to add to the build process.

The build process would then check the presence of these external build
dependencies early on, allowing for much clearer error messages and precise
instructions on how to provide the proper build environment.

Most distros provide handy ways of querying which packages provide which
files, so I believe the specification of external file dependences to be a
nice step up from where we are right now, without wading into
full-system-integration territory.

Leo