| Store | Cart

Re: [Distutils] Working toward Linux wheel support

From: Daniel Holth <dho...@gmail.com>
Thu, 20 Aug 2015 18:38:49 +0000
We've also considered using truncated SHA-256 hashes. Then the tag would
not be readable, but it would always be the same length.

On Thu, Aug 20, 2015 at 2:27 PM Nate Coraor <n...@bx.psu.edu> wrote:

> On Fri, Aug 14, 2015 at 3:38 AM, Nathaniel Smith <n...@pobox.com> wrote:>>> On Thu, Aug 13, 2015 at 7:27 PM, Robert Collins>> <robe...@robertcollins.net> wrote:>> > On 14 August 2015 at 14:14, Nathaniel Smith <n...@pobox.com> wrote:>> > ...>>> >> Of course if you have an alternative proposal than I'm all ears :-).>> >>> > Yeah :)>> >>> > So, I want to dedicate some time to contributing to this discussion>> > meaningfully, but I can't for the next few weeks - Jury duty, Kiwi>> > PyCon and polishing up the PEP's I'm already committed to...>>>> Totally hear that... it's not super urgent anyway. We should make it>> clear to Nate -- hi Nate! -- that there's no reason that solving this>> problem should block putting together the basic>> binary-compatibility.cfg infrastructure.>>>> Hi!>> I've been working on bits of this as I've also been working on, as a test> case, building out psycopg2 wheels for lots of different popular distros on> i386 and x86_64, UCS2 and UCS4, under Docker. As a result, it's clear that> my Linux distro tagging work in wheel's pep425tags has some issues. I've> been adding to this list of distributions but it's going to need a lot more> work:>>> https://bitbucket.org/pypa/wheel/pull-requests/54/soabi-2x-platform-os-distro-support-for/diff#Lwheel/pep425tags.pyT61>> So I need a bit of guidance here. I've arbitrarily chosen some tags -> `rhel` for example - and wonder if, like PEP 425's mapping of Python> implementations to tags, a defined mapping of Linux distributions to> shorthand tags is necessary (of course this would be difficult to keep up> to date, but binary-compatibility.cfg would make it less relevant in the> long run).>> Alternatively, I could simply trust and normalize> platform.linux_distribution()[0], but this means that the platform tag on> RHEL would be something like> `linux_x86_64_red_hat_enterprise_linux_server_6_5`>> Finally, by *default*, the built platform tag will include whatever> version information is provided in platform.linux_distribution()[1], but> the "major-only" version is also included in the list of platforms, so a> default debian tag might look like `linux_x86_64_debian_7_8`, but it would> be possible to build (and install) `linux_x86_64_debian_7`. However, it may> be the case that the default (at least for building, maybe not for> installing) ought to be the major-only tag since it should really be ABI> compatible with any minor release of that distro.>> --nate>>>>> > I think the approach of being able to ask the *platform* for things>> > needed to build-or-use known artifacts is going to enable a bunch of>> > different answers in this space. I'm much more enthusiastic about that>> > than doing anything that ends up putting PyPI in competition with the>> > distribution space.>> >>> > My criteria for success are:>> >>> > - there's *a* migration path from what we have today to what we>> > propose. Doesn't have to be good, just exist.>> >>> >  - authors of scipy, numpy, cryptography etc can upload binary wheels>> > for *linux, Mac OSX and Windows 32/64 in a safe and sane way>>>> So the problem is that, IMO, "sane" here means "not building a>> separate wheel for every version of distro on distrowatch". So I can>> see two ways to do that:>> - my suggestion that we just pick a particular highly-compatible>> distro like centos 5 to build against, and make a standard list of>> which libraries can be assumed to be provided>> - the PEP-497-or-number-to-be-determined approach, in which we still>> have to pick a highly-compatible distro like centos 5 to build>> against, but each wheel has a list of which libraries from that distro>> it is counting on being provided>>>> I can see the appeal of the latter approach, since if you want to do>> the former approach right you need to be careful about exactly which>> libraries you're assuming are present, etc. They both could work. But>> in practice, you still have to pick which distro you are going to use>> to build, and you still have to say "when I say I need libblas.so.1,>> what I mean is that I need a file that is ABI-compatible with the>> version of libblas.so.1 that existed in centos 5 exactly, not any>> other libblas.so.1". And then in practice not every distro will have>> such a thing, so for a project like numpy that wants to make things>> easy for a wide variety of users, we'll still only be able to take>> advantage of external dependencies for libraries that are effectively>> universally available and compatible anyway and end up vendoring the>> rest... so in the end basically we'd be distributing exactly the same>> wheels under either of these proposals, just the latter requires a>> much much more complicated scheme for metadata and installation.>>>> And in practice I think the main alternative possibility if we don't>> come up with some solid guidance for how packages can build>> works-everywhere-wheels is that we'll see wheels for>> latest-version-of-Ubuntu-only, plus the occasional smattering of other>> distros, varying randomly on a project-by-project basis. Which would>> suck.>>>> >  - we don't need to do things like uploading wheels containing>> > non-Python shared libraries, nor upload statically linked modules>> >>> >>> > In fact, I think uploading regular .so files is just a huge heartache>> > waiting to happen, so I'm almost inclined to add:>> >>> >  -  we don't support uploading external non-Python libraries [ without>> > prejuidice for changing our minds in the future]>>>> Windows and OS X don't (reliably) have any package manager. So PyPI>> *is* inevitably going to contain non-Python shared libraries or>> statically linked modules or something like that. (And in fact it>> already contains such things today.) I'm not sure what the alternative>> would even be.>>>> This also means that projects like numpy are already forced to accept>> that we're on the hook for security updates in our dependencies etc.,>> so doing it on Linux too is not really that scary.>>>> Oh, I just thought of another issue: an extremely important>> requirement for numpy/scipy/etc. wheels is that they be reliably>> installable without root access. People *really* care about this:>> missing your grant deadline b/c you can't upgrade some package to fix>> some showstopper bug b/c university IT support is not answering calls>> at midnight on Sunday = rather poor UX.>>>> Given that, the only situation I can see where we would ever>> distribute wheels that require system blas on Linux, is if we were>> able to do it alongside wheels that do not require system blas, and>> pip were clever enough to reliably always pick the latter except in>> cases where the system blas was actually present and working.>>>> > There was a post that referenced a numpy ABI, dunno if it was in this>> > thread - I need to drill down into that, because I don't understand>> > why thats not a regular version resolution problem,unlike the Python>> > ABI, which pip can't install [and shouldn't be able to!]>>>> The problem is that numpy is very unusual among Python packages in>> that exposes a large and widely-used *C* API/ABI:>>>>     http://docs.scipy.org/doc/numpy/reference/c-api.html>>>> This means that when you build, e.g., scipy, then you get a binary>> that depends on things like the in-memory layout of numpy's internal>> objects. We'd like it to be the case that when we release a new>> version of numpy, pip could realize "hey, this new version says it has>> an incompatible ABI that will break your currently installed version>> of scipy -- I'd better fetch a new version of scipy as well, or at>> least rebuild the same version I already have". Notice that at the>> time scipy is built, it is not known which future version of numpy>> will require a rebuild. There are a lot of ways this might work on>> both the numpy and pip sides -- definitely fodder for a separate>> thread -- but that's the basic problem.>>>> -n>>>> -->> Nathaniel J. Smith -- http://vorpus.org>>> _______________________________________________> Distutils-SIG maillist  -  Dist...@python.org> https://mail.python.org/mailman/listinfo/distutils-sig>

Distutils-SIG maillist  -  Dist...@python.org

Recent Messages in this Thread
Nate Coraor Jul 16, 2015 05:41 pm
Nick Coghlan Sep 06, 2015 11:32 pm
Nick Coghlan Sep 05, 2015 02:56 am
Marcus Smith Sep 07, 2015 04:11 am
Nate Coraor Sep 03, 2015 05:22 pm
Nick Coghlan Sep 07, 2015 04:26 am
Daniel Holth Aug 20, 2015 07:22 pm
Donald Stufft Aug 20, 2015 07:25 pm
Donald Stufft Sep 02, 2015 11:45 pm
Daniel Holth Sep 03, 2015 12:15 pm
Nate Coraor Aug 24, 2015 03:03 pm
Wes Turner Aug 24, 2015 05:51 pm
Nick Coghlan Jul 17, 2015 08:22 am
Chris Barker Jul 17, 2015 03:36 pm
Antoine Pitrou Jul 17, 2015 03:46 pm
Chris Barker Jul 17, 2015 03:53 pm
Andrea Bedini Jul 18, 2015 07:00 am
Tres Seaver Jul 21, 2015 03:25 am
Leonardo Rochael Almeida Jul 21, 2015 03:07 pm
Marcus Smith Jul 17, 2015 04:50 pm
Olivier Grisel Jul 17, 2015 06:34 pm
Daniel Holth Jul 17, 2015 08:18 pm
Chris Barker - NOAA Federal Jul 18, 2015 01:13 am
Daniel Holth Jul 18, 2015 02:11 am
Paul Moore Jul 18, 2015 11:51 am
Nick Coghlan Jul 20, 2015 05:50 am
Chris Barker Jul 20, 2015 05:37 pm
Paul Moore Jul 20, 2015 06:37 pm
Nick Coghlan Jul 27, 2015 02:19 pm
Nate Coraor Jul 27, 2015 07:07 pm
Oscar Benjamin Jul 21, 2015 04:38 pm
Chris Barker Jul 24, 2015 06:52 pm
Oscar Benjamin Jul 28, 2015 03:02 pm
Wes Turner Jul 28, 2015 04:21 pm
Nate Coraor Aug 12, 2015 08:21 pm
Robert Collins Aug 12, 2015 11:49 pm
Nathaniel Smith Aug 13, 2015 01:05 am
Nate Coraor Aug 13, 2015 02:07 pm
Leonardo Rochael Almeida Aug 13, 2015 07:30 pm
Wes Turner Aug 13, 2015 07:43 pm
Nathaniel Smith Aug 14, 2015 01:47 am
Wes Turner Aug 14, 2015 01:50 am
Nathaniel Smith Aug 14, 2015 02:33 am
Wes Turner Aug 14, 2015 02:41 am
Leonardo Rochael Almeida Sep 08, 2015 07:18 pm
Donald Stufft Sep 08, 2015 07:22 pm
Leonardo Rochael Almeida Sep 08, 2015 07:39 pm
Nathaniel Smith Aug 14, 2015 01:25 am
Robert Collins Aug 14, 2015 01:31 am
Wes Turner Aug 14, 2015 01:38 am
Robert Collins Aug 14, 2015 01:44 am
Wes Turner Aug 14, 2015 01:44 am
Nathaniel Smith Aug 14, 2015 02:14 am
Wes Turner Aug 14, 2015 02:24 am
Robert Collins Aug 14, 2015 02:27 am
Nathaniel Smith Aug 14, 2015 07:38 am
David Cournapeau Aug 13, 2015 05:52 pm
Nathaniel Smith Aug 14, 2015 04:07 am
Chris Barker Aug 14, 2015 04:04 pm
David Cournapeau Aug 14, 2015 04:20 pm
Chris Barker Aug 14, 2015 04:00 pm
Leonardo Rochael Almeida Jul 20, 2015 01:42 am
Nick Coghlan Jul 20, 2015 06:00 am
Chris Barker Jul 20, 2015 05:39 pm
Marcus Smith Sep 06, 2015 04:09 pm
Nick Coghlan Sep 05, 2015 08:35 am
Nick Coghlan Sep 05, 2015 06:44 am
Nick Coghlan Sep 05, 2015 06:43 am
Nathaniel Smith Sep 05, 2015 06:46 am
Donald Stufft Sep 05, 2015 03:06 am
Wes Turner Sep 08, 2015 07:14 pm
Daniel Holth Sep 08, 2015 07:32 pm
Nathaniel Smith Sep 09, 2015 11:49 pm
Nate Coraor Sep 21, 2015 03:33 pm
Nate Coraor Sep 09, 2015 03:06 pm
Donald Stufft Sep 08, 2015 06:33 pm
Donald Stufft Sep 07, 2015 04:02 pm
Marcus Smith Sep 07, 2015 05:51 pm
Wes Turner Sep 07, 2015 10:16 pm
Nate Coraor Sep 03, 2015 02:04 pm
Daniel Holth Sep 03, 2015 01:56 pm
Antoine Pitrou Aug 20, 2015 07:51 pm
Nate Coraor Aug 20, 2015 07:40 pm
Donald Stufft Aug 20, 2015 07:19 pm
Antoine Pitrou Aug 20, 2015 07:14 pm
Steve Dower Aug 14, 2015 04:17 pm
Daniel Holth Aug 20, 2015 06:38 pm
Chris Barker Aug 14, 2015 08:16 pm
Alexander Walters Aug 14, 2015 10:32 pm
Nate Coraor Aug 20, 2015 06:26 pm
Nick Coghlan Sep 05, 2015 02:12 am
Daniel Holth Sep 01, 2015 01:57 pm
Wes Turner Aug 26, 2015 01:42 am
Nate Coraor Aug 27, 2015 07:21 pm
Ben Finney Sep 06, 2015 11:42 pm
Messages in this thread