| Store | Cart

Re: [Distutils] Working toward Linux wheel support

From: Nathaniel Smith <n...@pobox.com>
Wed, 9 Sep 2015 16:49:31 -0700
On Wed, Sep 9, 2015 at 8:06 AM, Nate Coraor <n...@bx.psu.edu> wrote:
> On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith <n...@pobox.com> wrote:>>>> On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft <don...@stufft.io> wrote:>> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) wrote:>> >> >>>>> >> >>> I'll create PRs for this against wheel and pip shortly. I can also>> >> >>> work>> >> >>> on a PEP for the platform tag - I don't think it's going to need to>> >> >>> be a>> >> >>> big one. Are there any preferences as to whether this should be a>> >> >>> new PEP>> >> >>> or an update to 425?>> >> >>>>> >>> > Coming back to this, I'm wondering if we should include the libc>> > implementation/version in a less generic, but still generic linux wheel.>> > Right>> > now if you staticly link I think the only platform ABIs you need to>> > worry about>> > are libc and Python itself. Python itself is handled already but libc is>> > not.>> > The only thing I've seen so far is "build on an old enough version of>> > glibc>> > that it handles anything sane", but not all versions of Linux even use>> > glibc at>> > all.>>>> This feels kinda half-baked to me?>>>> "linux" is a useful tag because it has a clear meaning: "there exists>> a linux system somewhere that can run this, but no guarantees about>> which one, good luck". When building a wheel it's easy to tell whether>> this tag can be correctly applied.>>> I'm not sure how it'd be possible to tell. The same meaning for a generic> tag would be true of any wheel built, regardless of whether the wheel has> dependencies in addition to libc.

Sure... my point is just that "linux" is unambiguous and fills a
niche: it unambiguously says "you're on your own", and sometimes
that's the best we can hope to say.

>> Distro-specific tags are useful because they also have a fairly clear>> meaning: "here's a specific class of systems that can run this, so>> long as you install enough packages to fulfill the external>> dependencies". Again, when building a wheel it's pretty easy to tell>> whether this tag can be correctly applied. (Of course someone could>> screw this up, e.g. by building on a system is technically distro X>> but has some incompatible hand-compiled libraries installed, but 99%>> of the time we can guess correctly.)>>>> If we define a LSB-style base system and give it a tag, like I don't>> know, the "Python base environment", call it "linux_pybe1_core" or>> something, that describes what libraries are and aren't available and>> their ABIs, and provide docs/tooling to help people explicitly create>> such wheels and check whether they're compatible with their system,>> then this is also useful -- we have proof that this is sufficient to>> actually distribute arbitrary software usefully, given that multiple>> distributors have converged on this strategy. (I've been talking to>> some people off-list about maybe actually putting together a proposal>> like this...)>>>> To me "linux_glibc_2.18" falls between the cracks though. If this>> starts being what you get by default when you build a wheel, then>> people will use it for wheels that are *not* statically linked, and>> what that tag will mean is "there exists some system that can run>> this, and that has glibc 2.18 on it, and also some other unspecified>> stuff, good luck". Which is pretty useless -- we might as well just>> stick with "linux" in this case. OTOH if it's something that builders>> have to opt into, then we could document that it's only to be used for>> wheels that are statically linked except for glibc, and make it mean>> "*any* system which has glibc 2.18 or later on it can run this". Which>> would be useful in some cases.>>> This is a tooling issue. If wheel (the package) inspects the built .so files> and finds they are not dynamically linked to anything not included with> glibc, it can apply the glibc tag. Otherwise, it'd apply the distro tag.> There's no possibility for human error here, unless they explicitly override> the platform tag.>>>>> But at this point it's basically a>> version of the "defined base environment" approach, and once you've>> gone that far you might as well take advantage of the various>> distributors' experience about what should actually be in that>> environment -- glibc isn't enough.>> While I agree that glibc isn't always enough, defining a base environment> that may not be met by the "standard" install of popular distributions makes> unprivileged wheel installation much more difficult.

Yeah, which is why my suggestion is that we steal the "base
environment" definition from the folks like Continuum and Enthought
who have already done the work of determining what is in the
"standard" install of popular distributions, and have spent years
actually distributing packages to unprivileged users :-).

> It's also not going to> work out of the box on older distributions that wouldn't provide whatever> standardized mechanism is defined for a list of "base environments currently> provided by this system" (unless pip does the work itself at runtime to> determine whether a base environment is met).

Right -- which is basically what pip will have to do to figure out the
current glibc version too, right?

Trying to guess whether the installed versions of several libraries
are really ABI compatible with what we expect is harder than trying to
guess whether the installed version of glibc alone is really ABI
compatible with what we expect, but in both cases it's basically a
heuristic (some distros could have local patches to their glibc that
break ABI, who knows) and in both cases it's basically safe to just
assume it will work (because if we stick to libraries that other
distributors are already depending on then we have years of experience
that it pretty much always works).

> Maybe an important question:> how many popular packages with C Extensions have dependencies in addition to> glibc?

Certainly enough that the major distributors of binary packages on
Linux, like Continuum and Enthought, have decided that they need to
require more than glibc :-).

libstdc++ is an example of one particularly common external dependency.

To be clear: if you're talking specifically about the model where we
validate that the extensions are statically linked before we enable
the glibc tag, then I don't think it will do any harm to have it as an
option. It just seems redundant with the more general solution.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
Distutils-SIG maillist  -  Dist...@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Recent Messages in this Thread
Nate Coraor Jul 16, 2015 05:41 pm
Nick Coghlan Sep 06, 2015 11:32 pm
Nick Coghlan Sep 05, 2015 02:56 am
Marcus Smith Sep 07, 2015 04:11 am
Nate Coraor Sep 03, 2015 05:22 pm
Nick Coghlan Sep 07, 2015 04:26 am
Daniel Holth Aug 20, 2015 07:22 pm
Donald Stufft Aug 20, 2015 07:25 pm
Donald Stufft Sep 02, 2015 11:45 pm
Daniel Holth Sep 03, 2015 12:15 pm
Nate Coraor Aug 24, 2015 03:03 pm
Wes Turner Aug 24, 2015 05:51 pm
Nick Coghlan Jul 17, 2015 08:22 am
Chris Barker Jul 17, 2015 03:36 pm
Antoine Pitrou Jul 17, 2015 03:46 pm
Chris Barker Jul 17, 2015 03:53 pm
Andrea Bedini Jul 18, 2015 07:00 am
Tres Seaver Jul 21, 2015 03:25 am
Leonardo Rochael Almeida Jul 21, 2015 03:07 pm
Marcus Smith Jul 17, 2015 04:50 pm
Olivier Grisel Jul 17, 2015 06:34 pm
Daniel Holth Jul 17, 2015 08:18 pm
Chris Barker - NOAA Federal Jul 18, 2015 01:13 am
Daniel Holth Jul 18, 2015 02:11 am
Paul Moore Jul 18, 2015 11:51 am
Nick Coghlan Jul 20, 2015 05:50 am
Chris Barker Jul 20, 2015 05:37 pm
Paul Moore Jul 20, 2015 06:37 pm
Nick Coghlan Jul 27, 2015 02:19 pm
Nate Coraor Jul 27, 2015 07:07 pm
Oscar Benjamin Jul 21, 2015 04:38 pm
Chris Barker Jul 24, 2015 06:52 pm
Oscar Benjamin Jul 28, 2015 03:02 pm
Wes Turner Jul 28, 2015 04:21 pm
Nate Coraor Aug 12, 2015 08:21 pm
Robert Collins Aug 12, 2015 11:49 pm
Nathaniel Smith Aug 13, 2015 01:05 am
Nate Coraor Aug 13, 2015 02:07 pm
Leonardo Rochael Almeida Aug 13, 2015 07:30 pm
Wes Turner Aug 13, 2015 07:43 pm
Nathaniel Smith Aug 14, 2015 01:47 am
Wes Turner Aug 14, 2015 01:50 am
Nathaniel Smith Aug 14, 2015 02:33 am
Wes Turner Aug 14, 2015 02:41 am
Leonardo Rochael Almeida Sep 08, 2015 07:18 pm
Donald Stufft Sep 08, 2015 07:22 pm
Leonardo Rochael Almeida Sep 08, 2015 07:39 pm
Nathaniel Smith Aug 14, 2015 01:25 am
Robert Collins Aug 14, 2015 01:31 am
Wes Turner Aug 14, 2015 01:38 am
Robert Collins Aug 14, 2015 01:44 am
Wes Turner Aug 14, 2015 01:44 am
Nathaniel Smith Aug 14, 2015 02:14 am
Wes Turner Aug 14, 2015 02:24 am
Robert Collins Aug 14, 2015 02:27 am
Nathaniel Smith Aug 14, 2015 07:38 am
David Cournapeau Aug 13, 2015 05:52 pm
Nathaniel Smith Aug 14, 2015 04:07 am
Chris Barker Aug 14, 2015 04:04 pm
David Cournapeau Aug 14, 2015 04:20 pm
Chris Barker Aug 14, 2015 04:00 pm
Leonardo Rochael Almeida Jul 20, 2015 01:42 am
Nick Coghlan Jul 20, 2015 06:00 am
Chris Barker Jul 20, 2015 05:39 pm
Marcus Smith Sep 06, 2015 04:09 pm
Nick Coghlan Sep 05, 2015 08:35 am
Nick Coghlan Sep 05, 2015 06:44 am
Nick Coghlan Sep 05, 2015 06:43 am
Nathaniel Smith Sep 05, 2015 06:46 am
Donald Stufft Sep 05, 2015 03:06 am
Wes Turner Sep 08, 2015 07:14 pm
Daniel Holth Sep 08, 2015 07:32 pm
Nathaniel Smith Sep 09, 2015 11:49 pm
Nate Coraor Sep 21, 2015 03:33 pm
Nate Coraor Sep 09, 2015 03:06 pm
Donald Stufft Sep 08, 2015 06:33 pm
Donald Stufft Sep 07, 2015 04:02 pm
Marcus Smith Sep 07, 2015 05:51 pm
Wes Turner Sep 07, 2015 10:16 pm
Nate Coraor Sep 03, 2015 02:04 pm
Daniel Holth Sep 03, 2015 01:56 pm
Antoine Pitrou Aug 20, 2015 07:51 pm
Nate Coraor Aug 20, 2015 07:40 pm
Donald Stufft Aug 20, 2015 07:19 pm
Antoine Pitrou Aug 20, 2015 07:14 pm
Steve Dower Aug 14, 2015 04:17 pm
Daniel Holth Aug 20, 2015 06:38 pm
Chris Barker Aug 14, 2015 08:16 pm
Alexander Walters Aug 14, 2015 10:32 pm
Nate Coraor Aug 20, 2015 06:26 pm
Nick Coghlan Sep 05, 2015 02:12 am
Daniel Holth Sep 01, 2015 01:57 pm
Wes Turner Aug 26, 2015 01:42 am
Nate Coraor Aug 27, 2015 07:21 pm
Ben Finney Sep 06, 2015 11:42 pm
Messages in this thread