| Store | Cart

Re: [Distutils] Working toward Linux wheel support

From: Wes Turner <wes....@gmail.com>
Tue, 25 Aug 2015 20:42:54 -0500
On Tue, Aug 25, 2015 at 12:54 PM, Nate Coraor <n...@bx.psu.edu> wrote:

> I've started down this road of Linux platform detection, here's the work> so far:>>     https://bitbucket.org/natefoo/wheel/src/tip/wheel/platform/linux.py>

IDK whether codecs.open(file, 'r', encoding='utf8') is necessary or not?
There are probably distros with Unicode characters in their e.g.
lsb-release files.


>> I'm collecting distribution details here:>>     https://gist.github.com/natefoo/814c5bf936922dad97ff>

Oh wow; thanks!


>>> One thing to note, although it's not used, I'm attempting to label a> particular ABI as stable or unstable, so for example, Debian testing is> unstable, whereas full releases are stable. Arch and Gentoo are always> unstable, Ubuntu is always stable, etc. Hopefully this would be useful in> making a decision about what wheels to allow into PyPI.>

Is it possible to enumerate the set into a table?
e.g. [((distro,ver), {'ABI': 'stable'}), (...)]


>> --nate>> On Mon, Aug 24, 2015 at 2:17 PM, Nate Coraor <n...@bx.psu.edu> wrote:>>> On Mon, Aug 24, 2015 at 1:51 PM, Wes Turner <wes....@gmail.com> wrote:>>>>>>>>>>> On Mon, Aug 24, 2015 at 10:03 AM, Nate Coraor <n...@bx.psu.edu> wrote:>>>>>>> On Fri, Aug 21, 2015 at 2:51 AM, Nick Coghlan <ncog...@gmail.com>>>>> wrote:>>>>>>>>> On 21 August 2015 at 05:58, Robert Collins <robe...@robertcollins.net>>>>>> wrote:>>>>> > On 21 August 2015 at 07:25, Donald Stufft <don...@stufft.io> wrote:>>>>> >>>>>>> >> On August 20, 2015 at 3:23:09 PM, Daniel Holth (dho...@gmail.com)>>>>> wrote:>>>>> >>> If you need that for some reason just put the longer information>>>>> in the>>>>> >>> metadata, inside the WHEEL file for example. Surely "does it work>>>>> on my>>>>> >>> system" dominates, as opposed to "I have a wheel with this>>>>> mnemonic tag,>>>>> >>> now let me install debian 5 so I can get it to run".>>>>> >>>>>>>> >>>>>>>> >>>>>>> >> It’s less about “now let me install Debian 5” and more like tooling>>>>> that doesn’t run *on* the platform but which needs to make decisions based>>>>> on what platform a wheel is built for.>>>>> >>>>>> > Cramming that into the file name is a mistake IMO.>>>>> >>>>>> > Make it declarative data, make it indexable, and index it. We can do>>>>> > that locally as much as via the REST API.>>>>> >>>>>> > That btw is why the draft for referencing external dependencies>>>>> > specifies file names (because file names give an ABI in the context>>>>> of>>>>> > a platform) - but we do need to identify the platform, and>>>>> > platform.distribution should be good enough for that (or perhaps we>>>>> > start depending on lsb-release for detection>>>>>>>>>> LSB has too much stuff in it, so most distros aren't LSB compliant out>>>>> of the box - you have to install extra packages.>>>>>>>>>> /etc/os-release is a better option:>>>>> http://www.freedesktop.org/software/systemd/man/os-release.html>>>>>>>>>>>> As per this discussion, and because I've discovered that the entire>>>> platform module is deprecated in 3.5 (and other amusements, like a>>>> Ubuntu-modified version of platform that ships on Ubuntu - platform as>>>> shipped with CPython detects Ubuntu as debian), I'm switching to>>>> os-release, but even that is unreliable - the file does not exist in>>>> CentOS/RHEL 6, for example. On Debian testing/sid installs, VERSION and>>>> VERSION_ID are unset (which is not wrong - there is no release of testing,>>>> but it does make identifying the platform more complicated since even the>>>> codename is not provided other than at the end of PRETTY_NAME). Regardless>>>> of whether a hash or a human-identifiable string is used to identify the>>>> platform, there still needs to be a way to reliably detect it.>>>>>>>> Unless someone tells me not to, I'm going to default to using>>>> os-release and then fall back to other methods in the event that os-release>>>> isn't available, and this will be in some sort of library alongside>>>> pep425tags in wheel/pip.>>>>>>>> FWIW, os-release's `ID_LIKE` gives us some ability to make assumptions>>>> without explicit need for a binary-compatibility.cfg (although not blindly>>>> - for example, CentOS sets this to "rhel fedora", but of course RHEL/CentOS>>>> and Fedora versions are not congruent).>>>>>>>>>> IIUC, then the value of os-release>>> will be used to generalize>>> the compatible versions of *.so deps>>> of a given distribution at a point in time?>>>>>> This works for distros that don't change [libc] much during a release,>>> but for rolling release models (e.g. arch, gentoo),>>> IDK how this simplification will work.>>> (This is a graph with nodes and edges (with attributes), and rules).>>>>>>> Arch, Gentoo, and other rolling release distributions don't have a stable>> ABI, so by definition I don't think we can support redistributable wheels>> on them. I'm adding platform detection support for them regardless, but I>> don't think there's any way to allow wheels built for these platforms in>> PyPI.>>>>>>> * Keying/namespacing is a simplification which may work.>>>   * *conda preprocessing selectors* (and ~LSB-Python-Conda)>>>      ~'prune' large parts of the graph>>>>>> * Someone mentioned LSB[-Python-Base] (again as a simplification)>>>   * [[package, [version<=>verstr]]]>>>>>> Salt>>> * __salt__['grains']['os']  = "Fedora" || "Ubuntu">>> * __salt__['grains']['os_family'] = "RedHat" || "Debian">>> * __salt__['grains']['osrelease'] = "22" || "14.04">>> * __salt__['grains']['oscodename'] = "Twenty Two" || "trusty">>> * Docs: http://docs.saltstack.com/en/latest/topics/targeting/grains.html>>> * Docs:>>> http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.grains.html#salt.modules.grains.get>>> * Src:>>> https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1018>>> ("def os_data()")>>>>>> $ sudo salt-call --local grains.item os_family os osrelease oscodename>>> local:>>>     ---------->>>     os:>>>         Fedora>>>     os_family:>>>         RedHat>>>     oscodename:>>>         Twenty Two>>>     osrelease:>>>         22>>>>>>>>>>>>> --nate>>>>>>>>>>>>>>>>>>>>>>> My original concern with using that was that it *over*specifies the>>>>> distro (e.g. not only do CentOS and RHEL releases show up as different>>>>> platforms, but so do X.Y releases within a series), but the>>>>> binary-compatibility.txt idea resolves that issue, since a derived>>>>> distro can explicitly identify itself as binary compatible with its>>>>> upstream and be able to use the corresponding wheel files.>>>>>>>>>> Regards,>>>>> Nick.>>>>>>>>>> -->>>>> Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia>>>>> _______________________________________________>>>>> Distutils-SIG maillist  -  Dist...@python.org>>>>> https://mail.python.org/mailman/listinfo/distutils-sig>>>>>>>>>>>>>>>>> _______________________________________________>>>> Distutils-SIG maillist  -  Dist...@python.org>>>> https://mail.python.org/mailman/listinfo/distutils-sig>>>>>>>>>>>>>>

_______________________________________________
Distutils-SIG maillist  -  Dist...@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Recent Messages in this Thread
Nate Coraor Jul 16, 2015 05:41 pm
Nick Coghlan Sep 06, 2015 11:32 pm
Nick Coghlan Sep 05, 2015 02:56 am
Marcus Smith Sep 07, 2015 04:11 am
Nate Coraor Sep 03, 2015 05:22 pm
Nick Coghlan Sep 07, 2015 04:26 am
Daniel Holth Aug 20, 2015 07:22 pm
Donald Stufft Aug 20, 2015 07:25 pm
Donald Stufft Sep 02, 2015 11:45 pm
Daniel Holth Sep 03, 2015 12:15 pm
Nate Coraor Aug 24, 2015 03:03 pm
Wes Turner Aug 24, 2015 05:51 pm
Nick Coghlan Jul 17, 2015 08:22 am
Chris Barker Jul 17, 2015 03:36 pm
Antoine Pitrou Jul 17, 2015 03:46 pm
Chris Barker Jul 17, 2015 03:53 pm
Andrea Bedini Jul 18, 2015 07:00 am
Tres Seaver Jul 21, 2015 03:25 am
Leonardo Rochael Almeida Jul 21, 2015 03:07 pm
Marcus Smith Jul 17, 2015 04:50 pm
Olivier Grisel Jul 17, 2015 06:34 pm
Daniel Holth Jul 17, 2015 08:18 pm
Chris Barker - NOAA Federal Jul 18, 2015 01:13 am
Daniel Holth Jul 18, 2015 02:11 am
Paul Moore Jul 18, 2015 11:51 am
Nick Coghlan Jul 20, 2015 05:50 am
Chris Barker Jul 20, 2015 05:37 pm
Paul Moore Jul 20, 2015 06:37 pm
Nick Coghlan Jul 27, 2015 02:19 pm
Nate Coraor Jul 27, 2015 07:07 pm
Oscar Benjamin Jul 21, 2015 04:38 pm
Chris Barker Jul 24, 2015 06:52 pm
Oscar Benjamin Jul 28, 2015 03:02 pm
Wes Turner Jul 28, 2015 04:21 pm
Nate Coraor Aug 12, 2015 08:21 pm
Robert Collins Aug 12, 2015 11:49 pm
Nathaniel Smith Aug 13, 2015 01:05 am
Nate Coraor Aug 13, 2015 02:07 pm
Leonardo Rochael Almeida Aug 13, 2015 07:30 pm
Wes Turner Aug 13, 2015 07:43 pm
Nathaniel Smith Aug 14, 2015 01:47 am
Wes Turner Aug 14, 2015 01:50 am
Nathaniel Smith Aug 14, 2015 02:33 am
Wes Turner Aug 14, 2015 02:41 am
Leonardo Rochael Almeida Sep 08, 2015 07:18 pm
Donald Stufft Sep 08, 2015 07:22 pm
Leonardo Rochael Almeida Sep 08, 2015 07:39 pm
Nathaniel Smith Aug 14, 2015 01:25 am
Robert Collins Aug 14, 2015 01:31 am
Wes Turner Aug 14, 2015 01:38 am
Robert Collins Aug 14, 2015 01:44 am
Wes Turner Aug 14, 2015 01:44 am
Nathaniel Smith Aug 14, 2015 02:14 am
Wes Turner Aug 14, 2015 02:24 am
Robert Collins Aug 14, 2015 02:27 am
Nathaniel Smith Aug 14, 2015 07:38 am
David Cournapeau Aug 13, 2015 05:52 pm
Nathaniel Smith Aug 14, 2015 04:07 am
Chris Barker Aug 14, 2015 04:04 pm
David Cournapeau Aug 14, 2015 04:20 pm
Chris Barker Aug 14, 2015 04:00 pm
Leonardo Rochael Almeida Jul 20, 2015 01:42 am
Nick Coghlan Jul 20, 2015 06:00 am
Chris Barker Jul 20, 2015 05:39 pm
Marcus Smith Sep 06, 2015 04:09 pm
Nick Coghlan Sep 05, 2015 08:35 am
Nick Coghlan Sep 05, 2015 06:44 am
Nick Coghlan Sep 05, 2015 06:43 am
Nathaniel Smith Sep 05, 2015 06:46 am
Donald Stufft Sep 05, 2015 03:06 am
Wes Turner Sep 08, 2015 07:14 pm
Daniel Holth Sep 08, 2015 07:32 pm
Nathaniel Smith Sep 09, 2015 11:49 pm
Nate Coraor Sep 21, 2015 03:33 pm
Nate Coraor Sep 09, 2015 03:06 pm
Donald Stufft Sep 08, 2015 06:33 pm
Donald Stufft Sep 07, 2015 04:02 pm
Marcus Smith Sep 07, 2015 05:51 pm
Wes Turner Sep 07, 2015 10:16 pm
Nate Coraor Sep 03, 2015 02:04 pm
Daniel Holth Sep 03, 2015 01:56 pm
Antoine Pitrou Aug 20, 2015 07:51 pm
Nate Coraor Aug 20, 2015 07:40 pm
Donald Stufft Aug 20, 2015 07:19 pm
Antoine Pitrou Aug 20, 2015 07:14 pm
Steve Dower Aug 14, 2015 04:17 pm
Daniel Holth Aug 20, 2015 06:38 pm
Chris Barker Aug 14, 2015 08:16 pm
Alexander Walters Aug 14, 2015 10:32 pm
Nate Coraor Aug 20, 2015 06:26 pm
Nick Coghlan Sep 05, 2015 02:12 am
Daniel Holth Sep 01, 2015 01:57 pm
Wes Turner Aug 26, 2015 01:42 am
Nate Coraor Aug 27, 2015 07:21 pm
Ben Finney Sep 06, 2015 11:42 pm
Messages in this thread