Issue 32493: UUID Module - FreeBSD build failure (original) (raw)
Created on 2018-01-04 18:57 by David Carlier, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (39)
Author: David Carlier (David Carlier) *
Date: 2018-01-04 18:57
UUID module build fails on FreeBSD since it supports uuid_create function.
Author: Antoine Pitrou (pitrou) * 
Date: 2018-01-05 12:37
The AIX-specific code was contributed by Michael Felt in .
Author: Antoine Pitrou (pitrou) *
Date: 2018-01-05 12:38
Michael, does AIX have uint32_t? If so, we could happily drop the unsigned32 reference.
Author: Michael Felt (Michael.Felt) *
Date: 2018-01-08 11:35
On 05/01/2018 13:38, Antoine Pitrou wrote:
Antoine Pitrou <pitrou@free.fr> added the comment:
Michael, does AIX have uint32_t? If so, we could happily drop the unsigned32 reference.
Yes, AIX has unit32_t.
michael@x071:[/home/michael]grep unsigned32 /usr/include/.h /usr/include/sys/.h | grep typed /usr/include/uuid.h:typedef u_int32_t unsigned32; /* positive 32 bit integer */ /usr/include/uuid.h:typedef unsigned32 boolean32;
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32493>
Author: Antoine Pitrou (pitrou) *
Date: 2018-01-08 11:37
Seems to be in inttypes.h: https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.files/inttypes.h.htm
Author: David Carlier (David Carlier) *
Date: 2018-01-08 11:47
Perfect. That solves in the process OpenBSD uuid module build too.
Author: Michael Felt (Michael.Felt) *
Date: 2018-01-08 14:55
On 08/01/2018 12:47, David Carlier wrote:
David Carlier <dcarlier@afilias.info> added the comment:
Perfect. That solves in the process OpenBSD uuid module build too.
p.s. I did not 'invent' unsigned32 - just took the syntax from the man page: https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.basetrf2/uuid_create.htm
- and I just did the grep above to show the type existed AND was
directly related to unsigned32 - even in AIX include files.
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32493>
Author: David Carlier (David Carlier) *
Date: 2018-01-09 15:32
I guessed that :-) I trusted it worked just fine for you. To be honest I know nearly nothing about AIX specificities :-)
Author: Antoine Pitrou (pitrou) *
Date: 2018-01-09 19:38
New changeset b4ebaa7099c3413b42a97777581c4ca560fe7540 by Antoine Pitrou (David Carlier) in branch 'master': bpo-32493: Not only AIX, but FreeBSD has uuid_create support (#5089) https://github.com/python/cpython/commit/b4ebaa7099c3413b42a97777581c4ca560fe7540
Author: Antoine Pitrou (pitrou) *
Date: 2018-01-09 19:38
Thank you for the fix David!
Author: STINNER Victor (vstinner) *
Date: 2018-01-17 09:56
New changeset b4ebaa7099c3413b42a97777581c4ca560fe7540 by Antoine Pitrou (David Carlier) in branch 'master': bpo-32493: Not only AIX, but FreeBSD has uuid_create support (#5089)
This change introduced a regression on AMD64 FreeBSD 10.x Shared 3.x:
http://buildbot.python.org/all/#/builders/87/builds/463
====================================================================== FAIL: test_uuid1 (test.test_uuid.TestUUIDWithExtModule)
Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_uuid.py", line 320, in test_uuid1 equal(u.version, 1) AssertionError: 14 != 1
Author: David CARLIER (devnexen) *
Date: 2018-01-17 10:16
Might comes from uuid1 function itself ... e.g. line 704 not setting version "field".
Author: Michael Felt (Michael.Felt) *
Date: 2018-01-17 15:21
Actually, the libc/libuuid function called should be setting the version/variant values as well.
So, I think the AMD64 FreeBSD 10.x Shared 3.x uuid_generate() function is wrong (if that is what it is using - was it/can it also use the uuid_generate* routines?
FYI:
I notice the AIX include file does not actually define the bits for RFC_4122, etc.., but the include file comments does show that the implementation is taking them into account.
/** The strlen() of a UUID string (does not include terminating null) / #define UUID_STRLEN 36 /*
- Constant value indicating an as-yet undetermined UUID for a thing. Spells \c
- "\0\0UUID\x10_\x80UNKNOWN" in ASCII. Zeroes at the top make it very unlikely
- to collide with a uuid_create()d value, which puts a timestamp in that spot.
- The \c "\x10" and \c "\x80" are reserved/version/mask bits that make this
- conform to the format expected by the uuid.h API. */
#define UUID_UNKNOWN (uuid_t){0x00005555, 0x4944, 0x105F, 0x80, 0x55, {0x4E, 0x4B, 0x4E, 0x4F, 0x57, 0x4E}} #define UUID_UNKNOWN_STR "00005555-4944-105f-8055-4e4b4e4f574e"
I wrote a simple program to examine uuid for myself.
Maybe - if you ran a loop (looking at uuid1().variant and uuid1().version you could see if there is an incorrect version showing up. Rather than python - that would seem to be a library function error.
My very simple program:
cat -n ../x4.py 1 import _uuid 2 import uuid 3 _generate_time_safe = _uuid.generate_time_safe 4 _has_uuid_generate_time_safe = _uuid.has_uuid_generate_time_safe 5 uuid_time, _ = _generate_time_safe() 6 7 uu1a = uuid.UUID(bytes=uuid_time) 8 uu1b = uuid.uuid1() 9 uu4 = uuid.uuid4() 10 11 print("a)uuid1 = %x" % uu1a.int) 12 print("b)uuid1 = %x" % uu1b.int) 13 print("") 14 print("a uuid1 = ",uu1a.fields, uu1a.version, uu1a.variant) 15 print("b uuid1 = ",uu1b.fields, uu1b.version, uu1b.variant) 16 uu1c = uuid.uuid1() 17 print("c uuid1 = ",uu1c.fields, uu1c.version, uu1c.variant) 18 print("") 19 print(" uuid4 = %x" % uu4.int) 20 print(" uuid4 = ",uu4.fields, uu4.version, uu4.variant) 21 print("") 22 uu4 = uuid.uuid4() 23 print(" uuid4 = %x" % uu4.int) 24 print(" uuid4 = ",uu4.fields, uu4.version, uu4.variant)
Author: Michael Felt (Michael.Felt) *
Date: 2018-01-17 15:32
typo here:
So, I think the AMD64 FreeBSD 10.x Shared 3.x uuid_create() function is wrong (if that is what it is using - was it/can it also use the uuid_generate* routines?
i.e., does AMD FreeBSD use uuid_create() or uuid_generate() - or can it use both?
Could someone with AMD try: Modules/_uuidmodule.c
+12 static PyObject * +13 py_uuid_generate_time_safe(void) +14 { +15 uuid_t uuid; +16 #ifdef HAVE_UUID_GENERATE_TIME_SAFE +17 int res; +18 +19 res = uuid_generate_time_safe(uuid); +20 return Py_BuildValue("y#i", (const char *) uuid, sizeof(uuid), res); +21 #elif HAVE_UUID_GENERATE_TIME +22 uuid_generate_time(uuid); +23 return Py_BuildValue("y#O", (const char *) uuid, sizeof(uuid), Py_None); +24 #elif HAVE_UUID_CREATE +25 uint32_t status; +26 uuid_create(&uuid, &status); +27 return Py_BuildValue("y#i", (const char *) &uuid, sizeof(uuid), (int) status); +28 #else +29 #error "no uuid library function available" +30 #endif +31 }
Author: David CARLIER (devnexen) *
Date: 2018-01-17 15:36
uuid_create only I think.
Author: David CARLIER (devnexen) *
Date: 2018-01-17 15:41
Gives same outcome but maybe someone else can confir,
Author: Michael Felt (Michael.Felt) *
Date: 2018-01-22 23:41
On 1/17/2018 11:16 AM, David CARLIER wrote:
David CARLIER <devnexen@gmail.com> added the comment:
Might comes from uuid1 function itself ... e.g. line 704 not setting version "field". IMHO: for uuid1 - the version number should be in the 16 bytes filled by uuid_create. The implementation of uuid4() in uuid.py "has to" specify the attribute by setting the UUID version attribute to 4, so that when an uuid4 instance gets asked it's version it says 4. A random number gives no such guarantee.
Taking a hint from the FreeBSD man page for uuid_create() - that says it provides a DCE 1.1 RPC implementation - I found this link: http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm
Therein:
The timestamp is a 60 bit value. For UUID version 1, this is represented by Coordinated Universal Time (UTC) as a count of 100-nanosecond intervals since 00:00:00.00, 15 October 1582 (the date of Gregorian reform to the Christian calendar).
The version number is multiplexed in the 4 most significant bits of the time_hi_and_version field. UUID version Field <http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagtcjh_35> lists currently defined versions of the UUID.
Again - a random number is not likely to correctly set the 4 most significant bits - so another way is needed to make an instance it is version 4. IMHO - the output of uuid_create should satisfy the needs of uuid version 1.
nosy: +David CARLIER2
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32493>
Author: David CARLIER (devnexen) *
Date: 2018-01-23 14:34
In OpenBSD it s even "worse", the version for each call is random (not a surprise when looking at the source). The question is, do we go back to support only AIX or do we accept somehow wrong version ?
Author: Michael Felt (Michael.Felt) *
Date: 2018-02-01 22:40
Now that it is visible - maybe OpenBSD will treat it as a bug and get the "version bits" to work as their man page (https://man.openbsd.org/uuid.3) says it does:
STANDARDS <https://man.openbsd.org/uuid.3#STANDARDS>
The uuid_compare(), uuid_create(), uuid_create_nil(), uuid_equal(), uuid_from_string(), uuid_hash(), uuid_is_nil(), and uuid_to_string() functions are compatible with the DCE 1.1 RPC specification.
The FreeBSD man page (https://www.freebsd.org/cgi/man.cgi?query=uuid_create) says:
NAME <https://www.freebsd.org/cgi/man.cgi?query=uuid_create#end> uuidcompare,uuidcreate, uuidcreatenil,uuidequal,uuidfromstring, uuidhash, uuidisnil,uuidtostring -- DCE 1.1 compliant UUID func- tions
Re the DCE 1.1 RPC implementation/description - I found this link: http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm
And this comment:
The version number is multiplexed in the 4 most significant bits of the time_hi_and_version field. UUID version Field <http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagtcjh_35> lists currently defined versions of the UUID.
msb1 msb2 msb3 msb4 Version Description 0 0 0 1 1 DCE version, as specified herein. 0 0 1 0 2 DCE Security version, with embedded POSIX UIDs.
So, maybe either, or both: OpenBSD and FreeBSD fix their implementation. Problem solved?!
On 1/23/2018 3:34 PM, David CARLIER wrote:
David CARLIER <devnexen@gmail.com> added the comment:
In OpenBSD it s even "worse", the version for each call is random (not a surprise when looking at the source). The question is, do we go back to support only AIX or do we accept somehow wrong version ?
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32493>
Author: David CARLIER (devnexen) *
Date: 2018-02-03 12:40
Probably. Seems workable under Apple otherwise.
Author: Ned Deily (ned.deily) *
Date: 2018-02-27 14:27
We are still seeing the 3.7 and 3.8 FreeBSD buildbot failures Victor identified in . Can someone please take this on and produce a PR?
Author: Michael Felt (Michael.Felt) *
Date: 2018-03-01 23:00
If it is a bug in OpenBSD how do you propose we "fix" this.
imho, the problem is "outside" python.
however, until it is fixed in openbsd, would you accept an 3error macro that prevents the mod from being built on the specific version of openbsd that fails?
Author: Michael Felt (Michael.Felt) *
Date: 2018-03-01 23:01
oops, sain openbsd, should be freebsd
Author: Ned Deily (ned.deily) *
Date: 2018-05-24 05:35
This problem is still unresolved and causing buildbot failures. It really should be fixed before we tag 3.7.0rc1. Serhiy, could you take a look at it, please? Thanks!
http://buildbot.python.org/all/#/builders/87/builds/1014/steps/4/logs/stdio
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2018-05-24 05:40
I will be busy next few hours, but will take this issue as soon as I can. However I do not promise the result.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2018-05-24 15:00
Is AIX big-endian?
On *BSD systems uuid_t is a structure of integers with platform-depending endianess. Thus on little-endian platform UUID should be called with the bytes_le argument. This doesn't fix test on OpenBSD and NetBSD, but at least the result is stable (version=4).
Using bytes_le on Linux breaks tests. Seems uuid_generate_time_safe() always returns bytes in big-endian order.
PR 7098 adds _uuid.little_endian which is true on little-endian platforms using uuid_create(), and false otherwise. Actually there are many ways of solving this problem, the choice of this design was arbitrary. _uuid.generate_time_safe() could return a 3-tuple instead of 2-tuple, or there could be two separate functions: _uuid.generate_time_safe() and _uuid.create().
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2018-05-24 17:38
PR 7099 is an alternate solution. I uses uuid_enc_be() which is not part of the DCE 1.1 RPC specification but exists on all *BSD.
Both solutions gives the same result on *BSD and AIX. I have no idea what solution is appropriate for little-endian non-BSD platform (if there is such platform supporting uuid_create()).
Author: STINNER Victor (vstinner) *
Date: 2018-05-24 22:45
New changeset 17d8830312d82e7de42ab89739b0771f712645ff by Victor Stinner (Serhiy Storchaka) in branch 'master': bpo-32493: Fix uuid.uuid1() on FreeBSD. (GH-7099) https://github.com/python/cpython/commit/17d8830312d82e7de42ab89739b0771f712645ff
Author: miss-islington (miss-islington)
Date: 2018-05-24 23:23
New changeset 5734f41a9b46b4fd65b6ba90240b108f8a0b7c57 by Miss Islington (bot) in branch '3.7': bpo-32493: Fix uuid.uuid1() on FreeBSD. (GH-7099) https://github.com/python/cpython/commit/5734f41a9b46b4fd65b6ba90240b108f8a0b7c57
Author: STINNER Victor (vstinner) *
Date: 2018-05-24 23:29
Thank you Serhiy Storchaka for the fix! If someone cares about AIX, please open a different issue. (AIX has other more important functions that you be fixed, IHMO.)
Author: Michael Felt (Michael.Felt) *
Date: 2018-06-08 06:45
@vstinner - I know this is "closed", however, I submit a minor PR to fix an error in PR7104. The logic of configure.ac always defines HAVE_UUID_ENC_BE.
I cannot proceed with PR5183 (aka ) until this is repaired.
Thx!
Author: STINNER Victor (vstinner) *
Date: 2018-06-08 06:48
@vstinner - I know this is "closed", however, I submit a minor PR to fix an error in PR7104. The logic of configure.ac always defines HAVE_UUID_ENC_BE. I cannot proceed with PR5183 (aka ) until this is repaired.
The title of this issue makes it very specific to FreeBSD, so it would be better to create a new one for AIX. What your write for the commit description is not appropriate for a commit, but it's fine for the description of a bug.
Please open a new issue to track uuid changes for AIX. I know that you have a second one for Python <= 3.6.
Author: Michael Felt (Michael.Felt) *
Date: 2018-06-08 14:55
OK. I know I do not understand this well - when it goes in A, or B.
So, if I understand correctly, like this issue was created after uuid_create() was added for AIX and created issues for FreeBSD )that I did not know had uuid_create(), I need to create a new issue where I say this issue created an issue for AIX?
Seems like a lot of work. I only put it here because it is a very simple change directly related to the last PR.
Just verify that is what you want and I'll get to it as I can. In any case, I'll pull the PR - as it is specific to AIX.
On 6/8/2018 12:48 AM, STINNER Victor wrote:
STINNER Victor <vstinner@redhat.com> added the comment:
@vstinner - I know this is "closed", however, I submit a minor PR to fix an error in PR7104. The logic of configure.ac always defines HAVE_UUID_ENC_BE. I cannot proceed with PR5183 (aka ) until this is repaired. The title of this issue makes it very specific to FreeBSD, so it would be better to create a new one for AIX. What your write for the commit description is not appropriate for a commit, but it's fine for the description of a bug.
Please open a new issue to track uuid changes for AIX. I know that you have a second one for Python <= 3.6.
resolution: fixed -> status: closed -> open
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32493>
Author: Ned Deily (ned.deily) *
Date: 2018-06-09 21:59
New changeset 20cd5c6e21559f35feded5b3cfa9069a281d4325 by Ned Deily (Michael Felt) in branch '3.7': bpo-32493: Correct test for uuid_enc_be availability in configure.ac. (GH-7511) https://github.com/python/cpython/commit/20cd5c6e21559f35feded5b3cfa9069a281d4325
Author: Ned Deily (ned.deily) *
Date: 2018-06-09 22:19
New changeset ced0adb2638e4c02945bfa82e15595918eef74f1 by Ned Deily in branch 'master': bpo-32493: Correct test for uuid_enc_be availability in configure.ac. (GH-7511) (GH-7567) https://github.com/python/cpython/commit/ced0adb2638e4c02945bfa82e15595918eef74f1
Author: Ned Deily (ned.deily) *
Date: 2018-06-09 22:46
As noted on PR 7511 and PR 7567, I have merged Michael's configure.ac fixes for testing for uuid_enc_be availability (independent of platform) that were incorrect in earlier commits for this issue. Merged for 3.7.0rc1 and master (3.8). Thanks, everyone!
Author: STINNER Victor (vstinner) *
Date: 2018-06-10 11:52
As noted on PR 7511 and PR 7567, I have merged Michael's configure.ac fixes for testing for uuid_enc_be availability (independent of platform) that were incorrect in earlier commits for this issue. Merged for 3.7.0rc1 and master (3.8). Thanks, everyone!
Thanks you Ned and Michael. Sorry for the confusion. I was first confused that the fix for master added a lot of code using ctypes. The final fix is the right fix for 3.7 and master ;-)
Michael: if you want to fix uuid on 3.6, I would suggest to open a new issue, because it seems that the fix is much more complex and unrelated to _uuid (module added to Python 3.7).
Author: Michael Felt (Michael.Felt) *
Date: 2018-06-10 20:51
I was not aware that _uuid was new to python3-3.7. I thought it had been around for a long time.
Bpo-28009 goes back two years and i was unaware of uuid_create().
Would it be easier to split it into 3 issues? One for unixdll, one for netstat, and one for test_uuid?
Michael
Sent from my iPhone
On 10 Jun 2018, at 13:52, STINNER Victor <report@bugs.python.org> wrote:
STINNER Victor <vstinner@redhat.com> added the comment:
As noted on PR 7511 and PR 7567, I have merged Michael's configure.ac fixes for testing for uuid_enc_be availability (independent of platform) that were incorrect in earlier commits for this issue. Merged for 3.7.0rc1 and master (3.8). Thanks, everyone!
Thanks you Ned and Michael. Sorry for the confusion. I was first confused that the fix for master added a lot of code using ctypes. The final fix is the right fix for 3.7 and master ;-)
Michael: if you want to fix uuid on 3.6, I would suggest to open a new issue, because it seems that the fix is much more complex and unrelated to _uuid (module added to Python 3.7).
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32493>
Author: STINNER Victor (vstinner) *
Date: 2018-06-11 11:30
Would it be easier to split it into 3 issues? One for unixdll, one for netstat, and one for test_uuid?
One issue for AIX should be enough.
History
Date
User
Action
Args
2022-04-11 14:58:56
admin
set
github: 76674
2018-06-11 11:30:30
vstinner
set
messages: +
2018-06-10 20:51:50
Michael.Felt
set
messages: +
2018-06-10 11:52:47
vstinner
set
messages: +
2018-06-09 22:46:36
ned.deily
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> resolved
2018-06-09 22:19:59
ned.deily
set
messages: +
2018-06-09 22:04:14
ned.deily
set
stage: resolved -> patch review
pull_requests: + <pull%5Frequest7198>
2018-06-09 21:59:05
ned.deily
set
messages: +
2018-06-08 14:55:05
Michael.Felt
set
messages: +
2018-06-08 06:48:20
vstinner
set
status: closed -> open
resolution: fixed -> (no value)
messages: +
2018-06-08 06:45:12
Michael.Felt
set
messages: +
2018-06-08 01:35:32
Michael.Felt
set
pull_requests: + <pull%5Frequest7139>
2018-06-06 21:38:34
Michael.Felt
set
pull_requests: + <pull%5Frequest7082>
2018-05-24 23:29:50
ned.deily
set
priority: deferred blocker ->
2018-05-24 23:29:03
vstinner
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> resolved
2018-05-24 23:23:01
miss-islington
set
nosy: + miss-islington
messages: +
2018-05-24 22:46:19
miss-islington
set
pull_requests: + <pull%5Frequest6743>
2018-05-24 22:45:18
vstinner
set
messages: +
2018-05-24 17:38:03
serhiy.storchaka
set
messages: +
2018-05-24 17:33:41
serhiy.storchaka
set
pull_requests: + <pull%5Frequest6736>
2018-05-24 15:01:14
serhiy.storchaka
set
stage: needs patch -> patch review
2018-05-24 15:00:44
serhiy.storchaka
set
messages: +
stage: patch review -> needs patch
2018-05-24 14:49:17
serhiy.storchaka
set
keywords: + patch
stage: needs patch -> patch review
pull_requests: + <pull%5Frequest6735>
2018-05-24 05:40:46
serhiy.storchaka
set
messages: +
2018-05-24 05:35:11
ned.deily
set
nosy: + serhiy.storchaka
messages: +
2018-03-01 23:01:40
Michael.Felt
set
messages: +
2018-03-01 23:00:42
Michael.Felt
set
messages: +
2018-02-27 14:27:31
ned.deily
set
priority: normal -> deferred blocker
versions: + Python 3.8
nosy: + ned.deily
messages: +
2018-02-21 02:04:43
koobs
set
stage: resolved -> needs patch
2018-02-03 12:40:17
devnexen
set
messages: +
2018-02-01 22:40:47
Michael.Felt
set
messages: +
2018-01-26 19:04:59
barry
set
nosy: + barry
2018-01-23 14:34:20
devnexen
set
messages: +
2018-01-22 23:41:55
Michael.Felt
set
messages: +
2018-01-17 15:41:20
devnexen
set
messages: +
2018-01-17 15:36:45
devnexen
set
messages: +
2018-01-17 15:32:24
Michael.Felt
set
messages: +
2018-01-17 15:21:24
Michael.Felt
set
messages: +
2018-01-17 10:16:53
devnexen
set
nosy: + devnexen
messages: +
2018-01-17 09:56:48
vstinner
set
status: closed -> open
nosy: + vstinner
messages: +
resolution: fixed -> (no value)
2018-01-09 19:38:34
pitrou
set
status: open -> closed
resolution: fixed
messages: +
stage: resolved
2018-01-09 19:38:10
pitrou
set
messages: +
2018-01-09 15:32:47
David Carlier
set
messages: +
2018-01-08 14:55:56
Michael.Felt
set
messages: +
2018-01-08 11:47:34
David Carlier
set
messages: +
2018-01-08 11:37:22
pitrou
set
messages: +
2018-01-08 11:35:34
Michael.Felt
set
messages: +
2018-01-05 12:38:28
pitrou
set
messages: +
2018-01-05 12:37:07
pitrou
set
nosy: + Michael.Felt, pitrou
messages: +
2018-01-04 18:57:50
David Carlier
create