| msg336503 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2019-02-25 09:03 |
| TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x: * The tests should be skipped on this buildbot worker * Or the feature should be fixed on FreeBSD https://buildbot.python.org/all/#/builders/168/builds/617 Example: ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 3770, in test_shared_memory_ShareableList_basics sl = shared_memory.ShareableList( File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/shared_memory.py", line 305, in __init__ self.shm = SharedMemory(name, create=True, size=requested_size) File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/shared_memory.py", line 88, in __init__ self._fd = _posixshmem.shm_open( OSError: [Errno 22] Invalid argument: 'psm_f66c2bc7b9' Failures: ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ERROR: test_shared_memory_ShareableList_pickling (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ERROR: test_shared_memory_SharedMemoryManager_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ERROR: test_shared_memory_across_processes (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ERROR: test_shared_memory_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_fork.WithProcessesTestSharedMemory) ERROR: test_shared_memory_ShareableList_pickling (test.test_multiprocessing_fork.WithProcessesTestSharedMemory) ERROR: test_shared_memory_SharedMemoryManager_basics (test.test_multiprocessing_fork.WithProcessesTestSharedMemory) ERROR: test_shared_memory_across_processes (test.test_multiprocessing_fork.WithProcessesTestSharedMemory) ERROR: test_shared_memory_basics (test.test_multiprocessing_fork.WithProcessesTestSharedMemory) ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ERROR: test_shared_memory_ShareableList_pickling (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ERROR: test_shared_memory_SharedMemoryManager_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ERROR: test_shared_memory_across_processes (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ERROR: test_shared_memory_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) |
|
|
| msg336522 - (view) |
Author: Davin Potts (davin) * |
Date: 2019-02-25 14:07 |
| Though apparently undocumented on FreeBSD, their implementation of shm_open differs from others in the following way: all names for shared memory blocks *must* begin with a slash. This requirement does not exist on OpenBSD. According to its man page on shm_open, FreeBSD does at least communicate the following non-standard, additional requirement: Two processes opening the same path are guaranteed to access the same shared memory object if and only if path begins with a slash (`/') character. Given that this requirement is not universal and because a leading slash controls other behaviors on platforms like Windows, it would be confusing to make a leading slash a universal requirement. Likewise, requiring users on FreeBSD to be aware of this nuance would be contrary to the goals of the SharedMemory class. I will prepare a patch to prepend a leading slash onto the requested shared memory block name and detect the need for it. I have verified that this does solve the problem on FreeBSD and that all tests then pass. I will test NetBSD and DragonflyBSD to see if they also impose FreeBSD's undocumented requirement. |
|
|
| msg336531 - (view) |
Author: Davin Potts (davin) * |
Date: 2019-02-25 15:08 |
| In local testing, I found the following systems to impose the leading slash as a requirement for simply creating a shared memory block: * NetBSD 8.0 * FreeBSD 12.x * TrueOS 18.12 (the OS formerly known as PC-BSD) I found the following systems to have no required leading slash and all tests currently pass without modification: * OpenBSD 6.4 * DragonflyBSD 5.4 |
|
|
| msg336553 - (view) |
Author: Davin Potts (davin) * |
Date: 2019-02-25 21:04 |
| I have locally tested GH-12036 on all 5 of the aforementioned OSes and all are made happy by the patch. Victor: If we want to go ahead and apply this patch right away to hopefully make the FreeBSD buildbot go green, the nature of this change is sufficiently small that it will still be easy to change during the alpha period. |
|
|
| msg336559 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2019-02-25 21:29 |
| Apply the patch, we will see if the buildbot color changes :-) |
|
|
| msg336566 - (view) |
Author: Davin Potts (davin) * |
Date: 2019-02-25 22:41 |
| New changeset aadef2b41600cb6a4f845cdc4cea001c916d8745 by Davin Potts in branch 'master': bpo-36102: Prepend slash to all POSIX shared memory block names (#12036) https://github.com/python/cpython/commit/aadef2b41600cb6a4f845cdc4cea001c916d8745 |
|
|
| msg336569 - (view) |
Author: Davin Potts (davin) * |
Date: 2019-02-25 22:46 |
| Closing. Thank you Giampaolo for jumping in so quickly to review! Thank you Victor for catching this on the buildbot. Though what is this talk of "_if_ the color changes"? ;) |
|
|
| msg336571 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2019-02-25 23:13 |
| FYI the test also failed on AMD64 FreeBSD 10-STABLE Non-Debug 3.x: https://buildbot.python.org/all/#/builders/167/builds/600 I will keep on eye on this buildbot worker as well ;-) |
|
|
| msg336595 - (view) |
Author: Eryk Sun (eryksun) *  |
Date: 2019-02-26 02:19 |
| > Given that this requirement is not universal and because a leading > slash controls other behaviors on platforms like Windows Forward slash has no special meaning anywhere in the names of NT objects, such as memory Section objects (aka file-mapping objects) and registry Key objects. Only the file API translates forward slash to backslash, and that's only implemented for the sake of DOS compatibility in a few core functions that normalize and translate DOS paths. Maybe you're thinking of the "Global\\" and "Session\\" prefixes for named kernel objects, which require a backslash since the name is really a handle-relative NT path. The session's object directory contains object symbolic links named "Global" and "Session". A name such as "Global\spam" resolves to "spam" in the global object directory, but "Global/spam", with forward slash, is just a regular name in the local directory. |
|
|