| msg264721 - (view) |
Author: Xavier de Gaye (xdegaye) *  |
Date: 2016-05-03 13:37 |
| test_concurrent_futures fails on an android emulator running an x86 system image at API level 21. There are multiple ERRORs, all resulting in the same traceback: Traceback (most recent call last): File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_concurrent_futures.py", line 68, in setU p self.executor = self.executor_type(max_workers=self.worker_count) File "/sdcard/org.bitbucket.pyona/lib/python3.6/concurrent/futures/process.py", line 390, in __init__ EXTRA_QUEUED_CALLS) File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/context.py", line 101, in Queue return Queue(maxsize, ctx=self.get_context()) File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/queues.py", line 42, in __init__ self._rlock = ctx.Lock() File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/context.py", line 66, in Lock return Lock(ctx=self.get_context()) File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/synchronize.py", line 163, in __in it__ SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx) File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/synchronize.py", line 60, in __ini t__ unlink_now) OSError: [Errno 38] Function not implemented The ERRORs: =========== test_duplicate_futures (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR test_killed_child (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_map_chunksize (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_max_workers_negative (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_no_stale_references (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_shutdown_race_issue12456 (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_traceback (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR test_first_completed_some_already_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR test_first_exception_some_already_complete (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR |
|
|
| msg264796 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-04 09:06 |
| On android: root@generic_x86:/data/local/tmp # python Python 3.6.0a0 (default:811ccdee6f87+, May 4 2016, 10:31:14) [GCC 4.9 20140827 (prerelease)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from _multiprocessing import sem_unlink >>> sem_unlink('foo') Traceback (most recent call last): File "", line 1, in OSError: [Errno 38] Function not implemented >>> |
|
|
| msg264802 - (view) |
Author: Berker Peksag (berker.peksag) *  |
Date: 2016-05-04 10:59 |
| It normally should raise an ImportError if sem_unlink is not available. Could you try the attached patch please? |
|
|
| msg264824 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-04 15:15 |
| Patch tested, it does not fix the problem. This is because in pyconfig.h: /* Define to 1 if you have the `sem_unlink' function. */ #define HAVE_SEM_UNLINK 1 Android declares the function but does not implement it. This is not the only case, for example issue #26857. |
|
|
| msg265075 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2016-05-07 16:11 |
| Thanks for testing. Here is a new patch. |
|
|
| msg265079 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-07 17:00 |
| Here is the result of the test: root@generic_x86:/data/local/tmp # python -m test -v test_concurrent_futures == CPython 3.6.0a0 (default:811ccdee6f87+, May 7 2016, 18:51:28) [GCC 4.9 20140827 (prerelease)] == Linux-3.4.67+-i686-with-libc little-endian == hash algorithm: fnv 32bit == /data/local/tmp/test_python_2024 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0) Run tests sequentially 0:00:00 [1/1] test_concurrent_futures test_concurrent_futures skipped -- This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770. 1 test skipped: test_concurrent_futures Total duration: 0:00:01 |
|
|
| msg265080 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-07 17:01 |
| > Here is the result of the test: I meant, results with the new patch. |
|
|
| msg265082 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-05-07 17:38 |
| New changeset 1030aa8357a7 by Berker Peksag in branch 'default': Issue #26924: Do not define _multiprocessing.sem_unlink under Android https://hg.python.org/cpython/rev/1030aa8357a7 |
|
|
| msg265083 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2016-05-07 17:39 |
| Thanks, Xavier. |
|
|
| msg265084 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-05-07 18:13 |
| New changeset eee959fee5f5 by Berker Peksag in branch 'default': Issue #26924: Fix Windows buildbots https://hg.python.org/cpython/rev/eee959fee5f5 |
|
|
| msg265085 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2016-05-07 18:17 |
| Xavier, could you try this alternative patch? |
|
|
| msg265090 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-07 19:19 |
| Attached the results with sem_unlink_alt.diff. |
|
|
| msg265091 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-07 19:52 |
| There is a typo in sem_unlink_alt.diff, not in the patch itself but in the neighbouring lines. In Modules/_multiprocessing/semaphore.c, it should be, I think, after one does 's/define sem_unlink(name)/define SEM_UNLINK(name)/': @@ -194,8 +194,8 @@ # define SEM_FAILED ((sem_t *)-1) #endif -#ifndef HAVE_SEM_UNLINK -# define sem_unlink(name) 0 +#if !defined(HAVE_SEM_UNLINK) | |
defined(__ANDROID__) +# define SEM_UNLINK(name) 0 #endif #ifndef HAVE_SEM_TIMEDWAIT But applying this new patch, gives exactly the same results. Maybe it is not just sem_unlink() that is not implemented. I will run a gdb session to find out. |
|
| msg265092 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-07 20:31 |
| sem_open() is not implemented as well. Here is a gdb session with a breakpoint set at semlock_new() (SEM_FAILED is NULL on android): Breakpoint 1, semlock_new (type=0xb6d070c0 <_PyMp_SemLockType>, args=(1, 1, 1, '/mp-dkzrq4ed', True), kwds=0x0) at /home/xavier/src/packages/android/cpython/Modules/_multiprocessing/semaphore.c:420 420 { (gdb) next 428 if (!PyArg_ParseTupleAndKeywords(args, kwds, "iiisi", kwlist, (gdb) 432 if (kind != RECURSIVE_MUTEX && kind != SEMAPHORE) { (gdb) 437 if (!unlink) { (gdb) 424 char *name, *name_copy = NULL; (gdb) 445 handle = SEM_CREATE(name, value, maxvalue); (gdb) 447 if (handle == SEM_FAILED | |
SEM_GET_LAST_ERROR() != 0) (gdb) print handle $5 = (SEM_HANDLE) 0x0 (gdb) next 460 if (handle != SEM_FAILED) (gdb) 462 PyMem_Free(name_copy); (gdb) 463 _PyMp_SetError(NULL, MP_STANDARD_ERROR); (gdb) 464 return NULL; (gdb) This is run with sem_unlink_alt.diff modified to include the change to 'define SEM_UNLINK(name) 0' although I guess this does not change anything. Sorry for misleading you about sem_unlink(). |
|
| msg265093 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2016-05-07 20:59 |
| Thanks! Perhaps we should not set HAVE_SEM_* to 1 if we are on Android in the configure step. |
|
|
| msg265129 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-08 11:16 |
| Undefining the SEM_* macros in multiprocessing.h when __ANDROID__ is set, would allow for an easier update when those functions become implemented at a future API level. The __ANDROID_API__ macro can be used for that purpose. For example, if this happens at API 24, this could be a changed to: #if defined(__ANDROID__) # include <android/api-level.h> # if __ANDROID_API__ < 24 # undef sem_open # ... # endif #endif |
|
|
| msg265130 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-05-08 11:19 |
| > # undef sem_open I meant of course: # undef HAVE_SEM_OPEN |
|
|