| msg184000 - (view) |
Author: Terry J. Reedy (terry.reedy) *  |
Date: 2013-03-12 00:04 |
| test_multiprocessing is giving multiple errors like Process Process-21: Traceback (most recent call last): File "F:\Python\dev\py27\lib\multiprocessing\process.py", line 258, in _bootstrap self.run() File "F:\Python\dev\py27\lib\multiprocessing\process.py", line 114, in run self._target(*self._args, **self._kwargs) File "F:\Python\dev\py27\lib\test\test_multiprocessing.py", line 1703, in _test conn.send('hello') IOError: [Errno 10038] An operation was attempted on something that is not a socket ... File "F:\Python\dev\py27\lib\test\test_multiprocessing.py", line 1419, in _putter manager.connect() File "F:\Python\dev\py27\lib\multiprocessing\managers.py", line 500, in connect conn = Client(self._address, authkey=self._authkey) File "F:\Python\dev\py27\lib\multiprocessing\connection.py", line 465, in XmlClient return ConnectionWrapper(Client(*args, **kwds), _xml_dumps, _xml_loads) File "F:\Python\dev\py27\lib\multiprocessing\connection.py", line 175, in Client answer_challenge(c, authkey) File "F:\Python\dev\py27\lib\multiprocessing\connection.py", line 420, in answer_challenge message = connection.recv_bytes(256) # reject large message IOError: [Errno 10038] An operation was attempted on something that is not a socket After a few of these, it hangs and I have to either kill multiple processes with TaskManager (or maybe kill-python would work) or kill the console. All tracebacks seem to one of these two. The common factor is 'not a socket'. It seems that something in the test should be skipped on windows. I got similar results running test_multiprocessing by itself. The process numbers are haphazard. |
|
|
| msg184001 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 00:26 |
| For 3.3 and 3.4, test went on for several minutes putting out [###### refs] lines but finally said '1 test OK'. I tried 3.2 again and it never outputs refs lines but fairly soon gives errors. |
|
|
| msg184013 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 06:49 |
| running kill-python results in Warning -- threading._dangling was modified by test_multiprocessing Warning -- multiprocessing.process._dangling was modified by test_multiprocessing test test_multiprocessing failed -- multiple errors occurred; run in verbose mode for details -v added the attached details, and the test finished, though failed. |
|
|
| msg184018 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 11:31 |
| Does this happen every time you run the tests? (I don't see these errors.) |
|
|
| msg184030 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 16:31 |
| All 4 or 5 times I tried on 3.2, yes. |
|
|
| msg184035 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 17:09 |
| Could you try the following program: import socket import multiprocessing import multiprocessing.reduction import multiprocessing.connection def socketpair(): with socket.socket() as l: l.bind(('localhost', 0)) l.listen(1) s = socket.socket() s.connect(l.getsockname()) a, _ = l.accept() return s, a def bar(s): print(s) s.sendall(b'from bar') if __name__ == '__main__': a, b = socketpair() p = multiprocessing.Process(target=bar, args=(b,)) p.start() b.close() print(a.recv(100)) |
|
|
| msg184037 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 17:32 |
| In Command Prompt, 3.2 gave same error as before, 3.3 a different error. multi-test.txt has full tracebacks. |
|
|
| msg184039 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 18:41 |
| Now could you try the attached file? (It will not work on 2.7 because a missing socket.fromfd().) P.S. It looks like the error for 3.3 is associated with a file f:\python\mypy\traceback.py which presumably clashes with the one in the standard library. |
|
|
| msg184046 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 19:33 |
| Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before, both with installed python and yesterdays debug builds. |
|
|
| msg184050 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 21:24 |
| > Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before, > both with installed python and yesterdays debug builds. It looks like on your machine socket handles are not correctly inherited by child processes -- I had assumed that they always would be. I suppose to fix things for 3.2 and earlier it would be necessary to backport the functionality of socket.socket.share() and socket.fromshare() from 3.3. |
|
|
| msg184054 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 22:16 |
| My original report was for 32 bit debug build on 64 bit Win 7 machine. I just re-ran test_multiprocessing with installed 64 bit python with same result. Was "I don't see these errors." on different Windows or non-Windows. One option is to skip the failing sub-tests on Windows, like some other sub-tests: skipped 'does not work with windows sockets' and consider the limitation on use of multi-processing in 2.7,3.2 as "won't fix". (My view is that 3.2 users should upgrade as soon as dependencies allow.) Backporting new features requires pydev discussion. |
|
|
| msg184055 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 22:24 |
| > My original report was for 32 bit debug build on 64 bit Win 7 machine. > I just re-ran test_multiprocessing with installed 64 bit python with same > result. Was "I don't see these errors." on different Windows or non-Windows. On 64-bit Windows 7 with both 32 and 64 bit builds. > One option is to skip the failing sub-tests on Windows, like some other sub-tests: > skipped 'does not work with windows sockets' > and consider the limitation on use of multi-processing in 2.7,3.2 as "won't fix". > (My view is that 3.2 users should upgrade as soon as dependencies allow.) Yes, I would be inclined to do that. > Backporting new features requires pydev discussion. I only meant exposing that functionality in the private _multiprocessing extension. |
|
|
| msg184059 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 23:27 |
| Same OS, different hardware, drivers, and buried settings, surprisingly different behavior. Oh, well. I will test any changes you decide on that need testing. |
|
|
| msg221683 - (view) |
Author: brian yardy (brianyardy) |
Date: 2014-06-27 12:10 |
| All 4 or 5 times I tried on 3.2, yes. In Command Prompt, 3.2 gave same error as before, 3.3 a different error. multi-test.txt has full tracebacks.'http://www.einstantloan.co.uk/' |
|
|
| msg221695 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2014-06-27 17:21 |
| This is no longer a 3.x issue. 3.2 and 3.3 get security fixes only. For 3.4, test_multiprocessing is split into 4 files and all run in multiple tries. Test_multiprocessing_spawn takes a minute, but it does 264 + 20 skipped tests, including a few 'wait' tests. Three tries with 2.7 also passed. |
|
|