| msg101380 - (view) |
Author: (news1234) |
Date: 2010-03-20 17:08 |
| Following code snippet will behave differently on Linux and windows hosts. Under linux the script can only be run once. The second call will raise an exception, as the previous program is already listening to pot 8089. Under Windows however the program can be started twice. and will print twice "serving". This surprises me The script: # ########################## import socket,sys from multiprocessing.managers import BaseManager mngr = BaseManager(address=('127.0.0.1',8089),authkey='verysecret') try: srvr = mngr.get_server() except socket.error as e: print "probably address already in use" sys.exit() print "serving" srvr.serve_forever() Perhaps the reason for the problem might be related to http://bugs.python.org/issue2550 I'd suggest to fix multiprocessing.managers.BaseManager such, that it behaves identially on both platforms. |
|
|
| msg112946 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2010-08-05 04:10 |
| Ran code snippet on Windows Vista with 3.1.2 and it ran fine. I'll close this unless anyone objects. |
|
|
| msg112971 - (view) |
Author: Jesse Noller (jnoller) *  |
Date: 2010-08-05 11:15 |
| Mark - did you observe the behavior in the Op? He's not stating the the code snippet runs fine, but that the second run at the same time on windows to cause a conflict. We need to show that either running it twice, at the same time against the same socket fails on windows as it should or we still have a bug |
|
|
| msg150485 - (view) |
Author: Phill (gearb0x) |
Date: 2012-01-03 00:40 |
| I have run into the same problem with python 3.2 & 2.7 on windows 7 with the Listener object. It opens the same port twice for some wierd reason I have tried the example in this bug report and i am getting the same behavoir as the original poster |
|
|
| msg150486 - (view) |
Author: Phill (gearb0x) |
Date: 2012-01-03 00:41 |
| Im not sure whether to open a new bug report for my issue or just leave it here. |
|
|
| msg150511 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-01-03 16:40 |
| That's because SocketListener uses SO_REUSEADDR. It seems that, with SO_REUSEADDR, Windows allows binding to a port even though there's a socket already bound to the same port in the LISTEN state: this is wrong, the semantics of SO_REUSEADDR was intended for sockets in TIME-WAIT state, and Linux and BSD systems implement this properly (i.e. fail with EADDRINUSE when there's a socket in LISTEN state). The problem, if we remove this flag, is that managers binding to a specific port will get EADDRINUSE in case of rapid restart. Since I'm not convinced that this is really an issue, I'd suggest to close this as "won't fix". Another option would be to complain to Microsoft :-) @Phill: I'm not sure I understand your problem: could you be more specific (or open a new issue)? |
|
|
| msg150516 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-01-03 17:04 |
| There's a length MSDN article about this: http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621%28v=vs.85%29.aspx Executive summary: it's a can of worms. However, let me point out the following sentence: “Ports without SO_EXCLUSIVEADDRUSE set may be reused as soon as the socket on which bind was previously called is closed.” ...which seems to suggest we shouldn't use SO_REUSEADDR under Windows, since Windows sockets appear to have the Unix SO_REUSEADDR semantics by default. |
|
|
| msg150519 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-01-03 17:36 |
| OK, so just removing SO_REUSEADDR on Windows should do the trick... Seriously, why can't they simply conform to existing standards :-( If someone wants to provide a patch + test, go ahead! |
|
|
| msg150524 - (view) |
Author: Phill (gearb0x) |
Date: 2012-01-03 20:17 |
| @neologix: nah its fine, if you guys are gonna re open this one I wont worry about opening a new bug. If the above gets solved on windows my problem will just go away, thanks |
|
|
| msg150675 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-01-05 17:49 |
| > If the above gets solved on windows my problem will just go away, thanks Would you like to propose a patch with test? |
|
|
| msg150700 - (view) |
Author: Phill (gearb0x) |
Date: 2012-01-06 00:19 |
| Normally I would be happy to but my combined python experience is about 30 minutes so I am probably not the man |
|
|
| msg150704 - (view) |
Author: Phill (gearb0x) |
Date: 2012-01-06 00:48 |
| I have commented out the line: self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) In lib/multiprocessing/connection.py as a test and it works fine, the problem still persists for named pipes (im not sure if thats how named pipes are supposed to behave though) |
|
|
| msg150705 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-01-06 01:06 |
| > im not sure if thats how named pipes are supposed to behave though I'm not sure what you mean. Are you creating named pipes yourself? |
|
|
| msg150709 - (view) |
Author: Phill (gearb0x) |
Date: 2012-01-06 02:18 |
| Rather than listening on a socket, listening on a named pipe eg: address = (r'\\.\pipe\Test', 'AF_PIPE') listener = Listener(*address) conn = listener.accept() It doesnt raise an exception when i run the script again a second time. Like I said, I dont know much about named pipes and im not even sure thats how they are intended to work in this context. IE: if one process is listening, can another listen on that named pipe as well? Phill |
|
|
| msg150716 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-01-06 08:19 |
| > Like I said, I dont know much about named pipes and im not even sure thats how they are intended to work in this context. IE: if one process is listening, can another listen on that named pipe as well? Under Unix, you'd get a EADDRINUSE with a Unix domain socket. I don't know much about Windows, but here's what CreateNamedPipe doc says: """ FILE_FLAG_FIRST_PIPE_INSTANCE 0x00080000 If you attempt to create multiple instances of a pipe with this flag, creation of the first instance succeeds, but creation of the next instance fails with ERROR_ACCESS_DENIED. Windows 2000: This flag is not supported until Windows 2000 SP2 and Windows XP. """ So it seems that we should probably pass FILE_FLAG_FIRST_PIPE_INSTANCE under Windows. |
|
|
| msg150723 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-01-06 12:30 |
| Le vendredi 06 janvier 2012 à 02:18 +0000, Phill a écrit : > Phill <beerb0x@gmail.com> added the comment: > > Rather than listening on a socket, listening on a named pipe > > eg: > address = (r'\\.\pipe\Test', 'AF_PIPE') > listener = Listener(*address) > conn = listener.accept() > > It doesnt raise an exception when i run the script again a second time. According to MSDN, this is normal: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365150%28v=vs.85%29.aspx However, there's also a flag named FILE_FLAG_FIRST_PIPE_INSTANCE that we could use in PipeListener, which would then raise an error if a listener pipe was created a second time. It would probably make more sense, although I don't know whether some programs may rely on creating a pipe multiple times. |
|
|
| msg150730 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-01-06 13:19 |
| (oops, hadn't seen Charles-François's answer before replying) |
|
|
| msg150812 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-01-07 18:49 |
| I noticed that if bind() fails (in this case with EADDRINUSE), the socket isn't closed (FD leak). Here's a patch. |
|
|
| msg150900 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-01-08 19:42 |
| Here's a patch addressing the multiple bind() problem on Windows. Note that this problem also affects other parts of the stdlib, which use SO_REUSEADDR when available. Also, there's an rather confusing comment in support.find_unused_port(): """ (This is easy to reproduce on Windows, unfortunately, and can be traced to the SO_REUSEADDR socket option having different semantics on Windows versus Unix/Linux. On Unix, you can't have two AF_INET SOCK_STREAM sockets bind, listen and then accept connections on identical host/ports. An EADDRINUSE socket.error will be raised at some point (depending on the platform and the order bind and listen were called on each socket). However, on Windows, if SO_REUSEADDR is set on the sockets, no EADDRINUSE will ever be raised when attempting to bind two identical host/ports. When accept() is called on each socket, the second caller's process will steal the port from the first caller, leaving them both in an awkwardly wedged state where they'll no longer respond to any signals or graceful kills, and must be forcibly killed via OpenProcess()/TerminateProcess(). The solution on Windows is to use the SO_EXCLUSIVEADDRUSE socket option instead of SO_REUSEADDR, which effectively affords the same semantics as SO_REUSEADDR on Unix. Given the propensity of Unix developers in the Open Source world compared to Windows ones, this is a common mistake. A quick look over OpenSSL's 0.9.8g source shows that they use SO_REUSEADDR when openssl.exe is called with the 's_server' option, for example. See http://bugs.python.org/issue2550 for more info. The following site also has a very thorough description about the implications of both REUSEADDR and EXCLUSIVEADDRUSE on Windows: http://msdn2.microsoft.com/en-us/library/ms740621(VS.85).aspx) """ I have no idea why it uses SO_REUSEADDR + SO_EXCLUSIVEADDRUSE instead of just keeping the default setting, since we don't want to allow multiple bind(), but bypass the TIME-WAIT lingering. |
|
|
| msg151079 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-01-11 19:10 |
| > I noticed that if bind() fails (in this case with EADDRINUSE), the > socket isn't closed (FD leak). Well it would probably be closed when the connection object is destroyed, but the patch looks ok anyway. |
|
|
| msg152617 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-02-04 12:58 |
| > Well it would probably be closed when the connection object is > destroyed, but the patch looks ok anyway. Let's be nice with those non refcount-based implementations out there :-) What do you think of the patch attached (connection_multiple_bind-1.diff) for the original problem? It does two things: 1. add the FILE_FLAG_FIRST_PIPE_INSTANCE when creating a pipe on Windows 2. remove SO_REUSEADDR altogether on Windows, since it doesn't seem to be required For 2, it should be correct if I understand Microsoft's documentation correctly, but then I wonder why test.support goes through the pain of using SO_REUSEADDR + SO_EXCLUSIVEADDRUSE instead of just dropping SO_REUSEADDR. Do you happen to know a Windows guru (or maybe should I ask on python-dev?). |
|
|
| msg152622 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-02-04 14:13 |
| New changeset 887d0ab5fb97 by Charles-François Natali in branch '2.7': Issue #8184: Fix a potential file descriptor leak when a http://hg.python.org/cpython/rev/887d0ab5fb97 New changeset ba1e0a1ac5b7 by Charles-François Natali in branch '3.2': Issue #8184: Fix a potential file descriptor leak when a http://hg.python.org/cpython/rev/ba1e0a1ac5b7 New changeset cca40a0ecffa by Charles-François Natali in branch 'default': Issue #8184: Fix a potential file descriptor leak when a http://hg.python.org/cpython/rev/cca40a0ecffa |
|
|
| msg152696 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-02-05 16:55 |
| > However, let me point out the following sentence: > “Ports without SO_EXCLUSIVEADDRUSE set may be reused as soon as the socket on which bind was previously called is closed.” > > ...which seems to suggest we shouldn't use SO_REUSEADDR under Windows, since Windows sockets appear to have the Unix SO_REUSEADDR semantics by default. Actually, it seems that even though the documentation doesn't mention sockets in TIME_WAIT state, SO_REUSEADDR is actually required on Windows: http://twistedmatrix.com/trac/ticket/1151#comment:18 So the proper solution is the one adopted by support.bind_port(), SO_REUSEADDR + SO_EXCLUSIVEADDRUSE. Since Windows semantics is different (I'd rather say broken) and SO_REUSEADDR is currently mis-used throughout the stdlib, what do you think of adding a set_reuse() method to socket that would do the right thing? |
|
|
| msg152697 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-02-05 17:16 |
| > Actually, it seems that even though the documentation doesn't mention > sockets in TIME_WAIT state, SO_REUSEADDR is actually required on > Windows: > http://twistedmatrix.com/trac/ticket/1151#comment:18 According to that message, we would only need (or desire) SO_EXCLUSIVEADDRUSE. Note however what the MSDN doc says: “Conversely, a socket with the SO_EXCLUSIVEADDRUSE set cannot necessarily be reused immediately after socket closure. For example, if a listening socket with SO_EXCLUSIVEADDRUSE set accepts a connection and is then subsequently closed, another socket (also with SO_EXCLUSIVEADDRUSE) cannot bind to the same port as the first socket until the original connection becomes inactive.” ... and goes on to discuss the virtues of shutdown() etc. So I still think we need no option at all under Windows, and SO_EXCLUSIVEADDR actually prevents the behaviour we are trying to avoid. > So the proper solution is the one adopted by support.bind_port(), > SO_REUSEADDR + SO_EXCLUSIVEADDRUSE. I think you read it wrong. bind_port() only uses SO_EXCLUSIVEADDRUSE and forbids SO_REUSEADDR. Not surprising considering the commentor above, Trent, is the same that wrote the bind_port() code :) |
|
|
| msg152698 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-02-05 17:56 |
| > I think you read it wrong. Duh, I managed to misread both the comment and the code :-) What my subconscious refused to admit is the fact that on Windows, SO_REUSEADDR allows you to bind to any port - even though the other application didn't set SO_REUSEADDR on its socket - which is a security nightmare. Anyway, in that case, my last patch should be correct (tested on one of the Win7 buildbots). |
|
|
| msg152770 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-02-06 20:27 |
| > Anyway, in that case, my last patch should be correct (tested on > one of the Win7 buildbots). It looks good to me. |
|
|
| msg152905 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2012-02-08 20:16 |
| New changeset 434301d9f664 by Charles-François Natali in branch 'default': Issue #8184: multiprocessing: On Windows, don't set SO_REUSEADDR on Connection http://hg.python.org/cpython/rev/434301d9f664 |
|
|
| msg152973 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-02-09 18:24 |
| Committed to default. I won't backport it to other branches, since it's more of a feature request than a bug fix. |
|
|