| msg232697 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2014-12-16 02:29 |
| Currently, the implementation of __setstate__ at Lib/multiprocessing/heap.py:54 looks like this: def __setstate__(self, state): self.size, self.name = self._state = state self.buffer = mmap.mmap(-1, self.size, tagname=self.name) assert _winapi.GetLastError() == _winapi.ERROR_ALREADY_EXISTS This assertion can fail when the assignment to buffer triggers memory [re]allocation, which could clear the last error result. So far, this fails reliably on debug builds using VS 2015 (which causes some tests to timeout when all of the child processes fail to start), but could also fail in threaded code at any time. I don't know why release builds or builds with VS 2010 did not trigger this, but I would speculate that either VS 2015 is using a different API to allocate memory or Windows 8.1 has changed the behaviour of an API. Whether a function resets the last error code is deliberately unspecified (http://msdn.microsoft.com/en-us/library/windows/desktop/ms679360(v=vs.85).aspx - "some functions set the last-error code to 0 on success and others do not"). In my opinion, the assertion should just be removed. A hack that appears to work currently is to add "self.buffer = None" before the existing assignment (to pre-expand self.__dict__ and avoid the allocation). If the assertion is important, someone may want to add a parameter to mmap() to require that the memory-mapped file already exists and throws otherwise, but I am not volunteering to do this. |
|
|
| msg232698 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2014-12-16 02:30 |
| A buildbot failure due to this can be seen here: http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/183/steps/test/logs/stdio |
|
|
| msg232728 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2014-12-16 08:50 |
| I agree that this is a fragile assertion; it's too far removed from the CreateFileMapping call which can generate it and almost impossible to work around (in calling code) if it should fail in the way we're seeing in the buildbot. I think we're better off relying on a genuine exception bubbling up from the CreateFileMapping/MapViewOfFile calls than trying to assert the no-exception error return. While the "preallocate self.buffer" hack you mention would probably have the effect of preventing the assertion, it's really just adding another layer of unwanted complexity. (And might still not work in some future memory-allocation algorithm). @Richard: if you're watching, have I missed anything? |
|
|
| msg232749 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2014-12-16 17:24 |
| Or perhaps: buffer = mmap.mmap(-1, self.size, tagname=self.name) assert _winapi.GetLastError() == _winapi.ERROR_ALREADY_EXISTS self.buffer = buffer ? |
|
|
| msg232755 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2014-12-16 18:39 |
| That was actually my first hack and it also works. The fundamental problem here is that GetLastError() is not actually meant to indicate *whether* an error has occurred, but when one has occurred it describes what it was. Indeed, in this case no error has occurred - CreateFileMapping was called and there is a file mapping available. OpenFileMapping is provided if an error is required when the mapping does not already exist. The best option is definitely a separate method (or parameter) on mmap to open/raise rather than open/create. I may just comment out the assertion with a reference to this bug in the meantime, so that the buildbots aren't getting stuck every time. |
|
|
| msg232820 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2014-12-17 14:40 |
| New changeset 1794d754ff3c by Steve Dower in branch 'default': Issue #23060: Suppresses a multiprocessing assert that fails incorrectly https://hg.python.org/cpython/rev/1794d754ff3c |
|
|
| msg232822 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2014-12-17 14:41 |
| I've commented out the assertion for now with a comment pointing to this issue, so that'll keep the buildbots running while we figure out how to deal with this properly. |
|
|