bpo-33078 - FIX queue size on pickling error by tomMoral · Pull Request #6119 · python/cpython (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation10 Commits4 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
The Queue._feed does not properly handle the size of the Queue on errors. This can lead to a situation where the Queue is considered as Full when it is empty. Here is a reproducing script:
import multiprocessing as mp
q = mp.Queue(1)
class FailPickle():
def __reduce__(self):
raise ValueError()
q.put(FailPickle())
print("Queue is full:", q.full())
q.put(0)
print(f"Got result: {q.get()}")
This PR fixes this behavior.
https://bugs.python.org/issue33078
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix! A few comments below.
| @@ -255,6 +255,7 @@ def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe, |
|---|
| info('error in queue thread: %s', e) |
| return |
| else: |
| queue_sem.release() |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add a comment explaining why you need to adjust the semaphore's value?
| q.put(NotSerializable()) |
|---|
| q.put(True) |
| # bpo-30595: use a timeout of 1 second for slow buildbots |
| self.assertTrue(q.get(timeout=1.0)) |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should you add a check for q.qsize() as well?
| @@ -0,0 +1,2 @@ |
|---|
| Fix the size handeling in multiprocessing.Queue when a pickling error |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo: "handling".
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.
Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.
| q = self.Queue(maxsize=1) |
|---|
| q.put(NotSerializable()) |
| q.put(True) |
| self.assertEqual(q.qsize(), 1) |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this check correct? Depending on when the worker thread triggers, it might return 2, no?
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is correct because maxsize=1 so the second q.put should return only once the failure has been triggered.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good point, thank you.
Thanks @tomMoral for the PR, and @pitrou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request
(cherry picked from commit e2f33ad)
Co-authored-by: Thomas Moreau thomas.moreau.2010@gmail.com
pitrou pushed a commit that referenced this pull request
(cherry picked from commit e2f33ad)
Co-authored-by: Thomas Moreau thomas.moreau.2010@gmail.com
jo2y pushed a commit to jo2y/cpython that referenced this pull request
