| msg172855 - (view) |
Author: Ian Carr-de Avelon (Ian) |
Date: 2012-10-14 07:27 |
| I'm new to Python and I've hit what appears to me to be a bug, but may be a "feature", so a tutorial bug. I tried to loop through the items in a list, test each and remove those which fail the test. Simplifying to illustrate: >>> print test [1, 2, 3, 4, 5] >>> for item in test: ... print item ... test.remove(item) ... 1 3 5 >>> print test [2, 4] Whereas I would expect to see all items looped through and non left in the list. I have worked with languages where you are explicitly warned that you must not mess with the loop variable, or where the behaviour you will get is explained in detail, so you can use it. Not having anything flagged up in eg 5.6. Looping Techniques on http://docs.python.org/tutorial/datastructures.html I assumed that the behaviour would be safe and intuative. Yours Ian |
|
|
| msg172856 - (view) |
Author: Chris Jerdonek (chris.jerdonek) *  |
Date: 2012-10-14 07:36 |
| > I have worked with languages where you are explicitly warned that you must not mess with the loop variable There is a warning in this part of the tutorial: "It is not safe to modify the sequence being iterated over in the loop..." (from http://docs.python.org/dev/tutorial/controlflow.html#for-statements ) But it may be good to add a note to the section you reference as well. |
|
|
| msg172861 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 09:06 |
| Attached is a simple way of addressing this (essentially copying the verbiage and example from the other page). If we want, we could make the sample code different so that the reader doesn't see the same thing twice. |
|
|
| msg172875 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2012-10-14 11:24 |
| It is safe to modify a sequence during iteration if it's size not increased. >>> words = ['cat', 'window', 'defenestrate'] >>> for i, w in enumerate(words): ... if len(w) > 6: ... words[i] = w[:5] + '…' ... >>> words ['cat', 'window', 'defen…'] |
|
|
| msg172897 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 18:14 |
| > It is safe to modify a sequence during iteration if it's size not increased. What do you mean by "safe"? The example given by the original commenter does not increase the size either. I believe "safe" is meant in the sense of avoiding possibly unexpected behavior. I can clarify the language if you feel it is ambiguous. |
|
|
| msg172899 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2012-10-14 18:31 |
| Well, I guess Serhiy meant "neither increase nor decrease". In the end, the exact behavior will never be clear if you don't state explicitly how it's implemented: a counter that starts at 0 and is increased on __next__ until it's equal to len(list) (as checked at every iteration step). But in general the advice should be: if you want to insert or remove elements during iteration, iterate over a copy. |
|
|
| msg172906 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 19:38 |
| > But in general the advice should be: if you want to insert or remove elements during iteration, iterate over a copy. I would expand this to cover changing the list in any way. I think the point being made is that iteration doesn't implicitly make a copy. There are cases where modifying the list in place can also yield unexpected results. For example (naive list reversal): >>> words = ['cat', 'window', 'defenestrate'] >>> for i, word in enumerate(words): ... words[-i-1] = word ... >>> words ['cat', 'window', 'cat'] |
|
|
| msg172926 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 21:56 |
| Attaching revised patch. |
|
|
| msg172930 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 22:10 |
| Reattaching. I duplicated a variable definition that was defined previously. |
|
|
| msg172932 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2012-10-14 22:22 |
| I mean it does not lead to crash, hang, etc. Even growing list during iteration can be safe you move forward faster than list grows or if you known where to stop. Unexpected behavior for one people can be expected for others. |
|
|
| msg172933 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 22:34 |
| > I mean it does not lead to crash, hang, etc. I agree. I removed the word "safe" in the patch I attached to reduce ambiguity. Regarding unexpected behavior, remember that the tutorial is for beginners/newcomers. |
|
|
| msg172934 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2012-10-14 23:00 |
| > I agree. I removed the word "safe" in the patch I attached to reduce > ambiguity. Yes, so much the better. It will be nice somewhere in deep clarify for experts what happens with list iterator if the list changed. And iterating over modifyed (if you insert/remove keys, but not when you only update values) dict or set really yield unexpected results (i.e. they expected in them undefinability). This is really not recommended. |
|
|
| msg172935 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-14 23:04 |
| > It will be nice somewhere in deep clarify for experts what happens with list iterator if the list changed. There is a note somewhat to this effect here: http://docs.python.org/dev/reference/compound_stmts.html#the-for-statement "Note There is a subtlety when the sequence is being modified by the loop...." |
|
|
| msg173007 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2012-10-16 01:24 |
| > > It will be nice somewhere in deep clarify > > for experts what happens with list iterator if the list changed. Resist the urge to over-specify. Unless the behavior is tested and known to be consistent across all implementations, I'm content with the current docs: "Note There is a subtlety when the sequence is being modified by the loop...." Chris's currently patch seems reasonable to me and I don't think anything further is a good idea. With nearly any data structure in any language, most programmers learn to be cautious about and generally avoid looping over a structure while mutating it. Adding yet more documentation details won't make the issue go away. I recommend posting the current patch and closing this issue to be done with it. |
|
|
| msg173008 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-16 01:30 |
| Thanks, Raymond. I will be doing that later today. |
|
|
| msg173011 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-10-16 02:51 |
| New changeset 1f1bf6a3abbc by Chris Jerdonek in branch '3.2': Issue #16225: Add additional note to tutorial about changing sequence while looping. http://hg.python.org/cpython/rev/1f1bf6a3abbc New changeset 8cb14494d33c by Chris Jerdonek in branch '3.3': Issue #16225: Merge from 3.2: Add additional note to tutorial about looping. http://hg.python.org/cpython/rev/8cb14494d33c New changeset e0a407d41af5 by Chris Jerdonek in branch 'default': Issue #16225: Merge from 3.3: Add additional note to tutorial about looping. http://hg.python.org/cpython/rev/e0a407d41af5 |
|
|
| msg173012 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2012-10-16 03:02 |
| New changeset dc006b6212e7 by Chris Jerdonek in branch '2.7': Issue #16225: Backport from 3.2: Add additional note to tutorial about looping. http://hg.python.org/cpython/rev/dc006b6212e7 |
|
|
| msg173013 - (view) |
Author: Chris Jerdonek (chris.jerdonek) * |
Date: 2012-10-16 03:04 |
| Committed. Thanks for reporting the suggestion, Ian. |
|
|