[Python-Dev] Treating tokenize.Untokenizer as private (original) (raw)

Terry Reedy tjreedy at udel.edu
Mon Mar 31 21:45:58 CEST 2014

On 3/31/2014 2:30 PM, Eric Snow wrote:

On Tue, Feb 18, 2014 at 2:09 PM, Terry Reedy <tjreedy at udel.edu> wrote:

I am working through the multiple bugs afflicting tokenize.untokenize, which is described in the tokenize doc and has an even longer docstring. While the function could be implemented as one 70-line function, it happens to be implemented as a 4-line wrapper for a completely undocumented (Untokenizer class with 4 methods. (It is unmentioned in the doc and there are currently no docstrings.)

I view the class as a private implementation detail and would like to treat it as such, and perhaps even rename it Untokenizer to make that clear. The issue arises in #9974. It appears that a fix may require the addition of an instance attribute or .addwhitespace parameter. If there is objection to treating the whole class as private, I would at least like to treat addwhitespace as the private helper that it is. There is no reason to call it directly except for testing. Otherwise, it could just as well have been left inline at the one call site. Is this still an open question, Terry?

It is not currently for #9974 because after consideration of two proposed patches (one part of another issue), I decided that the conditions being guarded against either could not occur or could be checked in the caller, so that no api change was needed.

However, I believe the fix for another bug, http://bugs.python.org/issue20387 will require copying the code that correctly formats indents in the old compat method to the newer method, which incorrectly assumes that indents are spaces only. I might end up wishing I could refactor the code. So this may well become a live issue again, and I would still like to know what people think.

How do we do code searches (as for use of "Untokenize") these days?

-- Terry Jan Reedy

More information about the Python-Dev mailing list