[Python-Dev] Questionable AST wibbles (original) (raw)

Jeremy Hylton jeremy at alum.mit.edu
Fri Oct 21 17:13:47 CEST 2005

On 10/21/05, Neal Norwitz <nnorwitz at gmail.com> wrote:

There are a bunch of mods from the AST branch that got integrated into head. Hopefully, by doing this on python-dev more people will get involved. I'll describe high level things first, but there will be a ton of details later on. If people don't want to see this crap on python-dev, I can take this offline.

Thanks for all the notes and questions, Neal. There were a lot of changes made over a long time, and it's good to discuss some of them.

Highlevel overview of code size (rough decrease of 300 C source lines): * Python/compile.c -2733 (was 6822 now 4089) * Python/Python-ast.c +2281 (new file) * Python/asdl.c +92 (new file) * plus other minor mods

symtable.h has lots of changes to structs and APIs. Not sure what needs to be doc'ed.

The old symtable wasn't well documented and the API it exposed to Python programmers was lousy. We need to figure out a good Python API and document it.

I was very glad to see that ./python compileall.py Lib took virtually the same time before and after AST. Yeah! Unfortunately, I can't say the same for memory usage for running compileall:

Before AST: [10120 refs] After AST: [916096 refs]

That's great news! That is, I expected it to be a lot slower to compile and didn't have any particulary good ideas about how to speed it up. I expected there to be a lot of memory bloat and think we can fix that without undue effort :-).

A bunch of APIs changed and there is some additional name pollution. Since these are pretty internal APIs, I'm not sure that part is a big deal. I will try to find more name pollution and eliminate it by prefixing with Py.

Right. The code isn't binary compatible with Python 2.4 right now, but given the APIs that changed I wasn't too concerned about that. I'm not sure who should make the final decision there.

One API change which I think was a mistake was PyMangle() losing 2 parameters (I think this was how it was a long time ago). See typeobject.c, Python.h, compile.c.

I don't mind this one since it's an _Py function. I don't think code outside the core should use it.

pythonrun.h has a bunch of changes. I think a lot of the APIs changed, but there might be backwards compatible macros. I'm not sure. I need to review closely.

We should double-check. I tried to get rid of the nest of different functions that call each other by replacing the old ones with macros that call the newest ones (the functions that take the most arguments). It's not really a related change, except that it seemed like cleanup of compiler-related code. Also, a bunch of functions started taking const char* instead of char*. I think that's a net win, too.

code.h was added, but it mostly contains stuff from compile.h. Should we remove code.h and just put everything in compile.h? This will remove lots little changes. code.h & compile.h are tightly coupled. If we keep them separate, I would like to see some other changes.

I would like to keep them separate. The compiler produces code objects, but consumers of code objects don't need to know anything about the compiler. You did remind me that I intended to remove the #include "compile.h" lines from a bunch of files that merely consume code objects. What other changes would you like to see?

This probably is not a big deal, but I was surprised by this change:

+++ testrepr.py 20 Oct 2005 19:59:24 -0000 1.20 @@ -123,7 +123,7 @@ def testlambda(self): self.failUnless(repr(lambda x: x).startswith( - "<function <lambda")) + "<function lambda")) This one may be only marginally worse (names w/parameter unpacking): testgrammar.py - verify(f4.funccode.covarnames == ('two', '.2', 'compound', - 'argument', 'list')) + vereq(f4.funccode.covarnames, + ('two', '.1', 'compound', 'argument', 'list')) There are still more things I need to review. These were the biggest issues I found. I don't think most are that big of a deal, just wanted to point stuff out.

I don't have a strong sense for how important these changes are. I don't think the old behavior was documented, but I can imagine some code depending on these implementation details.

Jeremy

More information about the Python-Dev mailing list