[Python-Dev] Mysterious Python pyc file corruption problems (original) (raw)

Barry Warsaw barry at python.org
Fri May 17 21:16:41 CEST 2013

On May 17, 2013, at 03:02 PM, Terry Jan Reedy wrote:

What I mean is, is the corruption (not the detection of corruption) only happening during mass compilation of the stdlib? When user imports a single non-stdlib file he has written the first time, does that ever get corrupted.

It's not limited to the stdlib, but yes, as far as we can tell, it happens during package installation on an end-user system, when the trigger script mass byte-compiles a package's Python source files.

I intended to suggest that pycompile be changed to do that. Then Brett said it already had for 3.4. I see no reason why not to backport, but maybe someone else will.

I tend to agree.

The main design use of marshal is to produce .pyc files that consist of a prefix and marshalled codeobject. Perhaps marshal.dump(s) should be extended to take a third prefix='' parameter that would be prepended to the result as produced today. .dump first does .dumps, though inline. I assume that .dumps constructs a string by starting with [], appending pieces, and joining. At least, any composite object dump would. The change would amount to starting with [prefix] instead of []. Then pycompile would amount to pyc = marshal.dump(codeobject, file, pyc)

That wouldn't be a backportable change, and right now I'm trying to stay focused on fixing this specific problem. ;)

-Barry

More information about the Python-Dev mailing list