[Python-Dev] Mysterious Python pyc file corruption problems (original) (raw)

Thomas Wouters thomas at python.org
Fri May 17 00:10:13 CEST 2013

On Thu, May 16, 2013 at 11:19 PM, Guido van Rossum <guido at python.org> wrote:

This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background.

When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Even pycompile.py (used by compileall.py) uses this strategy. When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves three processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with "unknown type code".

The 'unknown type codes' can also be explained if the two processes writing to the .pyc files are different Python versions. As you may recall, at Google we used to use modified Python interpreters that used '.pyc-2.2', '.pyc-2.4', etc, for the pyc extension. That was because otherwise different Python versions would keep overwriting the .pyc files of shared Python modules, and "at Google scale" it caused all manner of problems... I guess Ubuntu is approaching Google scale ;-)

(The decision to rename to an awkward extension broke a lot of third-party tools; it was made before I -- or you, for that matter -- joined Google... Now we just turn on -B by default :)

However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling rstring()), so I think all the errors are actually explainable from this scenario.

-- --Guido van Rossum (python.org/~guido)

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thomas%40python.org

-- Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130517/9d419a4e/attachment.html>

More information about the Python-Dev mailing list