[Python-Dev] Mysterious Python pyc file corruption problems (original) (raw)

Barry Warsaw barry at python.org
Thu May 16 17:40:02 CEST 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

On May 15, 2013, at 06:06 PM, Tres Seaver wrote:

On 05/15/2013 04:58 PM, Barry Warsaw wrote:

This leads me to hypothesize that the bug is due to an as yet unidentified race condition during installation of Python source code on Ubuntu, which is normally when we automatically byte compile the source to .pyc files. Any chance you are using 'detox' or the equivalent to run tests on mutliple interpreters in parallel? The only "bad marshall data" errors I have seen lately seemed to be provoked by that kind of practice.

Nope. PyPI's detox isn't even available in Ubuntu currently. (The detox package in Ubuntu is something else.)

Tests should only be run at package build time, not installation time, and the byte compiling of source files at installation time should be single threaded and single process.

We've since found a few cases where Python 3.3 pyc files are probably corrupted, so that shoots down my theory about a race condition on reading/writing pyc files, since 3.3 implements atomic-rename and should be immune to that kind of thing.

It's still a mystery though.

iQIcBAEBCAAGBQJRlP3SAAoJEBJutWOnSwa/Xy8QAI6Ul/s0nF+rac8nGy6fieLB FHHEfmjIgj6MkyUUw/zcbR48ELiOPkkV+GM4HJnY/H2ZG9vZqwuQYWFI0cgIBmxj EmHfKK7OTdaaHZeNTt83RDwGRLUS/gYXQ7JVikGyFnSbftfmUoN/y/ndlX5DX1hT ecDHVtXCH/ti/kcOWe2OlMABONZQPW0qYB7/0PiCCmaOxulqUsz20Ofy8SfWmSPd Rbig5i8fSnI98dkLVUzyy1tbUkdRkLBro/hawu1V9y7qVkoYx1Jz6p8XkQLp3jES m22m+6CLrnD39HxvJGGNkIaYmu5xTW+rK/Li8OrfOKx6QVIZ+XQRJFkiXnKmiezk sMYv/psySWJ/BSImsQOSt/sLHJWAGh8fkMIBpx9tI3BWMvyMkI0Hs9l7JyQn0moo oSTNb9AbgRSrkh0rVv4fhOhd1Ir3LXYTGwwYE5+o7tB/Pp0AKi2tX/XTBctDpy86 xqNHOaCV0hRA2Y+/C2QAAA7LRruP0yv10DfkciVUHR7UzbXgViICEEUizGmnkni7 utGg9EDk5VcSeg2ySxhX9Uj3E2M/ijOuYpXUJ2Gwd4UNUT5XGK1+6i2JTO2pEQOM HqhsGqk4WJsfEBTIrAt4NSxZyEuQ2nRV3MIsNaVCDp1FDySZWt3Cckq8hkZ/6vOM 7ncE6aG1cJgq4WKErvCk =BMFW -----END PGP SIGNATURE-----

More information about the Python-Dev mailing list