[Python-Dev] Require loaders set package and loader (original) (raw)

Glyph glyph at twistedmatrix.com
Sun Apr 15 23:12:30 CEST 2012

On Apr 14, 2012, at 7:59 PM, Guido van Rossum wrote:

On Sat, Apr 14, 2012 at 5:06 PM, Christian Heimes <lists at cheimes.de> wrote:

Am 15.04.2012 00:56, schrieb Guido van Rossum:

Well, if it's a real file, and you need a stream, that's efficient, and if you need the data, you can read it. But if it comes from a loader, and you need a stream, you'd have to wrap it in a StringIO instance. So having two APIs, one to get a stream, and one to get the data, allows the implementation to be more optimal -- it would be bad to wrap a StringIO instance around data only so you can read the data from the stream again...

We need a third way to access a file. The two methods getdata() and getstream() aren't sufficient for libraries that need a read file that lives on the file system. In order to have real files the loader (or some other abstraction layer) needs to create a temporary directory for the current process and clean it up when the process ends. The file is saved to the temporary directory the first time it's accessed. Hm... Can you give an example of a library that needs a real file? That sounds like a poorly designed API.

Lots of C libraries use filenames or FILE*s where they should be using some much more abstract things; i.e., constellations of function pointers that are isomorphic to Python's "file-like objects". Are these APIs poorly designed? Sure, but they also exist ;).

Perhaps you're talking about APIs that take a filename instead of a stream? Maybe for those it would be best to start getting serious about a virtual filesystem... (Sorry, probably python-ideas stuff).

twisted.python.filepath... ;-)

The getfile() feature has a neat benefit. Since it transparently extracts files from the loader, users can ship binary extensions and shared libraries (dlls) in a ZIP file and use them without too much hassle. Yeah, DLLs are about the only example I can think of where even a virtual filesystem doesn't help...

In a previous life, I was frequently exposed to proprietary game-engine things that could only load resources (3D models, audio files, textures) from actual real files, and I had to do lots of unpacking stuff either from things tacked on to a .exe or inside a zip file. (I don't know how common this is any more in that world but I suspect "very".)

Unfortunately all the examples I can think of off the top of my head were in proprietary, now defunct code; but this is exactly the sort of polish that open-sourcing tends to apply, so I would guess problematic code in this regard would more often be invisible.

-glyph

More information about the Python-Dev mailing list