[Python-Dev] mmap module (original) (raw)

Paul Barrett Barrett@stsci.edu
Mon, 21 May 2001 08:17:09 -0400

Tim Peters wrote:

[Paul Barrett] > In the CVS log of the mmapmodule.c, Tim Peters says: > > "The code really needs to be rethought from scratch (not by me, though > ...)." That was in specific reference to the code I changed, in mmapfindmethod. The difficulty is that mmap is great for "large files", but the code before my change used a C int for the starting offset and also for the return > value; I boosted those to a C long, which covers 63 bits on 64-bit Linux > boxes, but doesn't help 64-bit Windows at all (where a C long remains 4 > bytes). The mmapobject struct uses sizet to declare the relevant members, > which is possibly better still than C long, but may still leave platform > capabilities out of reach for large files (e.g., "even Win95" allows > specifying 64-bit offsets when creating a mapped file view). C is a > friggin' mess here, and Python's PyArgParseTuple() and PyBuildValue() > don't cater to the full range of C integral types anyway. In other words, > if this code is ever to reach its full potential, it "really needs to be > rethought from scratch".

OK, thanks for the clarification.

> The ability to have offsets into a file that are not multiples of the > system pagesize would also be nice.

It's OS-specific. Python should grow warts to protect against it on the > OSes that care.

Well, hopefully the OS-differences wouldn't prevent implementing a more abstract interface.

> I'd be willing to submit a PEP on a new mmapmodule, once I know what > others would like.

Hard to say. This has the potential to become Python's next thread subsystem, i.e. an endless and ultimately hopeless x-platform nightmare. If you do write a PEP, I vote to say that we'll cover Windows and Linux (and maybe Mac OS X?) out of the box, but any other platform is at your own risk (it doesn't really help if somebody pops up volunteering to support a minority platform, because they eventually go away, their code stops > working, and it never gets fixed -- so it's use-at-your-own-risk in reality regardless).

Yes, I agree. Windows, Unix/Linux, and Mac OS X should be the supported platforms.

My intention is not to make major changes to the Python interface, but to fix bugs and to implement some additional features, such as a non-pagesize file offset. I'll try to get something written up in the near future.

-- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Group FAX: 410-338-4767 Baltimore, MD 21218