| msg140330 - (view) |
Author: Sergei Lebedev (superbobry) * |
Date: 2011-07-14 13:20 |
| Current `mmap` implementation raises a ValueError if a sum of offset and length exceeds file size, as reported by `fstat`. While perfectly valid for most use-cases, it doesn't work for "special" files, for example: >>> with open("/proc/sys/debug/exception-trace", "r+b") as f: ... mmap.mmap(f.fileno(), 0) ... Traceback (most recent call last): File "", line 2, in ValueError: mmap offset is greater than file size Same goes for almost any other /proc file, because most of them have S_ISREG() == True and st_size = 0. How about adding a keyword argument to `mmap.mmap()`, which disables fstat-based size checks? |
|
|
| msg140447 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2011-07-15 17:34 |
| A couple remarks: - if st_size == 0, then you have to specify an explicit length (your example passes 0, which would fail with EINVAL without the check) - most enties in /proc don't support mmap file operation, so it's likely to fail anyway (with ENODEV, EACCESS...). Do you have an example of a /proc entry with st_size == 0 that can be mmapped (mapping /proc/sys/debug/exception-trace fails with EACCESS)? > How about adding a keyword argument to `mmap.mmap()`, which disables > fstat-based size checks? I don't like the idea of adding an argument which doesn't have a counterpart in the POSIX version (especially to address such corner cases). |
|
|
| msg140805 - (view) |
Author: Ross Lagerwall (rosslagerwall) |
Date: 2011-07-21 10:42 |
| > I don't like the idea of adding an argument which doesn't have a > counterpart in the POSIX version (especially to address such corner > cases). Indeed, it seems rather messy for a corner case that may well not exist. If there are definite cases where this usage is needed, a solution may then have to be considered. |
|
|
| msg140806 - (view) |
Author: Sergei Lebedev (superbobry) * |
Date: 2011-07-21 11:02 |
| > Do you have an example of a /proc entry with st_size == 0 that can be mmapped > (mapping /proc/sys/debug/exception-trace fails with EACCESS)? Yes, I've ran into the issue, while trying to mmap /proc/xen/xsd_kva, which is an interface to XenBus [1]. Unfortunately, I'm not aware of any other cases. By the way, I've checked mmap(2) manpage -- it looks like the C-version has nothing against mmaping 0-sized files, Why does Python's `mmap` still checks file size? [1] http://wiki.xensource.com/xenwiki/XenBus |
|
|
| msg140824 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2011-07-21 19:05 |
| > By the way, I've checked mmap(2) manpage -- it looks like the C-version has > nothing > against mmaping 0-sized files, Why does Python's `mmap` still checks file > size? It doesn't check explicitely that the size is non-0, but rather that the offset is is less than the file size. Passing mmap(2) a 0 length doesn't make much sense anyway - for example, Linux returns EINVAL: http://lxr.free-electrons.com/source/mm/mmap.c?a=avr32#L979 """ unsigned long do_mmap_pgoff(...) [...] if (!len) return -EINVAL; """ >> Do you have an example of a /proc entry with st_size == 0 that can be >> mmapped >> (mapping /proc/sys/debug/exception-trace fails with EACCESS)? > Yes, I've ran into the issue, while trying to mmap /proc/xen/xsd_kva, which > is an > interface to XenBus [1]. Unfortunately, I'm not aware of any other cases. That's what I thought, it's really uncommon: in that case, I'm reluctant to making such a change, for the reason explained above. Ross, Victor? |
|
|
| msg140834 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2011-07-21 23:13 |
| > That's what I thought, it's really uncommon: in that case, I'm > reluctant to making such a change, for the reason explained above. > Ross, Victor? Why do you want a mmap? Why not using a file? |
|
|
| msg140861 - (view) |
Author: Ross Lagerwall (rosslagerwall) |
Date: 2011-07-22 05:11 |
| > That's what I thought, it's really uncommon: in that case, I'm > reluctant to making such a change, for the reason explained above. > Ross, Victor? Even if Linux did accept a a length of 0 (which it doesn't since 2.6.12), what would be the point? Wouldn't you be mapping *nothing*? Yeah, I'm against this change. |
|
|
| msg140862 - (view) |
Author: Sergei Lebedev (superbobry) * |
Date: 2011-07-22 05:16 |
| > Passing mmap(2) a 0 length doesn't make much sense anyway - for example, Linux returns EINVAL. Yup, but passing mmap(2) a zero-sized file and a non-zero length works just fine. > Why do you want a mmap? Why not using a file? Because Xen requires me to mmap it. Anyway, you're right, the use-case is too uncommon. |
|
|
| msg166551 - (view) |
Author: Ricky Ng-Adam (rngadam) |
Date: 2012-07-27 10:15 |
| I find this resolution to rejected somewhat questionable. It has been pointed that the mmap behavior in Python differs from the behavior of the underlying mmap as defined by the system documentation. I think the incorrect assumption here is that mmap is used to only map real files. In the case of virtual files such those found in sysfs and debugfs in Linux, mmap is a general operation that lets the underlying driver arbitrarily map to other memory regions. As an example, we are developing a new hardware platform. To maximize performance, instead of creating a file-based interface to the Linux driver, we are experimenting with mmap'ing the hardware memory address to the userspace by implementing the mmap operation in our Linux kernel module. This mmap operation works in C and in Node.JS, but not in Python where it can't get past the underlying debugfs (or sysfs) file having a size of 0... |
|
|
| msg218335 - (view) |
Author: Josh Triplett (joshtriplett) |
Date: 2014-05-12 16:45 |
| This rejection is indeed problematic. If mmap.mmap receives an explicit size, checking that size against stat will break on devices or special files. It's perfectly reasonable that mmap.mmap's automatic logic when passed length=0 (to map the entire file) won't work if stat says the file has length 0, but the mmap syscall works fine on special files with length 0 (or devices), and mmap.mmap should work in that case as well. |
|
|
| msg218351 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2014-05-12 19:01 |
| From a cursory inspection, we only check the size for regular files. What exception are you seeing exactly? What does stat return on this file? |
|
|
| msg218353 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2014-05-12 19:09 |
| Scratch that. I'll try to work on a patch that doesn't break guarantees for regular files (we try to limit the possibilities of segfault). |
|
|