Issue 13324: fcntl module doesn't support F_NOCACHE (OS X specific) results in high 'inactive' memory performance issues (original) (raw)
ISSUE DESCRIPTION:
In 2.6.7, 2.7.2 and also HEAD (r 73301) the fcntl module does not support the F_NOCACHE OSX specific mode.
The affect of this is that the default behaviour (data caching enabled) is used when parsing files etc. When data caching is enabled, OSX caches the data parsed from a file by python, keeping it available as 'inactive memory' even when it has been freed by python. In theory, this should be fine as OSX should recycle the inactive memory as it is required.
Unfortunately, at least under OSX 10.6.8 (and it seems 10.7) the system will do almost anything to avoid recycling inactive memory, including swallowing up all 'pure' free memory and then paging manically to the disk. The net affect of this is significantly degraded system performance.
For most users, operating with relatively small files and a large quantity of RAM this issue is probably not that obvious. However, for various reasons I'm working with files of 5-125+GB and thus it rapidly becomes a major issue.
FIX
Very simply, all the attached patch does it add support for F_NOCACHE to fcntl just like F_FULLFSYNC (another fcntl OSX specific flag) - it's a trivial change that allows you to control whether OSX uses data caching on a file handle, e.g, the following turns OFF data caching for the specified file:
fcntl.fcntl(theFile.fileno(), fcntl.F_NOCACHE, 1)
With this patch in place the inactive memory on my system stays (low) and flat, without it it rises until it maxes out all available memory and then starts paging.
With this patch in place the inactive memory on my system stays (low) and flat, without it it rises until it maxes out all available memory and then starts paging.
It's often a desired behaviour: paging out unused memory makes room for a bigger page cache, which yields better performance. Also, it's a simple heuristic to know which pages are actually in use. Linux has a systcl that can be used to tune the swap tendency (vm.swappiness). Anyway, since it seems to improve performance under some workloads and it's just a constant, I've applied your patch. Thanks!