bug#14752: [PATCH] sort: print warning when fork() failed for --compress (original) (raw)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From: Pádraig Brady
Subject: bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program
Date: Wed, 28 May 2014 16:15:51 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2

On 05/26/2014 10:10 PM, Pádraig Brady wrote:

On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >> So the issue here is that sort is allocating >> a large buffer up front thus impacting the fork(). >> Really sort(1) should be trying to avoid this issue >> in the first place, and the issue is already logged at: >> http://bugs.gnu.org/14752 > > Yes this is the same as I linked above. > Does any body have a patch for this, or should I start working on this?

I was waiting for a patch that didn't materialize. I'll have a look myself now.

So I had a look and the change while definitely worth doing is a bit invasive and so probably not appropriate for the impending release, as that's focusing on bug fixes rather than performance characteristics.

Some implementation notes for reference...

vfork() is portable only when one essentially just does an execve() right after the vfork(). Therefore just for fire and forget processes. Anything where you need to interact with the sub process like setting up files to communicate etc. is going to have portability issues. Even using execvp() is problematic I understand. Also sort is multithreaded which might further complicate things.

Leveraging posix_spawn() is promising, as the main reason for that interface is to provide an efficient fork()+exec() mechanism. That can be implemented using vfork() or with clone() on Linux. The implementation in glibc is only a token one however as it just uses fork()+exec() usually. One can override with the POSIX_SPAWN_USEVFORK flag, but there are many non obvious implementation gotchas with doing that. Again being multithreaded may complicate things here. Note the musl, freebsd, osx and solaris posix_spawn() implementations are efficient, which would be another reason to use this assuming the glibc/gnulib implementation is fixed up.

Another option would be for sort(1) to start up a helper child process, before we allocate much memory. Then we could communicate descriptors back and forth to that, and it could deal with forking the children. That would be portable too, but a little involved. Ideally we could keep the complication within posix_spawn() instead.

Note this is a general issue not just related to sort(1). Many servers for example whether written in java/python/ruby or whatever have this issue when they use lots of memory and would like to popen() something. So fixing up the glibc posix_spawn() implementation would be very useful so that popen() etc. within glibc and the various language runtimes could leverage.

Some links for reference:

http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html https://sourceware.org/ml/libc-help/2010-10/msg00001.html https://sourceware.org/bugzilla/show_bug.cgi?id=10354 https://sourceware.org/bugzilla/show_bug.cgi?id=378 http://git.musl-libc.org/cgit/musl/tree/src/process/posix_spawn.c http://ewontfix.com/7/ http://stackoverflow.com/questions/2731531/faster-forking-of-large-processes-on-linux http://stackoverflow.com/questions/8152076/spawn-process-from-multithreaded-application https://github.com/rtomayko/posix-spawn/blob/master/ext/posix-spawn.c http://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/ http://code.google.com/p/popen-noshell/source/browse/trunk/popen_noshell.c