[Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Wed Sep 7 14:32:59 CEST 2011

On Wed, 07 Sep 2011 13:38:23 +0200 Jesus Cea <jcea at jcea.es> wrote:

So, the problem is that a) "make test" takes quite a bit of RAM and b) the buildbot forks some "big" processes, so the virtual memory needed is BIG.

Note that buildbots run "make buildbottest", not "make test".

So I have two questions:

1. Can we reduce the memory footprint of the tests?. I can't understand why the python test process is taking so much memory.

Because the test suite will by construction load all the stdlib (minus the few modules which don't have a test suite), and creates numerous test scenarios. Depending on the memory allocator, fragmentation can make it difficult to reclaim memory that has been formally freed after a test is run.

If "-j" is used, tests get run in a separate process each, so that approach might be an answer.

2. Why buildbot is "forking()" big processes?. Can we do something to change this?.

Because we need to test for various functionalities, such as os.fork() and os.exec*(), but also the command-line behaviour of the interpreter, the distutils module, the packaging module, the subprocess module, the multiprocessing module... (this list is not exhaustive).

Regards

Antoine.

More information about the Python-Dev mailing list