[Python-Dev] Opcode cache in ceval loop (original) (raw)

Serhiy Storchaka storchaka at gmail.com
Tue Feb 2 13:45:43 EST 2016

On 02.02.16 19:45, Yury Selivanov wrote:

On 2016-02-02 12:41 PM, Serhiy Storchaka wrote:

On 01.02.16 21:10, Yury Selivanov wrote:

To measure the max/average memory impact, I tuned my code to optimize every code object on first run. Then I ran the entire Python test suite. Python test suite + standard library both contain around 72395 code objects, which required 20Mb of memory for caches. The test process consumed around 400Mb of memory. Thus, the absolute worst case scenario, the overhead is about 5%.

Test process consumes such much memory because few tests creates huge objects. If exclude these tests (note that tests that requires more than 1Gb are already excluded by default) and tests that creates a number of threads (threads consume much memory too), the rest of tests needs less than 100Mb of memory. Absolute required minimum is about 25Mb. Thus, the absolute worst case scenario, the overhead is about 100%. Can you give me the exact configuration of tests (command line to run) that would only consume 25mb?

I don't remember what exact tests consume the most of memory, but following tests are failed when run with less than 30Mb of memory:

test___all__ test_asynchat test_asyncio test_bz2 test_capi test_concurrent_futures test_ctypes test_decimal test_descr test_distutils test_docxmlrpc test_eintr test_email test_fork1 test_fstring test_ftplib test_functools test_gc test_gdb test_hashlib test_httplib test_httpservers test_idle test_imaplib test_import test_importlib test_io test_itertools test_json test_lib2to3 test_list test_logging test_longexp test_lzma test_mmap test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_os test_pickle test_poplib test_pydoc test_queue test_regrtest test_resource test_robotparser test_shutil test_smtplib test_socket test_sqlite test_ssl test_subprocess test_tarfile test_tcl test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_tix test_tk test_tools test_ttk_guionly test_ttk_textonly test_tuple test_unicode test_urllib2_localnet test_wait3 test_wait4 test_xmlrpc test_zipfile test_zlib

More information about the Python-Dev mailing list