[Python-Dev] Opcode cache in ceval loop (original) (raw)

Yury Selivanov yselivanov.ml at gmail.com
Tue Feb 2 14:23:10 EST 2016

On 2016-02-02 1:45 PM, Serhiy Storchaka wrote:

On 02.02.16 19:45, Yury Selivanov wrote:

On 2016-02-02 12:41 PM, Serhiy Storchaka wrote:

On 01.02.16 21:10, Yury Selivanov wrote:

To measure the max/average memory impact, I tuned my code to optimize every code object on first run. Then I ran the entire Python test suite. Python test suite + standard library both contain around 72395 code objects, which required 20Mb of memory for caches. The test process consumed around 400Mb of memory. Thus, the absolute worst case scenario, the overhead is about 5%.

Test process consumes such much memory because few tests creates huge objects. If exclude these tests (note that tests that requires more than 1Gb are already excluded by default) and tests that creates a number of threads (threads consume much memory too), the rest of tests needs less than 100Mb of memory. Absolute required minimum is about 25Mb. Thus, the absolute worst case scenario, the overhead is about 100%. Can you give me the exact configuration of tests (command line to run) that would only consume 25mb? I don't remember what exact tests consume the most of memory, but following tests are failed when run with less than 30Mb of memory: test_all testasynchat testasyncio testbz2 testcapi_ testconcurrentfutures testctypes testdecimal testdescr testdistutils testdocxmlrpc testeintr testemail testfork1 testfstring testftplib testfunctools testgc testgdb testhashlib testhttplib testhttpservers testidle testimaplib testimport testimportlib testio testitertools testjson testlib2to3 testlist testlogging testlongexp testlzma testmmap testmultiprocessingfork testmultiprocessingforkserver testmultiprocessingmainhandling testmultiprocessingspawn testos testpickle testpoplib testpydoc testqueue testregrtest testresource testrobotparser testshutil testsmtplib testsocket testsqlite testssl testsubprocess testtarfile testtcl testthread testthreadedimport testthreadedtempfile testthreading testthreadinglocal testthreadsignals testtix testtk testtools testttkguionly testttktextonly testtuple testunicode testurllib2localnet testwait3 testwait4 testxmlrpc testzipfile testzlib

Alright, I modified the code to optimize ALL code objects, and ran unit tests with the above tests excluded:

-- Max process mem (ru_maxrss) = 131858432 -- Opcode cache number of objects = 42109 -- Opcode cache total extra mem = 10901106

And asyncio tests:

-- Max process mem (ru_maxrss) = 57081856 -- Opcode cache number of objects = 4656 -- Opcode cache total extra mem = 1766681

So the absolute worst case for a small asyncio program is 3%, for unit tests (with the above list excluded) - 8%.

I think it'd be very hard to find a real-life program that consists of only code objects, and nothing else (no data to work with/process, no objects with dicts, no threads, basically nothing). Because only for such a program you would have a 100% memory overhead for the bytecode cache (when all code objects are optimized).

FWIW, here are stats for asyncio with only hot objects being optimized:

-- Max process mem (ru_maxrss) = 54775808 -- Opcode cache number of objects = 121 -- Opcode cache total extra mem = 43521

Yury

More information about the Python-Dev mailing list