GitHub - rkern/line_profiler: (OLD REPO) Line-by-line profiling for Python - Current repo -> (original) (raw)

line_profiler and kernprof

line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either line_profiler or the Python standard library's cProfile or profile modules, depending on what is available.

They are available under a BSD license.

Contents

Installation

Note: As of version 2.1.2, pip install line_profiler does not work. Please install as follows until it is fixed in the next release:

git clone https://github.com/rkern/line_profiler.git find line_profiler -name '*.pyx' -exec cython {} ; cd line_profiler pip install . --user

Releases of line_profiler can be installed using pip:

$ pip install line_profiler

Source releases and any binaries can be downloaded from the PyPI link.

http://pypi.python.org/pypi/line_profiler

To check out the development sources, you can use Git:

$ git clone https://github.com/rkern/line_profiler.git

You may also download source tarballs of any snapshot from that URL.

Source releases will require a C compiler in order to build line_profiler. In addition, git checkouts will also require Cython >= 0.10. Source releases on PyPI should contain the pregenerated C sources, so Cython should not be required in that case.

kernprof is a single-file pure Python script and does not require a compiler. If you wish to use it to run cProfile and not line-by-line profiling, you may copy it to a directory on your PATH manually and avoid trying to build any C extensions.

line_profiler

The current profiling tools supported in Python 2.7 and later only time function calls. This is a good first step for locating hotspots in one's program and is frequently all one needs to do to optimize the program. However, sometimes the cause of the hotspot is actually a single line in the function, and that line may not be obvious from just reading the source code. These cases are particularly frequent in scientific computing. Functions tend to be larger (sometimes because of legitimate algorithmic complexity, sometimes because the programmer is still trying to write FORTRAN code), and a single statement without function calls can trigger lots of computation when using libraries like numpy. cProfile only times explicit function calls, not special methods called because of syntax. Consequently, a relatively slow numpy operation on large arrays like this,

a[large_index_array] = some_other_large_array

is a hotspot that never gets broken out by cProfile because there is no explicit function call in that statement.

LineProfiler can be given functions to profile, and it will time the execution of each individual line inside those functions. In a typical workflow, one only cares about line timings of a few functions because wading through the results of timing every single line of code would be overwhelming. However, LineProfiler does need to be explicitly told what functions to profile. The easiest way to get started is to use the kernprof script.

$ kernprof -l script_to_profile.py

kernprof will create an instance of LineProfiler and insert it into the __builtins__ namespace with the name profile. It has been written to be used as a decorator, so in your script, you decorate the functions you want to profile with @profile.

@profile def slow_function(a, b, c): ...

The default behavior of kernprof is to put the results into a binary file script_to_profile.py.lprof . You can tell kernprof to immediately view the formatted results at the terminal with the [-v/--view] option. Otherwise, you can view the results later like so:

$ python -m line_profiler script_to_profile.py.lprof

For example, here are the results of profiling a single function from a decorated version of the pystone.py benchmark (the first two lines are output from pystone.py, not kernprof):

Pystone(1.1) time for 50000 passes = 2.48 This machine benchmarks at 20161.3 pystones/second Wrote profile results to pystone.py.lprof Timer unit: 1e-06 s

File: pystone.py Function: Proc2 at line 149 Total time: 0.606656 s

Line # Hits Time Per Hit % Time Line Contents

149 @profile 150 def Proc2(IntParIO): 151 50000 82003 1.6 13.5 IntLoc = IntParIO + 10 152 50000 63162 1.3 10.4 while 1: 153 50000 69065 1.4 11.4 if Char1Glob == 'A': 154 50000 66354 1.3 10.9 IntLoc = IntLoc - 1 155 50000 67263 1.3 11.1 IntParIO = IntLoc - IntGlob 156 50000 65494 1.3 10.8 EnumLoc = Ident1 157 50000 68001 1.4 11.2 if EnumLoc == Ident1: 158 50000 63739 1.3 10.5 break 159 50000 61575 1.2 10.1 return IntParIO

The source code of the function is printed with the timing information for each line. There are six columns of information.

If you are using IPython, there is an implementation of an %lprun magic command which will let you specify functions to profile and a statement to execute. It will also add its LineProfiler instance into the __builtins__, but typically, you would not use it like that.

For IPython 0.11+, you can install it by editing the IPython configuration file ~/.ipython/profile_default/ipython_config.py to add the 'line_profiler' item to the extensions list:

c.TerminalIPythonApp.extensions = [ 'line_profiler', ]

To get usage help for %lprun, use the standard IPython help mechanism:

In [1]: %lprun?

These two methods are expected to be the most frequent user-level ways of using LineProfiler and will usually be the easiest. However, if you are building other tools with LineProfiler, you will need to use the API. There are two ways to inform LineProfiler of functions to profile: you can pass them as arguments to the constructor or use the add_function(f) method after instantiation.

profile = LineProfiler(f, g) profile.add_function(h)

LineProfiler has the same run(), runctx(), and runcall() methods as cProfile.Profile as well as enable() and disable(). It should be noted, though, that enable() and disable() are not entirely safe when nested. Nesting is common when using LineProfiler as a decorator. In order to support nesting, use enable_by_count() and disable_by_count(). These functions will increment and decrement a counter and only actually enable or disable the profiler when the count transitions from or to 0.

After profiling, the dump_stats(filename) method will pickle the results out to the given file. print_stats([stream]) will print the formatted results to sys.stdout or whatever stream you specify. get_stats() will return LineStats object, which just holds two attributes: a dictionary containing the results and the timer unit.

kernprof

kernprof also works with cProfile, its third-party incarnation lsprof, or the pure-Python profile module depending on what is available. It has a few main features:

The results of profile script_to_profile.py will be written to script_to_profile.py.prof by default. It will be a typical marshalled file that can be read with pstats.Stats(). They may be interactively viewed with the command:

$ python -m pstats script_to_profile.py.prof

Such files may also be viewed with graphical tools like kcachegrind through the converter program pyprof2calltree or RunSnakeRun.

Frequently Asked Questions

To Do

cProfile uses a neat "rotating trees" data structure to minimize the overhead of looking up and recording entries. LineProfiler uses Python dictionaries and extension objects thanks to Cython. This mostly started out as a prototype that I wanted to play with as quickly as possible, so I passed on stealing the rotating trees for now. As usual, I got it working, and it seems to have acceptable performance, so I am much less motivated to use a different strategy now. Maybe later. Contributions accepted!

Bugs and Such

Bugs and pull requested can be submitted on GitHub.

Changes

2.1

2.0

1.1

1.0

1.0b3

1.0b2

1.0b1