Performance is poor on M1 silicon · Issue #1288 · nedbat/coveragepy (original) (raw)
Describe the bug
For my project, the overhead of running coverage is about 10% on my 2017 Macbook Pro (Intel i7), but a whopping 250% on my 2021 MacBook Pro (M1 Max), so that in fact running coverage run -m pytest on the M1 is more than 50% slower on the Intel i7.
Some more details: I ran a matrix 6 cases to illustrate the issue. The three machines are the Intel i7, M1 (native arm64), and M1 (Rosetta i386). For each machine, I run my test suite with the command
or
$ time coverage run -m pytest
The resulting run times are summarized in the table below:
+-------------------------------+--------------------------------+-----------------------------+
| | MacBook Pro (Apple M1 Max) | |
| +----------------+---------------+ 2017 Macbook Pro (Intel i7) |
| | Native (arm64) | Rosetta (x86) | |
+-------------------------------+----------------+---------------+-----------------------------+
| $ time pytest | 67.37 | 109.37 | 139.37 |
| $ time coverage run -m pytest | 236.45 | 125.03 | 153.01 |
+-------------------------------+--------------------------------+-----------------------------+
| Overhead | 251% | 14.3% | 9.8% |
+-------------------------------+--------------------------------+-----------------------------+
The take away is that the performance of coverage (at least on my project) is so poor on the native arm36 architecture that it's much faster to run it using i386 binaries and Rosetta, or even run it on the old Intel i7 machine. The performance is so poor that there must be an issue related to either the arm64 architecture (or rather binaries compiled to arm64), or perhaps a specific issue with one of the packages I'm using. Note that pytest itself runs fine --- it's only when run using coverage that there is an issue.
It may take a bit of work on both my part to get you code that you can use to reproduce. I'll answer all the questions I can, and if you need me to I will fork my project to give a minimal use case.
What version of Python are you using?
3.9.9 (using Homebrew and pyenv).
What version of coverage.py shows the problem?
6.2
What versions of what packages do you have installed?
attrs--21.2.0
casadi--3.5.5.post2
coverage--6.2
cycler--0.11.0
cyipopt--1.1.0
Cython--0.29.24
fonttools--4.28.2
iniconfig--1.1.1
jsonschema--4.2.1
kiwisolver--1.3.2
matplotlib--3.5.0
numpy--1.21.4
packaging--21.3
Pillow--8.4.0
pluggy--1.0.0
py--1.11.0
pyparsing--3.0.6
pyrsistent--0.18.0
pytest--6.2.5
python-dateutil--2.8.2
scipy--1.7.3
setuptools-scm--6.3.2
six--1.16.0
toml--0.10.2
tomli--1.2.2
What code are you running?
See my comment above. I can provide a minimal example if needed, but would probably have to excise all the casadi features so that you don't have to compile casadi yourself (which is the only option for native M1 right now).
What commands did you run?
$ time pytest
$ time coverage run -m pytest
Expected behavior
I expected that using the coverage command would add some overhead to running pytest, perhaps around 10%. Instead, using M1 native binaries, I got an additional 250% overhead.
Additional context
None.