Dockerize "Direct Execution of Fuzz Targets" by DaveLak · Pull Request #1904 · gitpython-developers/GitPython (original) (raw)
I did take it out for a spin by following the documentation, but encountered an error when building the image (please see below).
Normally I'd ask if this is me, but since it's docker, I'd hope it cannot be. The docker-port is provided by orbstack in my case, and it's possible that that something about that VM is different enough to cause failures. Also, I am doing this on an ARM machine, and probably the VM isn't x86 either.
12.22 /tmp/pip-install-wpyvqb77/atheris_1178434add2a4d5f977b1997318ecda9/setup_utils/find_libfuzzer.sh: line 44: clang: command not found
12.22 Failed to find libFuzzer; set either <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>L</mi><mi>A</mi><mi>N</mi><msub><mi>G</mi><mi>B</mi></msub><mi>I</mi><mi>N</mi><mi>t</mi><mi>o</mi><mi>p</mi><mi>o</mi><mi>i</mi><mi>n</mi><mi>t</mi><mi>t</mi><mi>o</mi><mi>y</mi><mi>o</mi><mi>u</mi><mi>r</mi><mi>C</mi><mi>l</mi><mi>a</mi><mi>n</mi><mi>g</mi><mi>b</mi><mi>i</mi><mi>n</mi><mi>a</mi><mi>r</mi><mi>y</mi><mo separator="true">,</mo><mi>o</mi><mi>r</mi></mrow><annotation encoding="application/x-tex">CLANG_BIN to point to your Clang binary, or </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.07153em;">C</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mord"><span class="mord mathnormal">G</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em;">B</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.07847em;">I</span><span class="mord mathnormal">Nt</span><span class="mord mathnormal">o</span><span class="mord mathnormal">p</span><span class="mord mathnormal">o</span><span class="mord mathnormal">in</span><span class="mord mathnormal">tt</span><span class="mord mathnormal">oyo</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em;">r</span><span class="mord mathnormal" style="margin-right:0.01968em;">Cl</span><span class="mord mathnormal">an</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mord mathnormal">bina</span><span class="mord mathnormal" style="margin-right:0.03588em;">ry</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.02778em;">or</span></span></span></span>LIBFUZZER_LIB to point directly to your libFuzzer .a file. If needed, download and build the latest version of Clang:
12.22 git clone --depth=1 https://github.com/llvm/llvm-project.git
12.22 cd llvm-project
12.22 cmake -DLLVM_ENABLE_PROJECTS='clang;compiler-rt' -G "Unix Makefiles" -S llvm -B build
12.22 NPROC=$(sysctl -n hw.logicalcpu 2>/dev/null || nproc)
12.22 cmake --build build --parallel $NPROC # This step is very slow.
12.22 Then, set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>L</mi><mi>A</mi><mi>N</mi><msub><mi>G</mi><mi>B</mi></msub><mi>I</mi><mi>N</mi><mo>=</mo><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">CLANG_BIN="</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.07153em;">C</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mord"><span class="mord mathnormal">G</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em;">B</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.07847em;">I</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord">"</span></span></span></span>(pwd)/bin/clang" and run pip again.
12.22 You should use this same Clang for building any Python extensions you plan to fuzz.
12.22
12.22 Traceback (most recent call last):
12.22 File "<string>", line 105, in get_libfuzzer_lib
12.22 File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
12.22 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
12.22 File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
12.22 raise CalledProcessError(retcode, process.args,
12.22 subprocess.CalledProcessError: Command '['/tmp/pip-install-wpyvqb77/atheris_1178434add2a4d5f977b1997318ecda9/setup_utils/find_libfuzzer.sh']' returned non-zero exit status 1.
12.22
12.22 During handling of the above exception, another exception occurred:
12.22
12.22 Traceback (most recent call last):
12.22 File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
12.22 main()
12.22 File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
12.22 json_out['return_val'] = hook(**hook_input['kwargs'])
12.22 File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
12.22 return _build_backend().build_wheel(wheel_directory, config_settings,
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 410, in build_wheel
12.22 return self._build_with_temp_dir(
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
12.22 self.run_setup()
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
12.22 super().run_setup(setup_script=setup_script)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
12.22 exec(code, locals())
12.22 File "<string>", line 444, in <module>
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 104, in setup
12.22 return distutils.core.setup(**attrs)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
12.22 return run_commands(dist)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
12.22 dist.run_commands()
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
12.22 self.run_command(cmd)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
12.22 super().run_command(command)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
12.22 cmd_obj.run()
12.22 File "/tmp/pip-build-env-97wquo94/normal/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 368, in run
12.22 self.run_command("build")
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
12.22 self.distribution.run_command(command)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
12.22 super().run_command(command)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
12.22 cmd_obj.run()
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
12.22 self.run_command(cmd_name)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
12.22 self.distribution.run_command(command)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
12.22 super().run_command(command)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
12.22 cmd_obj.run()
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 91, in run
12.22 _build_ext.run(self)
12.22 File "/tmp/pip-build-env-97wquo94/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
12.22 self.build_extensions()
12.22 File "<string>", line 333, in build_extensions
12.22 File "<string>", line 109, in get_libfuzzer_lib
12.22 RuntimeError: Failed to find libFuzzer; set either <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>L</mi><mi>A</mi><mi>N</mi><msub><mi>G</mi><mi>B</mi></msub><mi>I</mi><mi>N</mi><mi>t</mi><mi>o</mi><mi>p</mi><mi>o</mi><mi>i</mi><mi>n</mi><mi>t</mi><mi>t</mi><mi>o</mi><mi>y</mi><mi>o</mi><mi>u</mi><mi>r</mi><mi>C</mi><mi>l</mi><mi>a</mi><mi>n</mi><mi>g</mi><mi>b</mi><mi>i</mi><mi>n</mi><mi>a</mi><mi>r</mi><mi>y</mi><mo separator="true">,</mo><mi>o</mi><mi>r</mi></mrow><annotation encoding="application/x-tex">CLANG_BIN to point to your Clang binary, or </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.07153em;">C</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mord"><span class="mord mathnormal">G</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em;">B</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.07847em;">I</span><span class="mord mathnormal">Nt</span><span class="mord mathnormal">o</span><span class="mord mathnormal">p</span><span class="mord mathnormal">o</span><span class="mord mathnormal">in</span><span class="mord mathnormal">tt</span><span class="mord mathnormal">oyo</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em;">r</span><span class="mord mathnormal" style="margin-right:0.01968em;">Cl</span><span class="mord mathnormal">an</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mord mathnormal">bina</span><span class="mord mathnormal" style="margin-right:0.03588em;">ry</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.02778em;">or</span></span></span></span>LIBFUZZER_LIB to point directly to your libFuzzer .a file. If needed, download and build the latest version of Clang:
12.22 git clone --depth=1 https://github.com/llvm/llvm-project.git
12.22 cd llvm-project
12.22 cmake -DLLVM_ENABLE_PROJECTS='clang;compiler-rt' -G "Unix Makefiles" -S llvm -B build
12.22 NPROC=$(sysctl -n hw.logicalcpu 2>/dev/null || nproc)
12.22 cmake --build build --parallel $NPROC # This step is very slow.
12.22 Then, set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>L</mi><mi>A</mi><mi>N</mi><msub><mi>G</mi><mi>B</mi></msub><mi>I</mi><mi>N</mi><mo>=</mo><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">CLANG_BIN="</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.07153em;">C</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mord"><span class="mord mathnormal">G</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em;">B</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.07847em;">I</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord">"</span></span></span></span>(pwd)/bin/clang" and run pip again.
12.22 You should use this same Clang for building any Python extensions you plan to fuzz.
12.22
12.22 [end of output]
12.22
12.22 note: This error originates from a subprocess, and is likely not a problem with pip.
12.23 ERROR: Failed building wheel for atheris
12.23 Failed to build atheris
12.23 ERROR: Could not build wheels for atheris, which is required to install pyproject.toml-based projects
------
Dockerfile:13
--------------------
12 | # Update package managers, install necessary packages, and cleanup unnecessary files in a single RUN to keep the image smaller.
13 | >>> RUN apt-get update && \
14 | >>> apt-get install --no-install-recommends -y git && \
15 | >>> python -m pip install --upgrade pip && \
16 | >>> python -m pip install atheris && \
17 | >>> python -m pip install -e . && \
18 | >>> apt-get clean && \
19 | >>> apt-get autoremove -y && \
20 | >>> rm -rf /var/lib/apt/lists/* /root/.cache
21 |
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install --no-install-recommends -y git && python -m pip install --upgrade pip && python -m pip install atheris && python -m pip install -e . && apt-get clean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* /root/.cache" did not complete successfully: exit code: 1