📖 Bencher Self-Hosted with Docker Quick Start (original) (raw)

What is Bencher?

Bencher is a suite of continuous benchmarking tools. Have you ever had a performance regression impact your users? Bencher could have prevented that from happening. Bencher allows you to detect and prevent performance regressions before they make it to production.

Run: Run your benchmarks locally or in CI using your favorite benchmarking tools. The bencher CLI simply wraps your existing benchmark harness and stores its results.
Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
Catch: Catch performance regressions in CI. Bencher uses state of the art, customizable analytics to detect performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should be run in CI with Bencher to prevent performance regressions. Performance bugs are bugs!

Bencher Self-Hosted

Bencher is open source and self-hostable. If you are interested in using Bencher Cloud, check out the Bencher Cloud Quick Start tutorial. This tutorial will get you setup using Bencher Self-Hosted with Docker.

🐰 Once you feel comfortable using Bencher Self-Hosted, consider checking out the following resources:

Bencher Self-Hosted

Console Server Config

API Server Config

Bencher Architecture

Install Docker

In order to run the Console and API servers in this tutorial you will need to have docker installed. Check to see if you have docker installed. Run:

Terminal window


docker --version

Terminal window


docker --version

Terminal window


docker --version

You should see something like:


Docker version 20.10.17, build 100c701

It is okay if your version number is different. It’s just important that this command works. If not follow the instructions for installing docker.

Install `bencher` CLI

Select your operating system and run the provided command to install the bencher CLI. For more details, see the bencher CLI install documentation.

Terminal window


curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh

Terminal window


curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh

Terminal window


cargo install --git https://github.com/bencherdev/bencher --branch main --locked --force bencher_cli

Terminal window


powershell -c "irm https://bencher.dev/download/install-cli.ps1 | iex"

LinuxmacOSWindowsOther

Terminal window

⠀

Now, lets check that you have the bencher CLI installed. Run:

Terminal window


bencher --version

Terminal window


bencher --version

Terminal window


bencher --version

You should see:

Run `bencher up`

With docker and the bencher CLI installed, you can now run the Console and API servers with the bencher up CLI subcommand. Run:

Terminal window


bencher up

Terminal window


bencher up

Terminal window


bencher up

You should see something like:


Pulling `ghcr.io/bencherdev/bencher-api:latest` image...

Creating `bencher_api` container...

Starting `bencher_api` container...

Pulling `ghcr.io/bencherdev/bencher-console:latest` image...

Creating `bencher_console` container...

Starting `bencher_console` container...

🐰 Bencher Self-Hosted is up and running!

Console Server: http://localhost:3000

API Server: http://localhost:61016

Press Ctrl+C to stop Bencher Self-Hosted.

🐰 Bencher Self-Hosted logs...

Jan 08 16:49:07.727 INFO 🐰 Bencher API Server v0.5.1

...

Again, it is okay if your output is different. It’s just important that this command works.

🐰 If you get an error from ghcr.io saying:"authentication required"Try running: docker logout ghcr.io

Select your Benchmark Harness

If you already have benchmarks written, select your programming language and benchmarking harness from the list below. Otherwise, just skip this step. For more details, see the benchmark harness adapters documentation.

C++

Java

JavaScript

Python

Ruby

Rust

Shell

JSON

Track your Benchmarks

You are now ready to track your benchmark results! To do so, you will use the bencher run CLI subcommandto run your benchmarks and collect the results. Run:

Terminal window


bencher run --host https://localhost:61016 "make benchmarks"

Terminal window


bencher run --host https://localhost:61016 "make benchmarks"

Terminal window


bencher run --host https://localhost:61016 "make benchmarks --benchmark_format=json"

Terminal window


bencher run --host https://localhost:61016 "make benchmarks --benchmark_format=json"

Terminal window


bencher run --host https://localhost:61016 "dotnet run -c Release"

Terminal window


bencher run --host https://localhost:61016 "dotnet run -c Release"

Terminal window

⠀

Terminal window


bencher run --host https://localhost:61016 "go test -bench"

Terminal window


bencher run --host https://localhost:61016 "go test -bench"

Terminal window


bencher run --host https://localhost:61016 --file results.json "java -jar benchmarks.jar -rf json -rff results.json"

Terminal window


bencher run --host https://localhost:61016 --file results.json "java -jar benchmarks.jar -rf json -rff results.json"

Terminal window


bencher run --host https://localhost:61016 "node benchmark.js"

Terminal window


bencher run --host https://localhost:61016 "node benchmark.js"

Terminal window


bencher run --host https://localhost:61016 "node benchmark.js"

Terminal window


bencher run --host https://localhost:61016 "node benchmark.js"

Terminal window


bencher run --host https://localhost:61016 "bencher mock"

Terminal window


bencher run --host https://localhost:61016 "bencher mock"

Terminal window


bencher run --host https://localhost:61016 "asv run"

Terminal window


bencher run --host https://localhost:61016 "asv run"

Terminal window


bencher run --host https://localhost:61016 --file results.json "pytest --benchmark-json results.json benchmarks.py"

Terminal window


bencher run --host https://localhost:61016 --file results.json "pytest --benchmark-json results.json benchmarks.py"

Terminal window


bencher run --host https://localhost:61016 "ruby benchmarks.rb"

Terminal window


bencher run --host https://localhost:61016 "ruby benchmarks.rb"

Terminal window


bencher run --host https://localhost:61016 "cargo +nightly bench"

Terminal window


bencher run --host https://localhost:61016 "cargo +nightly bench"

Terminal window


bencher run --host https://localhost:61016 "cargo bench"

Terminal window


bencher run --host https://localhost:61016 "cargo bench"

Terminal window


bencher run --host https://localhost:61016 "cargo bench"

Terminal window


bencher run --host https://localhost:61016 "cargo bench"

Terminal window


bencher run --host https://localhost:61016 "cargo bench"

Terminal window


bencher run --host https://localhost:61016 "cargo bench"

Terminal window


bencher run --host https://localhost:61016 --file results.json "hyperfine --export-json results.json 'sleep 0.1'"

Terminal window


bencher run --host https://localhost:61016 --file results.json "hyperfine --export-json results.json 'sleep 0.1'"

You may need to modify the benchmark command to match your setup. If you don’t have any benchmarks yet, you can just use the bencher mock subcommand as your benchmark command to generate some mock data. If everything works as expected, the end of the output should look something like this:


View results:

- bencher::mock_0 (Latency): https://localhost:3000/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=f7022024-ae16-4782-8f0d-869d65a82930&measures=775999d3-d705-482f-acd8-41947f8e0fbc&start_time=1741390156000&end_time=1743982156000&report=709d3476-51a4-4939-9584-75d9a2c04c54

- bencher::mock_1 (Latency): https://localhost:3000/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=7a823440-216f-482d-a05f-8bf75e865bba&measures=775999d3-d705-482f-acd8-41947f8e0fbc&start_time=1741390156000&end_time=1743982156000&report=709d3476-51a4-4939-9584-75d9a2c04c54

- bencher::mock_2 (Latency): https://localhost:3000/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=8d9695ff-f352-4781-9561-3c69012fd9fe&measures=775999d3-d705-482f-acd8-41947f8e0fbc&start_time=1741390156000&end_time=1743982156000&report=709d3476-51a4-4939-9584-75d9a2c04c54

- bencher::mock_3 (Latency): https://localhost:3000/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=8ef6e256-8084-4afe-a7cf-eaa46384c19d&measures=775999d3-d705-482f-acd8-41947f8e0fbc&start_time=1741390156000&end_time=1743982156000&report=709d3476-51a4-4939-9584-75d9a2c04c54

- bencher::mock_4 (Latency): https://localhost:3000/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=1205e35a-c73b-4ff9-916c-40838a62ae0b&measures=775999d3-d705-482f-acd8-41947f8e0fbc&start_time=1741390156000&end_time=1743982156000&report=709d3476-51a4-4939-9584-75d9a2c04c54

Claim this project: https://localhost:3000/auth/signup?claim=d4b0cd5a-8422-40af-9872-8e18d5d062c4

You can now view the results for each of your benchmarks in the browser. Click or copy and paste the links from View results. To claim these results, click or copy and paste the Claim this project link into your browser.

🐰 Congrats! You tracked your first benchmark results with Bencher Self-Hosted! 🎉

Keep Going: How to Claim Benchmark Results ➡

Published: Sat, August 12, 2023 at 4:07:00 PM UTC | Last Updated: Sun, April 6, 2025 at 6:30:00 PM UTC

📖 Bencher Self-Hosted with Docker Quick Start (original) (raw)

What is Bencher?

Bencher Self-Hosted

Install Docker

Install bencher CLI

Run bencher up

Select your Benchmark Harness

Track your Benchmarks

Keep Going: How to Claim Benchmark Results ➡

Install `bencher` CLI

Run `bencher up`