DataFusion + Conbench Integration by dianaclarke · Pull Request #1791 · apache/datafusion (original) (raw)

Here's a minimal DataFusion + Conbench[1] proof of concept.

[1] https://github.com/conbench/conbench

A few notes (areas for improvement, caveats, etc):

• Criterion results are in nanoseconds, but the smallest unit
  Conbench currently speaks is seconds (because Conbench was initially
  for macro not micro benchmarking). I suspect most places in Conbench
  would work just fine if nanoseconds were passed in, but I need to
  audit the code for any places that assume seconds if it isn't a
  throughput benchmark.
• If the Criterion benchmarks were named better, I could tag them
  better in Conbench. For example, I suspect sqrt_20_12, sqrt_20_9,
  sqrt_22_12, and sqrt_22_14 are parameterized variations of the same
  benchmark, and if they were named something like "sqrt, foo=20,
  bar=12", I could batch them together & tag their parameters so that
  Conbench would automatically graph them in relation to each other. I
  was sort of able to do this with the following benchmarks (because
  there was a machine readable pattern). Anyhoo, that's easy enough to
  do down the road as a last integration step, and it does appear from
  the Criterion docs that they have their own recommendations for how to
  do this.
  • window partition by, u64_narrow, aggregate functions
  • window partition by, u64_narrow, built-in functions
  • window partition by, u64_wide, aggregate functions
  • window partition by, u64_wide, built-in functions
• While Criterion benchmarks can also measure throughput in some
  cases, all the arrow-datafusion benchmarks were in elapsed time (not
  sure about the arrow-rs benchmarks), so I didn't bother writing code
  to support potential throughput results from
  arrow-datafusion/arrow-rs, but we may need to revisit that.
• We probably want to add some additional context, like the
  arrow-rs/arrow-datafusion version, rust version, any compiler flags,
  etc.