Low-overhead profiling · Issue #77 · mikemccand/luceneutil (original) (raw)

Skip to content

Navigation Menu

• • GitHub Copilot Write better code with AI
  • GitHub Models New Manage and compare prompts
  • GitHub Advanced Security Find and fix vulnerabilities
  • Actions Automate any workflow
  • Codespaces Instant dev environments
  • Issues Plan and track work
  • Code Review Manage code changes
  • Discussions Collaborate outside of code
  • Code Search Find more, search less
• Explore
  • Learning Pathways
  • Events & Webinars
  • Ebooks & Whitepapers
  • Customer Stories
  • Partners
  • Executive Insights
• • GitHub Sponsors Fund open source developers
  • The ReadME Project GitHub community articles
• • Enterprise platform AI-powered developer platform
• Pricing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign up

Appearance settings

Description

I think the benchmark results would be more useful if we could see some basic analysis of the code under load. We could imagine summary tables, graphs, flame charts, whatever is needed to knock out the hotspots and keep them under control.

Instead of seeing just that performance dropped or improved, we might be able to reason about why: it could make things better for developers. And maybe, just maybe, we have some giant frogs boiling for years that nobody never noticed yet: you know, easy wins to improve perf that went undetected.

As a start I imagine breaking the problem down into two profiling mechamisms:

• "system profiler": linux perf equivalent. We can "teach" it java with some JVM flags + agent. See blog post: https://jvns.ca/blog/2016/02/24/perf-top-my-new-best-friend/. I think in this case it is a matter of downloading the agent and tweaking the JVM args, wrapping with correct perf commandlines. We don't have to start with a fancy flamegraph report, it could be a simple table to get things going. I really like how this tools "drills down" even into the kernel to find low-level bottlenecks. Maybe dtrace support on the mac could do the equivalent for local runs, and make it easier on developers.
• java profiling: just using java flight recorder, similar to lucene's test suite. There we just apply some flags and collect the data files correctly across different jvms/projects (it is convenient that the default filename includes pid for the multi-jvm case!): https://github.com/apache/lucene-solr/blob/master/gradle/testing/profiling.gradle . Hopefully the logic is easy to see: we just clean the relevant recordings beforehand and collect the relevant ones after and feed them to a reporter (https://github.com/apache/lucene-solr/blob/master/buildSrc/src/main/java/org/apache/lucene/gradle/ProfileResults.java). To exactly specify the overhead of the metrics we collect, we supply a configuration file: https://github.com/apache/lucene-solr/blob/master/gradle/testing/profiling.jfc