Much better stats for seeks and prefix filtering by pdillinger · Pull Request #11460 · facebook/rocksdb (original) (raw)

Summary: We want to know more about opportunities for better range filters, and the effectiveness of our own range filters. Currently the stats are very limited, essentially logging just hits and misses against prefix filters for range scans in BLOOM_FILTER_PREFIX_* without tracking the false positive rate. Perhaps confusingly, when prefix filters are used for point queries, the stats are currently going into the non-PREFIX tickers.

This change does several things:

• Introduce new stat tickers for seeks, LEVEL_SEEK
  • Most importantly, allows us to see opportunities for range filtering. Specifically, we can count how many times a seek in an SST file accesses at least one data block, and how many times at least one value() is then accessed. If a data block was accessed but no value(), we can generally assume that the key(s) seen was(were) not of interest so could have been filtered with the right kind of filter, avoiding the data block access.
  • We can get the same level of detail when a filter (for now, prefix Bloom/ribbon) is used, or not. Specifically, we can infer a false positive rate for prefix filters (not available before) from the seek "false positive" rate: when a data block is accessed but no value() is called. (There can be other explanations for a seek false positive, but in typical iterator usage it would indicate a filter false positive.)
  • The stats are divided between last level and non-last levels, to help understand potential tiered storage use cases.
• The old BLOOM_FILTER_PREFIX_* stats have a different meaning: no longer referring to iterators but to point queries using prefix filters. BLOOM_FILTER_PREFIX_TRUE_POSITIVE is added for computing the prefix false positive rate on point queries, which can be due to filter false positives as well as different keys with the same prefix.
• Similarly, the non-PREFIX BLOOM_FILTER stats are now for whole key filtering only.

Test Plan: unit tests updated (TODO: finish)

Performance test shows a consistent small improvement with these changes, both with clang and with gcc. I'm not sure why, but I'll take it. TODO: details