Monitoring with Prometheus and Grafana :: Apache Solr Reference Guide (original) (raw)

This graphic provides a more detailed view:

Figure 1. solr-exporter Diagram

There are three aspects to running solr-exporter:

• Modify the solr-exporter-config.xml to define the data to collect. Solr has a default configuration you can use, but if you would like to modify it before running the exporter the first time, see the section Exporter Configuration below.
• Start the exporter from within Solr. See the section below Starting the Exporter.
• Modify your Prometheus configuration to listen on the correct port. See the section below Prometheus Configuration

Starting the Exporter

You can start solr-exporter by running ./bin/solr-exporter (Linux) or .\bin\solr-exporter.cmd (Windows) from the prometheus-exporter/ directory.

The metrics exposed by solr-exporter can be seen at the metrics endpoint: http://localhost:8983/solr/admin/metrics.

See the commands below depending on your operating system and Solr operating mode:

• Linux
• Windows

User-managed / Single-node

$ cd prometheus-exporter
$ ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr --config-file ./conf/solr-exporter-config.xml --num-threads 8

SolrCloud

$ cd prometheus-exporter
$ ./bin/solr-exporter -p 9854 -z localhost:2181/solr --config-file ./conf/solr-exporter-config.xml --num-threads 16

User-managed / Single-node

> cd prometheus-exporter
> .\bin\solr-exporter.cmd -p 9854 -b http://localhost:8983/solr --config-file .\conf\solr-exporter-config.xml --num-threads 8

SolrCloud

> cd prometheus-exporter
> .\bin\solr-exporter -p 9854 -z localhost:2181/solr --config-file .\conf\solr-exporter-config.xml --num-threads 16

Command Line Parameters

The list of available parameters for the Prometheus Exporter. All parameters can be provided via an environment variable, instead of through the command line.

-h, --help

Displays command line help and usage.

-p, --port, $PORT

The port where Prometheus will listen for new data. This port will be used to configure Prometheus. It can be any port not already in use on your server.

-b, --baseurl, $SOLR_URL

The Solr base URL (such as http://localhost:8983/solr) when Solr is running in a The Solr base URL (such as http://localhost:8983/solr) when Solr is running in a user-managed cluster or a single-node installation. If you are running SolrCloud, do not specify this parameter. If neither the -b parameter nor the -z parameter are defined, the default is -b http://localhost:8983/solr.

-z, --zk-host, $ZK_HOST

The ZooKeeper connect string (such as localhost:9983, or localhost:2181/solr) when Solr is running SolrCloud. If you are running a user-managed cluster or single-node installation, do not specify this parameter. If neither the -b parameter nor the -z parameter are defined, the -b parameter default is used.

--config-file, $CONFIG_FILE

The path to the configuration file that defines the Solr metrics to read.

--num-threads, $NUM_THREADS

The number of threads. The solr-exporter creates thread pools for requests to Solr. Request latency can be improved by increasing the number of threads.

--scrape-interval, $SCRAPE_INTERVAL

The number of seconds between collecting metrics from Solr. The solr-exporter collects metrics from Solr every few seconds controlled by this setting. These metrics are cached and returned regardless of how frequently prometheus is configured to pull metrics from this tool. The freshness of the metrics can be improved by reducing the scrape interval but do not set it to a very low value because metrics collection can be expensive and can execute arbitrary searches to ping Solr.

--cluster-id, $CLUSTER_ID

A unique ID for the cluster to monitor. This ID will be added to all metrics as a label cluster_id and can be used as a filter in the Grafana dashboard if you operate multiple Solr clusters reporting to the same Prometheus instance. If this option is omitted, a hash of the baseUrl or zkHost will be used as ID by default.

-u, --credentials, $CREDENTIALS

Specify the credentials in the format username:password. Example: --credentials solr:SolrRocks.

-ssl, --ssl-enabled, $SSL_ENABLED

Enable mTLS connection to Solr. Expects following env variables: SOLR_SSL_KEY_STORE, SOLR_SSL_KEY_STORE_PASSWORD, SOLR_SSL_TRUST_STORE, SOLR_SSL_TRUST_STORE_PASSWORD. Example: --ssl-enabledThe environment variables are the same that Solr uses to enable mTLS.

Environment Variable Options

The ./bin scripts provided with the Prometheus Exporter support the use of custom java options through the following environment variables:

JAVA_HEAP

Sets the initial (Xms) and max (Xmx) Java heap size.

JAVA_MEM

Custom java memory settings (e.g., -Xms1g -Xmx2g). This is ignored if JAVA_HEAP is provided.

GC_TUNE

Custom Java garbage collection settings.

JAVA_OPTS

Extra JVM options.

ZK_CREDS_AND_ACLS

Credentials for connecting to a ZooKeeper host that is protected with ACLs. For more information on what to include in this variable, refer to the section ZooKeeper ACLs in Solr CLI or the example Getting Metrics from a Secured SolrCloud below.

CLASSPATH_PREFIX

Location of extra libraries to load when starting the solr-exporter.

All Command Line Parameters are able to be provided via environment variables when using the ./bin scripts.

Getting Metrics from a Secured SolrCloud

Your SolrCloud security configuration can be injected into solr-exporter using environment variables in a fashion similar to other clients using SolrJ. This is possible because the main script picks up Environment Variable Options and passes them on to the Java process.

Suppose you have a file basicauth.properties with the Solr Basic-Auth credentials:

httpBasicAuthUser=myUser httpBasicAuthPassword=myPassword

Then you can start the Exporter as follows (Linux).

$ cd prometheus-exporter
$ export JAVA_OPTS="-Djavax.net.ssl.trustStore=truststore.p12 -Djavax.net.ssl.trustStorePassword=truststorePassword -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory -Dsolr.httpclient.config=basicauth.properties"
$ export ZK_CREDS_AND_ACLS="-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider -DzkDigestUsername=readonly-user -DzkDigestPassword=zkUserPassword"
$ export CLASSPATH_PREFIX="../server/solr-webapp/webapp/WEB-INF/lib/commons-codec-1.11.jar"
$ ./bin/solr-exporter -p 9854 -z zk1:2181,zk2:2181,zk3:2181 --config-file ./conf/solr-exporter-config.xml --num-threads 16

NOTE

The Exporter needs the commons-codec library for SSL/BasicAuth, but does not bring it. Therefore the example reuses it from the Solr web app. Of course, you can use a different source.

Exporter Configuration

The configuration for the solr-exporter defines the data to get from Solr. This includes the metrics, but can also include queries to the PingRequestHandler, the Collections API, and a query to any query request handler.

A default example configuration is in prometheus-exporter/conf/solr-exporter-config.xml. Below is a slightly shortened version of it:

<config>

  <rules>

    <ping>
      <lst name="request">
        <lst name="query">
          <str name="path">/admin/ping</str>
        </lst>
        <arr name="jsonQueries">
          <str>
            . as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi><mi>b</mi><mi>j</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">object | </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.05724em;">bj</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord">∣</span></span></span></span>object |
            (if <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi><mi>b</mi><mi>j</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>s</mi><mi>t</mi><mi>a</mi><mi>t</mi><mi>u</mi><mi>s</mi><mo>=</mo><mo>=</mo><mi mathvariant="normal">&quot;</mi><mi>O</mi><mi>K</mi><mi mathvariant="normal">&quot;</mi><mi>t</mi><mi>h</mi><mi>e</mi><mi>n</mi><mn>1.0</mn><mi>e</mi><mi>l</mi><mi>s</mi><mi>e</mi><mn>0.0</mn><mi>e</mi><mi>n</mi><mi>d</mi><mo stretchy="false">)</mo><mi>a</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">object.status == &quot;OK&quot; then 1.0 else 0.0 end) as </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.05724em;">bj</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">u</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">==</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">&quot;</span><span class="mord mathnormal" style="margin-right:0.02778em;">O</span><span class="mord mathnormal" style="margin-right:0.07153em;">K</span><span class="mord">&quot;</span><span class="mord mathnormal">t</span><span class="mord mathnormal">h</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord">1.0</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.01968em;">l</span><span class="mord mathnormal">se</span><span class="mord">0.0</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">d</span><span class="mclose">)</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span></span></span></span>value |
            {
              name         : "solr_ping",
              type         : "GAUGE",
              help         : "See following URL: https://solr.apache.org/guide/solr/latest/deployment-guide/ping.html",
              label_names  : [],
              label_values : [],
              value        : $value
            }
          </str>
        </arr>
      </lst>
    </ping>

    <metrics>
      <lst name="request">
        <lst name="query">
          <str name="path">/admin/metrics</str>
          <lst name="params">
            <str name="group">all</str>
            <str name="type">all</str>
            <str name="prefix"></str>
            <str name="property"></str>
          </lst>
        </lst>
        <arr name="jsonQueries">
          <!--
            jetty metrics
          -->
          <str>
            .metrics["solr.jetty"] | to_entries | .[] | select(.key | startswith("org.eclipse.jetty.server.handler.DefaultHandler")) | select(.key | endswith("xx-responses")) as $object |
            <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi><mi>b</mi><mi>j</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>k</mi><mi>e</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mi>p</mi><mi>l</mi><mi>i</mi><mi>t</mi><mo stretchy="false">(</mo><mi mathvariant="normal">&quot;</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">&quot;</mi><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi><mi>l</mi><mi>a</mi><mi>s</mi><mi>t</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mi>p</mi><mi>l</mi><mi>i</mi><mi>t</mi><mo stretchy="false">(</mo><mi mathvariant="normal">&quot;</mi><mo>−</mo><mi mathvariant="normal">&quot;</mi><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi><mi>f</mi><mi>i</mi><mi>r</mi><mi>s</mi><mi>t</mi><mi>a</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">object.key | split(&quot;.&quot;) | last | split(&quot;-&quot;) | first as </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.05724em;">bj</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.03148em;">k</span><span class="mord mathnormal" style="margin-right:0.03588em;">ey</span><span class="mord">∣</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.01968em;">pl</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord">&quot;.&quot;</span><span class="mclose">)</span><span class="mord">∣</span><span class="mord mathnormal" style="margin-right:0.01968em;">l</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span><span class="mord mathnormal">t</span><span class="mord">∣</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.01968em;">pl</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord">&quot;</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">&quot;</span><span class="mclose">)</span><span class="mord">∣</span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal">rs</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span></span></span></span>status |
            <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi><mi>b</mi><mi>j</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>v</mi><mi>a</mi><mi>l</mi><mi>u</mi><mi>e</mi><mi mathvariant="normal">.</mi><mi>c</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>t</mi><mi>a</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">object.value.count as </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.05724em;">bj</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em;">l</span><span class="mord mathnormal">u</span><span class="mord mathnormal">e</span><span class="mord">.</span><span class="mord mathnormal">co</span><span class="mord mathnormal">u</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span></span></span></span>value |
            {
            name         : "solr_metrics_jetty_response_total",
            type         : "COUNTER",
            help         : "See following URL: https://solr.apache.org/guide/solr/latest/deployment-guide/metrics-reporting.html",
            label_names  : ["status"],
            label_values : [$status],
            value        : $value
            }
          </str>
...
        </arr>
      </lst>
    </metrics>

    <collections>
      <lst name="request">
        <lst name="query">
          <str name="path">/admin/collections</str>
          <lst name="params">
            <str name="action">CLUSTERSTATUS</str>
          </lst>
        </lst>
        <arr name="jsonQueries">
          <str>
            .cluster.live_nodes | length as $value|
            {
              name         : "solr_collections_live_nodes",
              type         : "GAUGE",
              help         : "See following URL: https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html#clusterstatus",
              label_names  : [],
              label_values : [],
              value        : $value
            }
          </str>
...
        </arr>
      </lst>
    </collections>

    <search>
      <lst name="request">
        <lst name="query">
          <str name="collection">collection1</str>
          <str name="path">/select</str>
          <lst name="params">
            <str name="q">*:*</str>
            <str name="start">0</str>
            <str name="rows">0</str>
            <str name="json.facet">
              {
                category: {
                  type: terms,
                  field: cat
                }
              }
            </str>
          </lst>
        </lst>
        <arr name="jsonQueries">
          <str>
            .facets.category.buckets[] as $object |
            <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi><mi>b</mi><mi>j</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>v</mi><mi>a</mi><mi>l</mi><mi>a</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">object.val as </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.05724em;">bj</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em;">l</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span></span></span></span>term |
            <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi><mi>b</mi><mi>j</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>c</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>t</mi><mi>a</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">object.count as </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.05724em;">bj</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal">co</span><span class="mord mathnormal">u</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span></span></span></span>value |
            {
              name         : "solr_facets_category",
              type         : "GAUGE",
              help         : "Category facets",
              label_names  : ["term"],
              label_values : [$term],
              value        : $value
            }
          </str>
        </arr>
      </lst>
    </search>

  </rules>

</config>

Configuration Tags and Elements

The solr-exporter works by making a request to Solr according to the definitions in the configuration file, scraping the response, and converting it to a JSON structure Prometheus can understand. The configuration file defines the elements to request, how to scrape them, and where to place the extracted data in the JSON template.

The solr-exporter configuration file always starts and closes with two simple elements:

<config>
  <rules>

  </rules>
</config>

Between these elements, the data the solr-exporter should request is defined. There are several possible types of requests to make:

Within each of these types, we need to define the query and how to work with the response. To do this, we define two additional elements:

<query>

Defines the query parameter(s) used for the request. This section uses several additional properties to define your query:

collection

The collection to issue the query against. Only used with SolrCloud clusters.

core

The core to issue the query against. Only used with user-managed clusters or single-node installations.

path

The path to the query endpoint where the request will be sent. Examples include admin/metrics or /select or admin/collections.

params

Additional query parameters. These will vary depending on the request type and the endpoint. For example, if using the Metrics endpoint, you can add parameters to limit the query to a certain group and/or prefix. If you’re using the Collections API, the command you want to use would be a parameter.

<jsonQueries>

This is an array that defines one or more JSON Queries in jq syntax. For more details about how to structure these queries, see the jq user manual.

A jq query has to output JSON in the following format:

{
  "name": "solr_ping",
  "type": "GAUGE",
  "help": "See following URL: https://solr.apache.org/guide/solr/latest/deployment-guide/ping.html",
  "label_names": ["base_url","core"],
  "label_values": ["http://localhost:8983/solr","collection1"],
  "value": 1.0
}

See the section Exposition Format below for information about what information should go into each property, and an example of how the above example is translated for Prometheus.

Exposition Format

The solr-exporter converts the JSON to the following exposition format:

# TYPE <name> <type>
# HELP <name> <help>
<name>{<label_names[0]>=<label_values[0]>,<label_names[1]>=<labelvalues[1]>,...} <value>

The following parameters should be set:

For example, solr-exporter converts the JSON in the previous section to the following:

# TYPE solr_ping gauge
# HELP solr_ping See following URL: https://solr.apache.org/guide/solr/latest/deployment-guide/ping.html
solr_ping{base_url="http://localhost:8983/solr",core="collection1"} 1.0

Prometheus Configuration

Prometheus is a separate server that you need to download and deploy. More information can be found at the Prometheus Getting Started page.

In order for Prometheus to know about the solr-exporter, the listen address must be added to the Prometheus server’s prometheus.yml configuration file, as in this example:

scrape_configs:
  - job_name: 'solr'
    static_configs:
      - targets: ['localhost:9854']

If you already have a section for scrape_configs, you can add the job_name and other values in the same section.

When you apply the settings to Prometheus, it will start to pull Solr’s metrics from solr-exporter.

You can test that the Prometheus server, solr-exporter, and Solr are working together by browsing to http://localhost:9090 and doing a query for solr_ping metric in the Prometheus GUI:

Figure 2. Prometheus Solr Ping expression

Sample Grafana Dashboard

To use Grafana for visualization, it must be downloaded and deployed separately. More information can be found on the Grafana Documentation site. Grafana consumes data from many sources, including the Prometheus server that you previously set up.

A Grafana sample dashboard is provided in the following JSON file: prometheus-exporter/conf/grafana-solr-dashboard.json. You can place this with your other Grafana dashboard configurations and modify it as necessary depending on any customization you’ve done for the solr-exporter configuration.

| | You can directly import the Solr dashboard via grafana.com by using the Import function with the dashboard id 12456. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

This screenshot shows what it might look like:

Figure 3. Grafana Dashboard