Support meters with the same name but different set of tag keys in PrometheusMeterRegistry (original) (raw)

In connection with PrometheusMeterRegistry and the new KafkaConsumerMetrics (thanks to @jkschneider) I have a problem that metrics won´t register correctly because of PrometheusMeterRegistry complaining about metrics with same name but different set of key tags. It claims that metrics of same name are required in prometheus to have same set of tags.

I believe the error thrown at https://github.com/micrometer-metrics/micrometer/blob/master/implementations/micrometer-registry-prometheus/src/main/java/io/micrometer/prometheus/PrometheusMeterRegistry.java#L357 is wrong. Prometheus perfectly accepts metrics of same name with disjunct set of tags (the metrics name itself is just a tag __name__). We have in our prometheus production instance metrics like up{application="xyz", job="abc"} and up{application="abc", job="123", host="prometheus"} coexisting without any issue whatsoever.

Example of exception text, stacktrace below:

Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter containing tag keys [application, client_id]. The meter you are attempting to register has keys [application, client_id, topic].

Context of variables

id = MeterId{name='kafka.consumer.records.lag.max', tags=[ImmutableTag{key='application', value='heimdall'}, ImmutableTag{key='client.id', value='consumer-1'}, ImmutableTag{key='topic', value='probe_t'}]}
existingCollector = MeterId{name='kafka.consumer.records.lag.max', tags=[ImmutableTag{key='application', value='heimdall'}, ImmutableTag{key='client.id', value='consumer-1'}]}

Stacktrace:

collectorByName:348, PrometheusMeterRegistry (io.micrometer.prometheus)
newGauge:225, PrometheusMeterRegistry (io.micrometer.prometheus)
lambda$gauge$1:244, MeterRegistry (io.micrometer.core.instrument)
apply:-1, 4552198 (io.micrometer.core.instrument.MeterRegistry$$Lambda$373)
lambda$registerMeterIfNecessary$5:514, MeterRegistry (io.micrometer.core.instrument)
apply:-1, 491928400 (io.micrometer.core.instrument.MeterRegistry$$Lambda$338)
getOrCreateMeter:563, MeterRegistry (io.micrometer.core.instrument)
registerMeterIfNecessary:525, MeterRegistry (io.micrometer.core.instrument)
registerMeterIfNecessary:514, MeterRegistry (io.micrometer.core.instrument)
gauge:244, MeterRegistry (io.micrometer.core.instrument)
register:128, Gauge$Builder (io.micrometer.core.instrument)
registerGaugeForObject:149, KafkaConsumerMetrics (io.micrometer.core.instrument.binder.kafka)
registerGaugeForObject:153, KafkaConsumerMetrics (io.micrometer.core.instrument.binder.kafka)
lambda$bindTo$0:79, KafkaConsumerMetrics (io.micrometer.core.instrument.binder.kafka)
accept:-1, 1260340889 (io.micrometer.core.instrument.binder.kafka.KafkaConsumerMetrics$$Lambda$328)
lambda$registerMetricsEventually$11:205, KafkaConsumerMetrics (io.micrometer.core.instrument.binder.kafka)
handleNotification:-1, 820539250 (io.micrometer.core.instrument.binder.kafka.KafkaConsumerMetrics$$Lambda$329)
handleNotification:1754, DefaultMBeanServerInterceptor$ListenerWrapper (com.sun.jmx.interceptor)
handleNotification:275, NotificationBroadcasterSupport (javax.management)
run:352, NotificationBroadcasterSupport$SendNotifJob (javax.management)
execute:337, NotificationBroadcasterSupport$1 (javax.management)
sendNotification:248, NotificationBroadcasterSupport (javax.management)
sendNotification:209, MBeanServerDelegate (javax.management)
sendNotification:1498, DefaultMBeanServerInterceptor (com.sun.jmx.interceptor)
registerWithRepository:1911, DefaultMBeanServerInterceptor (com.sun.jmx.interceptor)
registerDynamicMBean:966, DefaultMBeanServerInterceptor (com.sun.jmx.interceptor)
registerObject:900, DefaultMBeanServerInterceptor (com.sun.jmx.interceptor)
registerMBean:324, DefaultMBeanServerInterceptor (com.sun.jmx.interceptor)
registerMBean:522, JmxMBeanServer (com.sun.jmx.mbeanserver)
reregister:167, JmxReporter (org.apache.kafka.common.metrics)
metricChange:85, JmxReporter (org.apache.kafka.common.metrics)
registerMetric:545, Metrics (org.apache.kafka.common.metrics)
add:256, Sensor (org.apache.kafka.common.metrics)
add:241, Sensor (org.apache.kafka.common.metrics)
recordTopicFetchMetrics:1291, Fetcher$FetchManagerMetrics (org.apache.kafka.clients.consumer.internals)
access$3200:1246, Fetcher$FetchManagerMetrics (org.apache.kafka.clients.consumer.internals)
record:1230, Fetcher$FetchResponseMetricAggregator (org.apache.kafka.clients.consumer.internals)
drain:982, Fetcher$PartitionRecords (org.apache.kafka.clients.consumer.internals)
nextFetchedRecord:1033, Fetcher$PartitionRecords (org.apache.kafka.clients.consumer.internals)
fetchRecords:1095, Fetcher$PartitionRecords (org.apache.kafka.clients.consumer.internals)
access$1200:949, Fetcher$PartitionRecords (org.apache.kafka.clients.consumer.internals)
fetchRecords:570, Fetcher (org.apache.kafka.clients.consumer.internals)
fetchedRecords:531, Fetcher (org.apache.kafka.clients.consumer.internals)
pollOnce:1178, KafkaConsumer (org.apache.kafka.clients.consumer)
poll:1111, KafkaConsumer (org.apache.kafka.clients.consumer)
run:699, KafkaMessageListenerContainer$ListenerConsumer (org.springframework.kafka.listener)
call:511, Executors$RunnableAdapter (java.util.concurrent)
run:266, FutureTask (java.util.concurrent)
run:748, Thread (java.lang)