Metrics not sent on close (original) (raw)

After upgrading to latest micrometer 1.5.3 and prometheus-rsocket-proxy 1.0.0 (and Boot 2.3.1) the Timers behavior seems to be broken. We still can receive the LongTaskTimer metrics (probable on part of it) but no the Timer metrics!

It looks like the PrometheusRSocketClient#client.pushAndClose() doesn't flush the metrics on exist. If after the pushAndClose() the application waits for ~10 sec before it exit, the Timer metrics get sent!

The task-demo-metrics-prometheus can be used to reproduce and debug the problem (pls ignore the outdated readme). The Main.java is meant to use the plain Micrometer API while the
TaskDemoMetricsApplication is a demo Spring Cloud Task, that internally uses Spring Batch. The Spring Cloud Task uses Timers and LongTaskTimer metrics.

  1. use the provided docker-compose-prometheus.yml to instantiate test MySQL Db, SCDF server , prometheus rsocket proxy and prometheus DB.
docker-compose -f ./docker-compose-prometheus.yml up

Then use the Prometheus UI (http://localhost:9090) to explore the submitted metrics from the test and form the rsocket proxy. 2. Simple test
The Main.java test uses the plain Micrometer API to create a Timer called tzolov.task and LongTaskTimer called tzolov.long and connects the to the rsocket proxy over the 7001 port. When you run the Main.java the prometheus_proxy_scrape_active_connections shows a connection established between the app and the proxy and the tzolov.long (e.g. LongTaskTimer) metrics are received but the tzolov.task (e.g. Timer) metrics are never sent. This is visible on the following snapshot:

MaintTest

But If you uncomment the Thread.sleep(10000) after the pushAndClose the tzolov.task metrics will eventually reach the server.

3 Test Spring Cloud Task
Build and start the TaskDemoMetricsApplication. the build in application.properties will ensure it is connected to the provisioned MySQL and prometheus-proxy. Similarly with the previous example only the spring_cloud_task_active_ metrics (e.g. LongTaskTimer) are sent with no signs of the spring_cloud_task_ metrics.

SCT