Loading paginated data into Grafana (original) (raw)

I’m using one of the two datasources:

Both can call HTTP API endpoints.
The problem is the endpoint returns paginated response (JSON)… My panel will be a very simple table that just shows the endpoint’s response to the query (HTTP Request).
What I want to achieve is making Grafana grab the paginated data…
Example: When I click on page 1 it make a request with page_size=X&page_number=1 and so on… Is that possible ?
If not possible… what is the best thing to do here?
Should I use some kind of caching ? What cache suits best with my case and how to implement it?
I’ve been digging but I didn’t really come to something useful.

yosiasz June 23, 2022, 1:00pm 2

Can you know ahead of time how many pages are returned or total sum of records?

  1. One way is build a comma delimited variable 1 to pageNumbers and provide it as a drop down
  2. Build a table visualization that is built with data from the api itself. When you click on it, it updates the other visual via data links
  3. Uae telegraf to harvest the data via python? Sounds overkill

ebram96 June 24, 2022, 9:57am 3

Thank you a lot for your reply…

  1. I can make a call to the API to return the number of pages… But, how can I use that returned number as a drop-down selector ??
  2. Any guide/tutorial on how to implement this ? (I’m sorry, I did search but I couldn’t really find how it can be implemented)
  3. I’ll search more about pytelegraf and how it can be done.

yosiasz June 24, 2022, 1:06pm 4

  1. Dashboard variable and plug in that api or manually provide 1,2,3,4 etc
  2. See #1 :wink:

Years later and still a very poor implementation. There is a pagination “alpha” in “yesoreyeram-infinity-datasource”.
Yet, I filter the data based on from-to. Whenever there is less than 5 pages of data and I asked for 5 pages, instead of handling properly the empty pages, I get a nasty error. This makes the pagination useless: you must always have more data than the pages you asked for. There should at least be a proper handling of “no data” on page ‘x’.
The API limits to batches of 100 records. If the request produces 0-100, I can see without paging and all is fine.
Else, as soon as I use paging I have issues with smaller timewindows.
There is no real option, as I guess that if I use several panels and try to join them, as soon as one will have a “no data” it will break…