Solved

Limited returns from Grafana regex queries

  • 6 February 2024
  • 1 reply
  • 51 views
K
Rank
Userlevel 2
Badge +3

Hi. In our CDF project, we have 19 time series whose external IDs ends with .EUR. Examples:

  • NordPool.Price.Regulationmarket.RegulationDownPrices.NO1.EUR
  • NordPool.Price.Actual.Dayahead.SE4.EUR
  • NordPool.Price.Actual.Dayahead.NO2.EUR

Using the CDF Grafana connector, I would expect that I could extract all of these using a “Time series custom query” like this: 

ts{externalId=~".*EUR"}

However, the query only returns a single time series.

 

The same problem occurs for trying to apply a regex filter on name or on a metadata value. E.g. ts{metadata={timeSeriesCategory=~'.*Lifetime'}} returns only 6 of our 116 “Remaining Lifetime” time series.

Does anyone else have this problem? Is it a bug in the Grafana CDF connector?

icon

Best answer by Everton Colling 6 February 2024, 18:42

View original

1 reply

Everton Colling
Rank
Userlevel 4

Hi Kristian!

The behavior you've described is not a bug in Cognite’s Grafana connector but rather a limitation related to how data filtering is handled. The Grafana connector supports filtering functionality that allow users to perform client-side filtering for regex queries, which is applied after the initial data retrieval from the Cognite Data Fusion (CDF) platform. This initial data retrieval is subject to a hard limit of 1000 items per request, as imposed by the timeseries API. This means that when you perform a regex query, the connector first fetches up to 1000 timeseries from CDF, and then applies the regex pattern to filter these items on the client side.

Given this limitation, when using a broad regex pattern, there is a possibility that the desired timeseries are not included in the initial 1000 items fetched, resulting in fewer matches than expected. This is likely the reason you're observing limited returns for your queries.

In your example you are not performing any server side filtering, which means you are asking for ALL the timeseries in your CDF project (which in your case should be more than a 1000, as you are experiencing missing data), and then filtering out the externalId based on the regex pattern.

To ensure you can reliably retrieve the specific timeseries you need, we recommend utilizing server-side filters wherever possible. Server-side filters, such as exact matches ("=") on externalIdPrefix, assetExternalIds, unit, unitExternalId, unitQuantity, and specific metadata items, are processed by the CDF server itself, allowing you to more effectively narrow down the results within the initial 1000 item limit. This approach significantly increases your chances of retrieving all relevant timeseries for your dashboard.

I understand this is not clearly explained in our documentation for Cognite’s Grafana connector today, so we’ll be working on enhancing the documentation for the connector to make these limitations and recommended practices clearer.

Thank you for bringing this to our attention, and please feel free to reach out if you have any further questions or need additional guidance. We're here to help!

Reply


Terms & ConditionsCookie Settings