Hi experts,
In order to decide whether to use an aggregation strategy (or no aggregation at all), I need to know the number of data points between two timestamps without aggregation (server-side calculation).
I’m looking for this code of call :
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
client.time_series.data.count(external_id=<str>, start=<int | datetime>, end=<int | datetime>) -> intI'm trying to avoid having to perform a granularity calculation
Thanks !
Regards,
Pierre
Best answer by Everton Colling
Hi
That makes sense, and it's actually the same problem the Charting components in Cognite Data Fusion user interfaces are solving when deciding whether to render raw datapoints or aggregates. Here's the approach I would suggest you can use today:
As I mentioned before, instead of a separate count request followed by a data fetching request, fetch the aggregates eagerly (including count), then decide. The count aggregate comes back in the same response as average/min/max, so if a given time series turns out to be dense you already have what you need to plot, with no second request. You only have to do a second request in the sparse case, where you go back for raw data.
This would look something like this:
threshold = 10000 # threshold to switch between raw and aggregates
number_datapoints = 2000 # how many points you want to plot when using aggregates
1. Calculate granularity
granularity = (end - start) / number_datapoints
2. Fetch aggregates for the time series
aggregates = fetch_agg(start, end, granularity, aggregates=[count, average, min, max])
3. Sum count across all buckets
total_count = sum(bucket.count for bucket in aggregates)
4. Decide what to render
if total_count > threshold:
render aggregates # already available, no extra request
else:
raw = fetch_raw(start, end, include_outside_points=True)
render rawWith this approach, you don't fetch thousands of raw points first and then change strategy. The aggregate-first request is cheap and has dual purpose. It both tells you the count and gives you a plottable result if you end up needing aggregates. Raw data is only fetched once you know the series is sparse enough to require going for raw data points.
For now you will have to manage this logic yourself, but I agree this is a common enough need for anyone plotting time series data which makes it worth considering as a first-class SDK feature. I'll discuss it internally to evaluate adding it to the Python and JavaScript SDKs in the coming future.
Hi
Today we dont have a dedicated count() method that returns a single value without specifying granularity, but there are ways to compute it client side. Before recommending anything, could you help me understand:
The reason I am askins is because fetching a count aggregate costs basically the same as fetching count plus the other aggregates, so if the goal is choosing an aggregation strategy, a "count first, then fetch" flow is often slower than just eagerly fetching what you need in parallel and discarding the rest.
Hi
Thank you for your reply. Our challenge is to choose the best visualisation depending on the number of data points.
If a threshold is exceeded within a given time interval, then we perform an aggregation (min/max or average – we’re not sure yet).
The range of density in the time series is vast; some may have data points every second, whilst others may have just a few points per day.
The strategy that seems to be emerging, therefore, would be to determine the number of data points within the desired time interval in order to decide whether to apply aggregation, as it does not seem possible to have raw data alongside aggregated data. In my view, it is pointless to fetch thousands of data points only to change strategy partway through, isn’t it?
Hi
That makes sense, and it's actually the same problem the Charting components in Cognite Data Fusion user interfaces are solving when deciding whether to render raw datapoints or aggregates. Here's the approach I would suggest you can use today:
As I mentioned before, instead of a separate count request followed by a data fetching request, fetch the aggregates eagerly (including count), then decide. The count aggregate comes back in the same response as average/min/max, so if a given time series turns out to be dense you already have what you need to plot, with no second request. You only have to do a second request in the sparse case, where you go back for raw data.
This would look something like this:
threshold = 10000 # threshold to switch between raw and aggregates
number_datapoints = 2000 # how many points you want to plot when using aggregates
1. Calculate granularity
granularity = (end - start) / number_datapoints
2. Fetch aggregates for the time series
aggregates = fetch_agg(start, end, granularity, aggregates=[count, average, min, max])
3. Sum count across all buckets
total_count = sum(bucket.count for bucket in aggregates)
4. Decide what to render
if total_count > threshold:
render aggregates # already available, no extra request
else:
raw = fetch_raw(start, end, include_outside_points=True)
render rawWith this approach, you don't fetch thousands of raw points first and then change strategy. The aggregate-first request is cheap and has dual purpose. It both tells you the count and gives you a plottable result if you end up needing aggregates. Raw data is only fetched once you know the series is sparse enough to require going for raw data points.
For now you will have to manage this logic yourself, but I agree this is a common enough need for anyone plotting time series data which makes it worth considering as a first-class SDK feature. I'll discuss it internally to evaluate adding it to the Python and JavaScript SDKs in the coming future.
Thank you
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
