Number of datapoints between 2 timestamp without aggregation

Hi ​@RAMBOURG Pierre!

Today we dont have a dedicated count() method that returns a single value without specifying granularity, but there are ways to compute it client side. Before recommending anything, could you help me understand:

• What are you ultimately trying to do with the count? For example, are you deciding between plotting raw vs. aggregated data, sizing a request, or something else?
• Do you need it to be exact, or would an approximation be enough to make the decision?

The reason I am askins is because fetching a count aggregate costs basically the same as fetching count plus the other aggregates, so if the goal is choosing an aggregation strategy, a "count first, then fetch" flow is often slower than just eagerly fetching what you need in parallel and discarding the rest.

Hi ​@Everton Colling.

Thank you for your reply. Our challenge is to choose the best visualisation depending on the number of data points.

If a threshold is exceeded within a given time interval, then we perform an aggregation (min/max or average – we’re not sure yet).

The range of density in the time series is vast; some may have data points every second, whilst others may have just a few points per day.

The strategy that seems to be emerging, therefore, would be to determine the number of data points within the desired time interval in order to decide whether to apply aggregation, as it does not seem possible to have raw data alongside aggregated data. In my view, it is pointless to fetch thousands of data points only to change strategy partway through, isn’t it?

Hi ​@RAMBOURG Pierre!

That makes sense, and it's actually the same problem the Charting components in Cognite Data Fusion user interfaces are solving when deciding whether to render raw datapoints or aggregates. Here's the approach I would suggest you can use today:

As I mentioned before, instead of a separate count request followed by a data fetching request, fetch the aggregates eagerly (including count), then decide. The count aggregate comes back in the same response as average/min/max, so if a given time series turns out to be dense you already have what you need to plot, with no second request. You only have to do a second request in the sparse case, where you go back for raw data.

This would look something like this:

threshold = 10000          # threshold to switch between raw and aggregates
number_datapoints = 2000   # how many points you want to plot when using aggregates

1. Calculate granularity
   granularity = (end - start) / number_datapoints

2. Fetch aggregates for the time series
   aggregates = fetch_agg(start, end, granularity, aggregates=[count, average, min, max])

3. Sum count across all buckets
   total_count = sum(bucket.count for bucket in aggregates)

4. Decide what to render
   if total_count > threshold:
       render aggregates        # already available, no extra request
   else:
       raw = fetch_raw(start, end, include_outside_points=True)
       render raw

With this approach, you don't fetch thousands of raw points first and then change strategy. The aggregate-first request is cheap and has dual purpose. It both tells you the count and gives you a plottable result if you end up needing aggregates. Raw data is only fetched once you know the series is sparse enough to require going for raw data points.

For now you will have to manage this logic yourself, but I agree this is a common enough need for anyone plotting time series data which makes it worth considering as a first-class SDK feature. I'll discuss it internally to evaluate adding it to the Python and JavaScript SDKs in the coming future.