Skip to main content
K
Committed ⭐️⭐️⭐️
Khalfan
Committed ⭐️⭐️⭐️
October 14, 2024

Data aggregation for the online data

  • October 14, 2024
  • 4 replies
  • 85 views

dear all 

I was attempting to perform data aggregation based on the date. I am retrieving online data into CDF, which is updated every 2-3 minutes. I am trying to aggregate the data so that the date is updated every 24 hours instead of every 2-4 minutes.
I used this code to obtain the list of columns in my data frame.

# Check the structure of the DataFrame, including column names and the first few rows
print(dp.columns)  # This will show all column names
dp.head()          # This will show the first few rows of the DataFrame 

however, I got only this after running the code 

I tried to use another code again the first column is not date column. 

 

    4 replies

    matiasholte
    Practitioner ⭐️⭐️⭐️
    matiasholte
    Backend developer
    October 15, 2024

    Hi
    Yes, you can aggregate based on the date!
    https://cognite-sdk-python.readthedocs-hosted.com/en/latest/time_series.html#cognite.client._api.datapoints.DatapointsAPI.retrieve_dataframe

    If you set the granularity parameter to “1d”, and the aggregates parameter to the kind of aggregate(s) you want, you will retrieve daily data points aggregates instead of every individual data point. There are also a few examples in the SDK documentation I linked to above.

    By default, it will group the data from midnight to next midnight in the UTC time zone, but this can be changed using the timezone parameter.
    Hope this helps :)

    Matias

      K
      Committed ⭐️⭐️⭐️
      KhalfanAuthor
      Committed ⭐️⭐️⭐️
      October 16, 2024

      Hi Matias

      Thanks for advise. I followed the instruction provided in the given link and it works properly. 

       

      from datetime import datetime, timezone
      from cognite.client.data_classes import DatapointsQuery
      dp = client.time_series.data.retrieve_dataframe(
          external_id=[
              DatapointsQuery(external_id="XXXX:i=3568", aggregates="discrete_variance")
          ],
          granularity="1d",

          uniform_index=True) 

      best regards 

      khalfan 

       

        K
        Committed ⭐️⭐️⭐️
        KhalfanAuthor
        Committed ⭐️⭐️⭐️
        October 16, 2024

        Hi 

        I faced another challenge. I am trying to insert the created data frame into CDF Time Series. As I understand you need to set index for date as it is shown in the code below

        df.set_index(["Date"], inplace=True)

        However, the data I have are online data and there is no title in the column of data 

        Appreciate if you could support me on this regards 

        R
        Seasoned ⭐️⭐️⭐️
        Rajendra Pasupuleti
        Seasoned ⭐️⭐️⭐️
        October 16, 2024

        As you are looking for inserting timeseries data. Please check if below link helps. 

         

         

        Terms & ConditionsAccessibility statement