Skip to main content
Solved

Graphql query to retrieve data points


Hi,

We are using graphql query to get data points in CDF function. As I understand we could retrieve 1,00000 data points at a time if we provide the limit. Here is the sample query:

query MyQuery {
  listProductionData(
    filter: {externalId: {in: ["wellbore1"]}}
  ) {
    items {
      space
      data {
        Oil {
          1_day {
            dataPoints(end: 2240591400000, limit: 100000, start: 1672338600000) {
              value
              timestamp
            }
          }
        }
      }
    }
  }
}

I tried above query and I can get all the daily datapoints present in the range from  2023-01-01 to 2040-12-31. 

 

But when I tried below query :

 

query MyQuery {
  listProductionData(
    filter: {externalId: {in: ["wellbore1"]}}
  ) {
    items {
      space
      data {
        Oil {
          1_day {
            getDataPoints(start: 1672338600000, end: 2240591400000) {
              items {
                timestamp
                value
              }
            }
          }
        }
      }
    }
  }
}

we can get only 10 data points at a time. I don’t see any limit option when we use getDataPoints. Can you please help to understand the difference between dataPoints and getDataPoints? Any reason that limit option is not part of getDataPoints. Which one is safe to use considering your deprecation strategy? 

We are dealing with only daily and monthly data points so we don’t want to use cursor and it will be helpful to get all the data points at once with limit option and also as this will be executed in Cognite function in CDF workflow, we don’t want to track the cursors. 

Also please note, we don’t want to use SDK to get data points as our destination model matches with Graphql response structure so we thought to use Graphql result as is without doing lot of conversion needed to match with destination schema. 

 

Thanks. 

 

 

 

 

 

Best answer by Everton Colling

Hi Niranjan,

I understand that you're trying to retrieve data points using GraphQL queries in CDF and encountering some confusion between the dataPoints and getDataPoints methods. Let's address your concerns.

Differences Between dataPoints and getDataPoints

1. Deprecation and Availability:

  • The dataPoints method is going to be deprecated soon. It already does not appear in code completion in our query builder in Fusion.
    Code completion in GraphQL query builder in Fusion

     

  • The getDataPoints method was introduced to replace dataPoints and align with the timeseries/data/list endpoint.

2. Features and Pagination:

  • dataPoints does not offer pagination and is not unit aware, Since it’s response does not match with the timeseries/data/list endpoint, it was not possible to add new features from the timeseries API. Until it’s officially deprecated, it will continue to work, but it won’t get any new features.
  • getDataPoints supports pagination and unit conversion, making it a more robust and future-proof option. New features from the timeseries API should be added to this method.

Using getDataPoints Effectively

The getDataPoints method includes some new arguments that are not available in dataPoints, such as first and after, which handle pagination as per GraphQL specifications. Here’s a explanation of the arguments:

  • first: Equivalent to the limit argument in the dataPoints method. It specifies the number of data points to retrieve.
  • after: Equivalent to the cursor in the timeseries/data/list endpoint. It helps in fetching the next set of data points for pagination.

Here's an example of how to use the getDataPoints method with these arguments:

query MyQuery {
  listProductionData(
    filter: {externalId: {in: ["wellbore1"]}}
  ) {
    items {
      space
      data {
        Oil {
          1_day {
            getDataPoints(first: 100000, start: 1672338600000, end: 2240591400000) {
              unitExternalId
              items {
                timestamp
                value
              }
              pageInfo {
                hasNextPage
                endCursor
              }
            }
          }
        }
      }
    }
  }
}

Example Response and Pagination

The response will include a pageInfo section, which provides information about pagination:

{
  "data": {
    "listProductionData": {
      "items": [
        {
          "space": "some_space",
          "data": {
            "Oil": {
              "1_day": {
                "getDataPoints": {
                  "unitExternalId": "pressure:pa",
                  "items": [
                    {
                      "timestamp": "2023-01-01T00:00:00Z",
                      "value": 8577543.04019
                    },
                    // additional data points
                  ],
                  "pageInfo": {
                    "hasNextPage": true,
                    "endCursor": "Nd3t3/SiQkA="
                  }
                }
              }
            }
          }
        }
      ]
    }
  }
}

To fetch the next set of data points, use the endCursor value from the response in the after argument:

query MyQuery {
  listProductionData(
    filter: {externalId: {in: ["wellbore1"]}}
  ) {
    items {
      space
      data {
        Oil {
          1_day {
            getDataPoints(first: 100000, start: 1672338600000, end: 2240591400000, after: "Nd3t3/SiQkA=") {
              items {
                timestamp
                value
              }
              pageInfo {
                hasNextPage
                endCursor
              }
            }
          }
        }
      }
    }
  }
}

Summary

  • Avoid using dataPoints: It will be deprecated soon, lacks pagination, and is not unit aware.
  • Use getDataPoints: It supports pagination, unit conversion, and aligns with the latest API specifications.

We recognize that the current documentation is not ideal, and we will be working to improve it. In the meantime, you can refer to this example query with unit conversion for more details.

View original
Did this topic help you find an answer to your question?

2 replies

Everton Colling
Seasoned Practitioner
Forum|alt.badge.img
  • Seasoned Practitioner
  • 168 replies
  • Answer
  • May 22, 2024

Hi Niranjan,

I understand that you're trying to retrieve data points using GraphQL queries in CDF and encountering some confusion between the dataPoints and getDataPoints methods. Let's address your concerns.

Differences Between dataPoints and getDataPoints

1. Deprecation and Availability:

  • The dataPoints method is going to be deprecated soon. It already does not appear in code completion in our query builder in Fusion.
    Code completion in GraphQL query builder in Fusion

     

  • The getDataPoints method was introduced to replace dataPoints and align with the timeseries/data/list endpoint.

2. Features and Pagination:

  • dataPoints does not offer pagination and is not unit aware, Since it’s response does not match with the timeseries/data/list endpoint, it was not possible to add new features from the timeseries API. Until it’s officially deprecated, it will continue to work, but it won’t get any new features.
  • getDataPoints supports pagination and unit conversion, making it a more robust and future-proof option. New features from the timeseries API should be added to this method.

Using getDataPoints Effectively

The getDataPoints method includes some new arguments that are not available in dataPoints, such as first and after, which handle pagination as per GraphQL specifications. Here’s a explanation of the arguments:

  • first: Equivalent to the limit argument in the dataPoints method. It specifies the number of data points to retrieve.
  • after: Equivalent to the cursor in the timeseries/data/list endpoint. It helps in fetching the next set of data points for pagination.

Here's an example of how to use the getDataPoints method with these arguments:

query MyQuery {
  listProductionData(
    filter: {externalId: {in: ["wellbore1"]}}
  ) {
    items {
      space
      data {
        Oil {
          1_day {
            getDataPoints(first: 100000, start: 1672338600000, end: 2240591400000) {
              unitExternalId
              items {
                timestamp
                value
              }
              pageInfo {
                hasNextPage
                endCursor
              }
            }
          }
        }
      }
    }
  }
}

Example Response and Pagination

The response will include a pageInfo section, which provides information about pagination:

{
  "data": {
    "listProductionData": {
      "items": [
        {
          "space": "some_space",
          "data": {
            "Oil": {
              "1_day": {
                "getDataPoints": {
                  "unitExternalId": "pressure:pa",
                  "items": [
                    {
                      "timestamp": "2023-01-01T00:00:00Z",
                      "value": 8577543.04019
                    },
                    // additional data points
                  ],
                  "pageInfo": {
                    "hasNextPage": true,
                    "endCursor": "Nd3t3/SiQkA="
                  }
                }
              }
            }
          }
        }
      ]
    }
  }
}

To fetch the next set of data points, use the endCursor value from the response in the after argument:

query MyQuery {
  listProductionData(
    filter: {externalId: {in: ["wellbore1"]}}
  ) {
    items {
      space
      data {
        Oil {
          1_day {
            getDataPoints(first: 100000, start: 1672338600000, end: 2240591400000, after: "Nd3t3/SiQkA=") {
              items {
                timestamp
                value
              }
              pageInfo {
                hasNextPage
                endCursor
              }
            }
          }
        }
      }
    }
  }
}

Summary

  • Avoid using dataPoints: It will be deprecated soon, lacks pagination, and is not unit aware.
  • Use getDataPoints: It supports pagination, unit conversion, and aligns with the latest API specifications.

We recognize that the current documentation is not ideal, and we will be working to improve it. In the meantime, you can refer to this example query with unit conversion for more details.


Dilini Fernando
Seasoned Practitioner
Forum|alt.badge.img+2

Hi @Niranjan Madhukar Karvekar,

I hope the above helped. As of now, I'm closing this topic for now. Please don't hesitate to reply to post if you have any further questions.
 


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings