detNet RestAPI Cognite SDK example [Community Contributed]

Hmmm … Interesting … The code throws an exception with Granularity of 2h, 30m, 1m. When I changed back to Granularity = 1h that used to work before, it throws the same exception as well!!

It returns null for dps where dps is;

var dps = ts.AggregateDatapoints.Datapoints.Select(
                dp => new
                {
                    DateTimeOffset.FromUnixTimeMilliseconds(dp.Timestamp).DateTime,
                    dp.Average
                });

And says;

System.NullReferenceException: 'Object reference not set to an instance of an object.'

Com.Cognite.V1.Timeseries.Proto.DataPointListItem.AggregateDatapoints.get returned null.
 

Is this because Cognite does not receive this data from the source when I am running the code? Because this worked for 1h granularity before with no problem.

Best regards.

So, what’s probably happening is that there’s no datapoints inside of the time window you are sending in the request for the selected time series. In these situations the AggregateDatapoints object is null (empty list of Datapoints) and the code above will fail.

You can catch such a scenario by checking for null references before accessing the Select method in the LINQ query. One way to do this is to use the null-conditional operator (?.) to check if ts.AggregateDatapoints is not null before accessing its Datapoints property.

Here's a minimal example on how to deal with these cases:

var dps = ts.AggregateDatapoints?.Datapoints?.Select(
    dp => new
    {
        DateTimeOffset.FromUnixTimeMilliseconds(dp.Timestamp).DateTime,
        dp.Average
    }
);
if (dps == null)
{
    throw new Exception("No datapoints were available in the selected time window.");
}

In this example, the ?. operator checks if ts.AggregateDatapoints is not null before accessing its Datapoints property. If either ts.AggregateDatapoints or Datapoints is null, the Select method is not called and dps will be null.

After the LINQ query, you can check if dps is null and throw an exception if it is. You can customize the error message in the Exception constructor to provide more specific information about the error. If you prefer, you can also just print to the console instead of throwing an error (it depends on your use case).

You can also check the timestamp when the latest datapoint is available for a given time series by accessing the DataPoints.LatestAsync method.

Here's a minimal example on how to check the timestamp of the latest datapoint available in Cognite Data Fusion for a given time series (specified as tsExternalId):

var resLatestDps = await client.DataPoints.LatestAsync(new DataPointsLatestQuery
    {
        Items = new List<IdentityWithBefore> {new IdentityWithBefore(tsExternalId, "now")}
    }
);
var latestTimeStamp = resLatestDps.First().DataPoints?.First().Timestamp;
if (latestTimeStamp != null)
{
    var latestTimeStampString = DateTimeOffset.FromUnixTimeMilliseconds(
        latestTimeStamp.Value
    ).DateTime;
    Console.WriteLine(
        $"Latest data point found at {latestTimeStampString} for time series {tsExternalId}\n"
    );
}

Hi,

Thanks. That was right, there was no datapoints in the time interval selected.

But now the issue after selecting a new time interval is that, when I selected Granularity of “5m” and then “10s”, the timestamp that I get data in always 10 minutes. Why can’t I get data with the same granularity that I want? I have selected an specific externalID for this.

Best regards.

The aggregates are calculated on top of the original data that was ingested from the source system to Cognite Data Fusion (CDF). As described on the documentation, CDF doesn't return aggregates or interpolations for time ranges that have no data points.

For example, if the original data contains data points with a 1s granularity (every second a new data point is being measured and stored), there should be no issues to request aggregates with 10s, 1m, 10m, 1h granularities. However if the original data contains data points with a 1m granularity, asking for a granularity of 10s would still yield a new value only on a 1m granularity.

If you need to get a value for each 10s, you will need to upsample your data manually by implementing some sort of interpolation.

Aggregates are convenient calculations that help to deal with high amounts of data, but if you want you can also take a look at the original data that was ingested into CDF by passing null to both Aggregates and Granularity. Here’s a minimal example to request the original data points that were ingested into CDF:

var resDps = await client.DataPoints.ListAsync(new DataPointsQuery
{
    Start = "2d-ago",
    End = "now",
    Items = new List<DataPointsQueryItem>
    {
        new DataPointsQueryItem
        {
            ExternalId = tsExternalId,
            Aggregates = null,
            Granularity = null,
            Limit = 10_000
        }
    }
});
var ts = resDps.Items[0];

var dps = ts.NumericDatapoints?.Datapoints?.Select(
    dp => new
    {
        DateTimeOffset.FromUnixTimeMilliseconds(dp.Timestamp).DateTime,
        dp.Value
    });
if (dps == null)
    throw new Exception("No data points were located in the selected time window.");

Console.WriteLine($"Data fetched from time series {tsExternalId}:");
foreach (var dp in dps)
    Console.WriteLine(dp);

Note that with this request we are dealing with NumericDatapoints now and not AggregateDatapoints anymore, also the datapoints does not contain the aggregate anymore, but instead refer to the Value property.

HI @samirhaj, checking in to see if you need more information from us on the above, or if you’re all set? 

@Everton Colling For the follow up of the previous questions and as a new user I have two questions now;

• Could we use the Cognite C# Sdk as a common Rest Api Sdk in general to have RestApi client (similar to what RestSharp does for exampl) or is the Cognite Sdk is designed to access and retreive data from Cognite/CDF?
• What is the difference between Cognite C# Sdk and Cognite Python Sdk? Does both have more or less functionality built-in? When using the Python sdk, once the token is authenticated to access CDF, the message is you are connecting to “Cognite-Postman” but there is not such message in the C# sdk. Is this because with Python you could use Postman for data analytics? Moreover most of the data processing that Postman does also exists in CDF like aggregation, averaging, etc. Are there specific functionalities that Postman does have and CDF doesn’t? 

Best regads.

Could we use the Cognite C# Sdk as a common Rest Api Sdk in general to have RestApi client (similar to what RestSharp does for exampl) or is the Cognite Sdk is designed to access and retreive data from Cognite/CDF?

The Cognite .NET SDK was built to add convenience to developers while creating solutions that need to call endpoints exposed by the CDF API.

What is the difference between Cognite C# Sdk and Cognite Python Sdk? Does both have more or less functionality built-in? When using the Python sdk, once the token is authenticated to access CDF, the message is you are connecting to “Cognite-Postman” but there is not such message in the C# sdk. Is this because with Python you could use Postman for data analytics? Moreover most of the data processing that Postman does also exists in CDF like aggregation, averaging, etc. Are there specific functionalities that Postman does have and CDF doesn’t? 

The Cognite .NET SDK is a Community SDK (you can check the other available SDKs here), and as such is not officially supported as a Product. It does not contain implementation for all endpoints available in CDF and updates are made by the community. On the other hand, the Python SDK is an official SDK that is actively maintained by Cognite and covers a wider range of Cognite API endpoints. You can also connect to CDF API using Postman, check the link above for more details.

@Everton Colling Can you please help on how we can pass the Data set id in the above requests.

DataSets can be queried like any other CDF resources. Here is an example of a query to list DataSets:

var res = await client.DataSets.ListAsync(
    new DataSetQuery{Limit=10}
);

DataSets can also be used to filter queries to CDF. Each DataSet has an Id and ExternalId properties that can be used as filtering arguments. Here's an example on how to filter a TimeSeries query with a DataSetId:

var res = await client.TimeSeries.ListAsync(
    new TimeSeriesQuery{
        Filter = new TimeSeriesFilter{
            Unit="barg",
            DataSetIds=new List<Identity>{ 
                new Identity(2452112635370053) 
            }
        },
        Limit=2
    }
);

Note that the DataSetIds argument of the Filter accepts a list of objects of the type CogniteSdk.Identity. The Identity class is used to set either Id or ExternalId. In the example above I used the Id to reference the desired DataSetId.

@Everton Colling I will try it out. Appreciate your help.