Replies posted by Everton Colling

31 Replies

K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Grafana data srource. Event start time format

Hi Kristian,We have released a new version of the Grafana connector, version 4.1.2, which includes a fix for the issue you raised. Now startTime and endTime timestamps are parsed as datetime automatically in Grafana, not needing the transformation step.

2
Everton Colling
Practitioner
5 days ago
A
Seasoned
Adarsh DhimanSeasoned
asked in Developer and User Community

C# SDK upload file

Hi Adarsh,Here's a minimal example on how to upload a file (in this case equation.png) using the dotnet SDK:1) Start by creating a file resource in CDFvar fileDto = new FileCreate { Name = "equation.png", Source = "equation.png", MimeType = "image/png", ExternalId = "dotnet-sdk-test",};var fileCreate = await client.Files.UploadAsync(file: fileDto);The fileCreate object will have an UploadUrl property which you can use to upload your file.2) Use an HttpClient instance to upload the file (in this example, I’m using the same httpClient I used to instantiate the CogniteClient):var fileBytes = System.IO.File.ReadAllBytes("equation.png");var byteContent = new ByteArrayContent(fileBytes);byteContent.Headers.ContentType = new MediaTypeHeaderValue(fileCreate.MimeType);var response = await httpClient.PutAsync(fileCreate.UploadUrl, byteContent);Ensure that the HTTP response status is successful to confirm that the file has been uploaded correctly.3) Once your file is successfully uplo

4
A
Seasoned
8 days ago
A
Seasoned
Adarsh DhimanSeasoned
asked in Developer and User Community

C# SDK read raw data

Hi Adarsh,You’re on the right path by using the ListRowsAsync method from our dotnet SDK to fetch rows from a RAW table. This method allows you to retrieve data from a specified RAW table in a given database.The “project limit reached” error you mentioned indicates that you’ve hit a limit within your CDF project. These limits could relate to concurrency or data transfer rate, being tied to the API itself and independent of the SDK you’re using. I recommend consulting the Cognite API documentation for more information about the specific limits of the RAW service.In a CDF project, it’s common to have multiple workflows running simultaneously, which can contribute to reaching a project-wide cap. Since not all simultaneous jobs run all the time, you might notice these “project limit reached” errors to be intermittent, while the limits are temporarily reached. If you cannot properly control and distribute load between all simultaneous jobs, I recommend adopting retry mechanisms in your app

2
A
Seasoned
25 days ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Grafana data srource. Event start time format

Hi Kristian,Thank you for reaching out and bringing this to our attention. I've looked into the issue you're experiencing with the Grafana connector and the format of the timestamps.You are not doing anything wrong here. After testing it myself, I can confirm that in the latest version of Cognite's Grafana connector, startTime and endTime timestamps are indeed parsed as integers, representing milliseconds since the epoch. This is consistent with how timestamps are returned from CDF APIs—always as integers in this format. It appears that the automatic conversion of these timestamps to a datetime format compatible with Grafana, which was previously in place, is not working anymore. Cognite’s Engineering team will investigate what’s going on and evaluate if we can issue a fix.In the meantime, you can circumvent this issue by applying a simple transformation within Grafana to convert the field type from an integer to a human-readable datetime format. Below is a screenshot demonstrating how

2
Everton Colling
Practitioner
5 days ago
A
Seasoned
Adarsh DhimanSeasoned
asked in Developer and User Community

Row count limit in solution data model container view

Hi Adarsh,In the Cognite documentation, the limit you're asking about is referred to as Live instances. This term refers to the maximum number of individual instances (nodes and edges) you can have across all containers in your CDF project at any given time. The current default limit is set to 5,000,000 instances.To check your current utilization of live instances, you can do so through the Cognite Data Fusion user interface, under Explore → Model your data. On the top right corner you can see the summary of usage of data model storage as you can see in the screenshot below:If you click on see all, you’ll be able to visualize the current limits and the current usage in your CDF project.

4
A
Seasoned
1 month ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Grafana data source. Feature request / alternate solotions?

To help you get started and to make sure my suggestion holds, I’ve taken your example and created a dummy time series to mimic your use case, as below:ts = client.time_series.create( TimeSeriesWrite( external_id="MS-01-SX", name="MS-01 Switch", is_step=True, is_string=False, metadata={ "statusText0": "Inne", "statusText1": "Ute", "statusText2": "00-Stilling", "statusText3": "11-Stilling", } ))Then I populated it with a couple of states:start = datetime.now(tz=timezone.utc) - timedelta(hours=6)datapoints = [ (start, 0), (start + timedelta(minutes=30), 1), (start + timedelta(hours=2), 0), (start + timedelta(minutes=130), 1), (start + timedelta(hours=4), 0), (start + timedelta(hours=5), 1),]client.time_series.data.insert(datapoints, external_id="MS-01-SX")On Grafana, I can now fetch this timeseries using either the Time series search tab or the Time series custom query tab as belo

7
K
Seasoned
1 month ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Grafana data source. Feature request / alternate solotions?

Thanks for sharing more details of your use case Kristian!I see what you’re trying to visualize is what we are calling internally machine states, utilizing metadata from time series to convert numerical values into meaningful state descriptions. Currently, transforming data in this specific way directly in Grafana is not straightforward given the existing functionalities of the CDF Grafana data source plugin.We understand that our current offerings of numerical and string time series does not allow to store the full context of machine states like the ones you described. I’ll point this to my colleague @Glen Sykes, who is the Product Manager responsible for Time Series. Glen has been gathering requirements for a potential extension that would enable a more fitting representation of machine states within CDF.In the meantime, while we work on a more integrated solution, there are a couple of alternative approaches you might consider:Custom Scripting within Grafana: If you're comfortable w

7
K
Seasoned
1 month ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Grafana data source. Feature request / alternate solotions?

Hi Kristian,To better assist you, it would be beneficial if you could provide some examples of the states associated with the integer values, along with how they are currently stored in the metadata field.Additionally, it would be helpful to know how you plan to utilize this information in Grafana. Are you planning to visualize the state text directly on the chart, incorporate it into a table, display it alongside metrics, or perhaps have it visible within the legend? Any further details or visual aids you can provide, such as sketches or screenshots, would be valuable in ensuring that we understand what you are trying to achieve.

7
K
Seasoned
1 month ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Limited returns from Grafana regex queries

Hi Kristian!The behavior you've described is not a bug in Cognite’s Grafana connector but rather a limitation related to how data filtering is handled. The Grafana connector supports filtering functionality that allow users to perform client-side filtering for regex queries, which is applied after the initial data retrieval from the Cognite Data Fusion (CDF) platform. This initial data retrieval is subject to a hard limit of 1000 items per request, as imposed by the timeseries API. This means that when you perform a regex query, the connector first fetches up to 1000 timeseries from CDF, and then applies the regex pattern to filter these items on the client side.Given this limitation, when using a broad regex pattern, there is a possibility that the desired timeseries are not included in the initial 1000 items fetched, resulting in fewer matches than expected. This is likely the reason you're observing limited returns for your queries.In your example you are not performing any server s

1
Everton Colling
Practitioner
2 months ago
T
Active
tabdoolActive
asked in Developer and User Community

Synthetic time series query not supporting IF statements in 'expressions' parameter

Hi Talia!Our current capabilities for synthetic time series do not support the use of if expressions. The supported functions include various mathematical operators, trigonometric functions, and specific functions like ln, pow, sqrt, exp, abs, and map. However, conditional statements like if are not listed among these supported functions. For more detailed information and to explore potential workarounds, you can refer to the Cognite synthetic time series documentation: Synthetic time series documentation.In some cases the usage of functions like max and min can help you achieve the similar outcome as with an if expression, but we do understand that it’s not as flexible. If this is an important feature for you, I recommend making a feature request as a Product Idea here on the hub.

2
M
Practitioner
4 months ago
N
Seasoned
Neerajkumar BhatewaraSchlumberger
asked in Developer and User Community

SDK - Questions around Global Config

Hi @Gargi Bhosale, could you share the exact fetching code (logic) you used to perform the test (besides setting the ClientConfig)?

13
H
Practitioner
3 months ago
P
Committed
Paramale AratiCommitted
asked in Developer and User Community

Real data streaming in Power BI

Here is the tutorial I promised.As I mentioned before, this can be a valid solution to continuously push data from CDF to Power BI with a higher frequency. @Andre Alves  thanks again for the suggestion of using Power BI REST API endpoints.

7
Everton Colling
Practitioner
5 months ago
I
Seasoned
ibrahim.alsyedSeasoned

PowerBI Live connectionParked

Here is the tutorial I promised.As I mentioned before, this can be a valid solution to continuously push data from CDF to Power BI with a higher frequency. @Andre Alves  thanks again for the suggestion of using Power BI REST API endpoints.

9
Everton Colling
Practitioner
5 months ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Documentation on Unit catalog and unit conversions

Hi Kristian,Thank you for your interest in our unit catalog and conversion functionalities. You've indeed spotted a new feature that's currently in a limited availability phase, meaning it's being tested and refined with a smaller user group before its public release.This feature, aimed to facilitate unit conversion, is planned for general availability in our December release. The upcoming release will make it easier to specify units on time series and data models and query data in a compatible unit. As usual, the release will include comprehensive documentation for users like yourself.If you're interested in being part of our early adopters program for this feature, I encourage you to reach out to your Cognite Account Manager.

1
Everton Colling
Practitioner
5 months ago
I
Seasoned
ibrahim.alsyedSeasoned

PowerBI Live connectionParked

You are correct @andrelcalves, it is possible to create a scheduled job and push data from CDF to a push or streaming dataset in Power BI using the Power BI REST API.It's not as simple as creating the report using the CDF Power BI connector, but it is a viable solution. In this case, we are not using OData to transfer data, but instead using a wrapper to fetch data from CDF REST API and pushing data to Power BI REST API.I tested it a while ago just to confirm (followed this tutorial) and it worked fine. I created a simple Cognite Function and scheduled it to push data to Power BI every five minutes and it's been working fine since. I plan to write a tutorial here on the hub sharing how to use this method to stream data to Power BI datasets.

9
Everton Colling
Practitioner
5 months ago
P
Committed
Paramale AratiCommitted
asked in Developer and User Community

Real data streaming in Power BI

You are correct @andrelcalves, it is possible to create a scheduled job and push data from CDF to a push or streaming dataset in Power BI using the Power BI REST API.It's not as simple as creating the report using the CDF Power BI connector, but it is a viable solution. In this case, we are not using OData to transfer data, but instead using a wrapper to fetch data from CDF REST API and pushing data to Power BI REST API.I tested it a while ago just to confirm (followed this tutorial) and it worked fine. I created a simple Cognite Function and scheduled it to push data to Power BI every five minutes and it's been working fine since. I plan to write a tutorial here on the hub sharing how to use this method to stream data to Power BI datasets.

7
Everton Colling
Practitioner
5 months ago
N
Seasoned
Neerajkumar BhatewaraSchlumberger

Rest Endpoint - SDK - Is there any way we can invoke grapqhl rest endpoint via sdkGathering Interest

While this is not officially supported in the Cognite Python SDK today, there is a method to perform post requests in the SDK that allows you to do exactly what you want. Here’s a quick example on how to make a GraphQL request today using the Python SDK.client = CogniteClient()project = client.config.projectexternal_id = "LER001-TI518"now = datetime.utcnow()end = timestamp_to_ms(now)start = timestamp_to_ms(now - timedelta(hours=1))query = {"query": f""" query {{ listShower {{ items {{ name coldWaterTemperature {{ name externalId unit dataPoints(start: {start}, end: {end}, limit: 100000) {{ timestamp value }} }} }} }} }} """}response = client.post(url=f"/api/v1/projects/{project}/userapis/spaces/showerMixer/datamodels/ShowerMixer/versions/1/graphql", json=query)If everything is correct, you can inspect the response and get:<Response [200]>If th

3
C
Seasoned Practitioner
6 months ago
D
Seasoned
Diana Chimnaz JohanSeasoned
asked in Developer and User Community

Deploying cognite functions with requirement file

To deploy a machine learning model using Cognite Data Fusion (CDF), you can train your model locally using a relevant library such as scikit-learn and save the model as a .pickle or .joblib file. Once your model is trained and saved, you can upload this file to the CDF Files API, which makes the file available for predictions inside your function context.Here is a minimal example to exemplify how you could proceed: Local Model Training: Train your model using a relevant library, e.g., scikit-learn. Save the model to a file using the .pickle or .joblib format. import joblibfrom sklearn.linear_model import LinearRegression# Assume X_train and y_train are your datamodel = LinearRegression()model.fit(X_train, y_train)# Save model to a filejoblib.dump(model, "model.joblib") Upload Model to CDF: Upload the saved model file to CDF using the Files API. from cognite.client import CogniteClient# Create a client instanceclient = CogniteClient()# Upload the model filefile = client.files.upl

2
D
Seasoned
6 months ago
R
Active
rupaliActive
asked in Developer and User Community

Cognite Prosper Connector - Simulator Details

Hello Rupali,Thank you for your interest in the PROSPER Connector within Cognite Data Fusion (CDF).Simulator Software & Licensing: While CDF does offer Simulator Integration capabilities, we only provide connectors and a framework for uploading and configuring simulation models. The actual simulator software and its license are not provided by Cognite; you'll need to procure those independently.PROSPER Connector & Others: We have introduced the PROSPER Connector, and more connectors are actively being developed. Keep an eye out for updates!Simulator Details Configuration: The connector is only a part of the simulator integration framework. For uploading models and configuring simulations, you'd use the Simulator Configurator web application. This is where you'd typically provide simulator details or names.Access to Simulator Configurator: As of now, access to the Simulator Configurator web application is limited to a select group of customers participating in our early adopter

2
N
3 months ago
E
Practitioner
eashwar11
asked in Developer and User Community

time series data - calculate time weighted average

Hi Eashwar!What you are looking for is aggregations on time series data. Here’s a simple example of retrieving the average values for periods of 8 hours for the past week using the Python SDK:data = client.time_series.data.retrieve( external_id="FI-5002A.PV", start="1w-ago", end="now", aggregates="average", granularity="8h")For more information on how to retrieve data points from time series, you can check the API documentation or one of the SDK documentations.You can also read more about how aggregation works and the available options on the Cognite developer documentation.

3
Dilini Fernando
Seasoned Practitioner
9 months ago
E
Practitioner
eashwar11
asked in Developer and User Community

CDF Data model

Hi Eashwar! Data models are quite flexible, anything that you represent in RAW can be represented in a data model as well. With data models, you also has the capability to create custom types, associate relationships, dependencies that can simplify the data consumption.We are working on a library of sample data models, to help our customers to get started, but that’s not available today. To help you get started, here’s a very simple example of a crude diet data model that contains time based entries for the crude feed of distillation units. You can easily extend it by adding more properties and relationships according to the data available and the use case requirements.type CrudeAssay { name: String! id: Int! volume: Float volumeFraction: Float mass: Float massFraction: Float}type CrudeDiet { timestamp: Timestamp refinery: String distillationUnit: String diet: [CrudeAssay]}For more information about data models please check the documentation: https://docs.

3
Everton Colling
Practitioner
10 months ago
E
Practitioner
eashwar11
asked in Developer and User Community

CDF Data model

Hi Eashwar! The use case you are mentioning is an extremely valuable one and it’s definitely solvable using Cognite Data Fusion.I see that you are planning to rely on RAW tables, but I would not recommend it for this case. RAW is intended as a staging area, to store data dumped from extractors until it’s transformed into clean data (data stored into high performant CDF resources). I suggest creating a well defined data model for the source data and another one for the “calculated data” to store the multiple attributes and their dependencies. Any time indexed data should be stored into timeseries (that are referenced in the data model).To populate the datamodels with the source data you can use Transformations or Cognite Functions. To populate the calculations (formula that generates the new “tables”) data model, I suggest using Cognite Functions to include error handling/logging/data quality checks.When it comes to the CDF Petro-SIM connector, it’s an early adopter capability that is e

3
Everton Colling
Practitioner
10 months ago
S
Committed
samirhajCommitted
asked in How-To Guides

detNet RestAPI Cognite SDK example [Community Contributed]

DataSets can be queried like any other CDF resources. Here is an example of a query to list DataSets:var res = await client.DataSets.ListAsync(    new DataSetQuery{Limit=10});DataSets can also be used to filter queries to CDF. Each DataSet has an Id and ExternalId properties that can be used as filtering arguments. Here's an example on how to filter a TimeSeries query with a DataSetId:var res = await client.TimeSeries.ListAsync(    new TimeSeriesQuery{        Filter = new TimeSeriesFilter{            Unit="barg",            DataSetIds=new List<Identity>{                 new Identity(2452112635370053)             }        },        Limit=2    });Note that the DataSetIds argument of the Filter accepts a list of objects of the type CogniteSdk.Identity. The Identity class is used to set either Id or ExternalId. In the example above I used the Id to reference the desired DataSetId.

24
Dilini Fernando
Seasoned Practitioner
9 months ago
S
Committed
samirhajCommitted
asked in How-To Guides

detNet RestAPI Cognite SDK example [Community Contributed]

Could we use the Cognite C# Sdk as a common Rest Api Sdk in general to have RestApi client (similar to what RestSharp does for exampl) or is the Cognite Sdk is designed to access and retreive data from Cognite/CDF?The Cognite .NET SDK was built to add convenience to developers while creating solutions that need to call endpoints exposed by the CDF API.What is the difference between Cognite C# Sdk and Cognite Python Sdk? Does both have more or less functionality built-in? When using the Python sdk, once the token is authenticated to access CDF, the message is you are connecting to “Cognite-Postman” but there is not such message in the C# sdk. Is this because with Python you could use Postman for data analytics? Moreover most of the data processing that Postman does also exists in CDF like aggregation, averaging, etc. Are there specific functionalities that Postman does have and CDF doesn’t? The Cognite .NET SDK is a Community SDK (you can check the other available SDKs here), and as suc

24
Dilini Fernando
Seasoned Practitioner
9 months ago
S
Committed
samirhajCommitted
asked in How-To Guides

detNet RestAPI Cognite SDK example [Community Contributed]

The aggregates are calculated on top of the original data that was ingested from the source system to Cognite Data Fusion (CDF). As described on the documentation, CDF doesn't return aggregates or interpolations for time ranges that have no data points.For example, if the original data contains data points with a 1s granularity (every second a new data point is being measured and stored), there should be no issues to request aggregates with 10s, 1m, 10m, 1h granularities. However if the original data contains data points with a 1m granularity, asking for a granularity of 10s would still yield a new value only on a 1m granularity.If you need to get a value for each 10s, you will need to upsample your data manually by implementing some sort of interpolation.Aggregates are convenient calculations that help to deal with high amounts of data, but if you want you can also take a look at the original data that was ingested into CDF by passing null to both Aggregates and Granularity. Here’s a

24
Dilini Fernando
Seasoned Practitioner
9 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings