How-To Guides

Discover expert-crafted How-To Guides or contribute your expertise sharing your own

  • 41 Topics
  • 116 Replies

41 Topics

A
Seasoned Practitioner
Ankit KumarSenior Cloud Architect
posted in How-To Guides

How to build a connector for CDF in Power Apps [Cognite Official]

You can load data from Cognite Data Fusion (CDF) into Microsoft Power Apps, where you can build custom apps for your business needs.This article explains how to create and connect a custom Power Apps connector with the Cognite API and build a canvas app.Before you start​To perform the steps below, you need to be an administrator of Azure AD. Make sure you have registered the Cognite API and the CDF portal application in Azure AD and set up Azure AD and CDF groups to control access to CDF data.You'll also need to register a custom web app in Azure AD to give Power Apps access to the CDF data. Make sure to register the redirect URI, https://global.consent.azure-apim.net/redirect under Authentication > Add a platform > Web.Step 1: Create and authenticate a custom connector​ Sign in to Microsoft Power Apps with your Microsoft account. Start from Dataverse > Custom Connectors. Select New custom connector > Create from blank. On the General information tab: Host: Enter http

13
A
Seasoned Practitioner
13 days ago
M
Seasoned Practitioner
Mithila Jayalath ExternalSeasoned Practitioner
posted in How-To Guides

How to run multiple Instances of the Cognite DB Extractor on the same VM [Cognite Official]

1.0 Introduction 2.0 Steps 2.1 Create a System DSN Entry for the MS SQL Server Database Used as the Source for the Extractors. 2.2 Installing the First Instance of the Windows Service 2.3 Installing the Second Instance of the Windows Service. 2.4 Start the Services 1.0 IntroductionThe purpose of this document is to show how  to install multiple instances of the Cognite DB Extractor as a Windows service, where each instance is using a different config.yml file. 2.0 Steps2.1 Create a System DSN Entry for the MS SQL Server Database Used as the Source for the Extractors. Run the “ODBC Data Sources (32-bit)” program as an administrator.   Click on the “System DSN” tab and click on the “Add” button to create a new SQL Server DSN.   Use the wizard to name and configure the new System DSN entry.   When finished with the configuration, click on the “Test Data Source” button to confirm the entry is working; the “Test Results” popup window shown to the right should open to confirm the test was

1
M
Seasoned Practitioner
2 months ago
Frank Danielsen
Practitioner
Frank DanielsenPractitioner
published in How-To Guides

How to use extraction pipeline configuration to configure Cognite functions [Cognite Official]

IntroductionCo-author: @Jan Inge Bergseth Cognite Functions provide a run-time environment for hosting and running Python code, similar to Azure Functions, Google Cloud Functions, or Amazon Lambda. One of the benefits of utilizing the built in Functions capability of CDF is that it is tightly coupled with CDF and gives you, as a developer, an implicit Cognite client allowing you to seamlessly interact with your CDF data and data pipeline. CDF Extraction Pipelines allow you to monitor the data flow and gain visibility into the run history and receive notifications from data integration events that need to be paid attention to, all of which is an important part of the resulting data quality. CDF Extraction Pipelines offer a great feature for storing extractor configuration settings, allowing you to remotely configure the extractor. This can be extremely helpful as it avoids having to locally manage extractor configuration. This article will explore the combination of Extraction Pipelines

2814
Andre Alves
Seasoned
2 months ago
J
Practitioner
Johan Løwig Rambech DahlPractitioner
posted in How-To Guides

How to run and test Cognite DB-extractor v3

Purpose:Test v3 of the db-extractor as easily as possible with as little local set-up as needed. Outline:This example will use the Docker version of the db-extractor, and also use Docker to make a temporary PostgreSQL database. If you use another version of the extractor (like a Windows service) and/or want to use a database already installed on your local computer, it should still be fairly simple to use this example and make your own changes where applicable. Pre-requisites:Docker (can be avoided if you run a non-containerized extractor and have access to a database) A CDF-project with IdP-authentication Tenant Client Secret Capabilities connected to the client used in the extractor: Raw: List, Read, Write Extraction Pipelines Runs: Write One Extraction pipeline  Steps:Create a containerized PostgreSQL database Open a new terminal Get docker image for the postgreSQL database (only needed once). Run: docker pull postgres Create and run Postgres Container (only neede

0
J
Practitioner
2 months ago
S
Active
sudheerchonateActive
posted in How-To Guides

How to show edge source properties stored in separate type using GraphQL query [Community Contributed]

In the below GraphQL schema actors [Person] edge properties (role, compensation) are captured in ActedIn type. How to get edge properties of the Actor stored in separate type using GraphQL query. type Movie {title: Stringactors: [Person]@relation(edgeSource: "ActedIn"type: { space: "imdb", externalId: "HAS_ACTOR" }direction: OUTWARDS)director: Person}type Person {name: String!}type ActedIn @edge {role: Stringcompensation: Floatmetadata: JSONObjectlines: [String]}

0
S
Active
2 months ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to use the MQTT hosted extractor [Cognite Official]

MQTT is one of the standards for IoT communication in the industry. It is a lightweight, publish-subscribe, machine to machine network protocol for message queue/message queuing service. As it is widely used, it made sense to have a dedicated extractor for that protocol. At Cognite, we have a hosted extractor for MQTT, that facilitates the extraction of data from a broker. “Hosted” means it runs on our infrastructure and you don’t have to deploy anything on yours to extract data: you just have to configure it. In this article, we will see how to configure and use it. An example scenario involving MQTT could be sensors installed on industrial machines  publishing timeseries data like temperature, pression etc., to a broker. Data would be published to different topics. When extracting data from the broker, with the Cognite extractor, we can have different behaviors based on the topic. Here, we will publish data to four different topics, on an MQTT broker. Those topics are named as follow

0
Pierre Pernot
Practitioner
3 months ago
kelvin
Seasoned Practitioner
kelvinSeasoned Practitioner
posted in How-To Guides

How to rapidly build data and workflow apps with Cognite Data Fusion and Retool [Cognite Official]

The steps in this post assume that you have access to an instance of Retool and Cognite Data Fusion. It also assumes basic knowledge of Cognite resource types like Assets and Events.A thing that customers of Cognite have realized is that it’s incredibly hard to predict today, which business processes and use cases you need to solve in the future, or even tomorrow (literally). Thus, one of the philosophies behind Cognite Data Fusion, is to enable a solid and flexible data foundation that makes building solutions and processes for new problems faster, and more scalable.One way of speeding up the building of new apps and workflows, is to make data readily available in open, well defined and performant APIs. This lets builders use a wide range of tools, and select the best tool for the task at hand.Here we’ll show how to build a simple mobile-friendly application with Retool and Cognite Data Fusion, to manually log production downtime events, and attach them to an Asset in Cognite Data Fus

2
kelvin
Seasoned Practitioner
3 months ago
S
Practitioner
Shashan UdawattePractitioner
published in How-To Guides

How to query Aggregate using the filter on related types [Cognite Official]

In Fusion Data Modeling(FDM) when you try to use aggregate queries currently only supports filtering on a View's own properties. Currently, users cannot filter on related types in aggregate queries. See the below example: In Fusion Data Modeling(FDM) when you try to use aggregate queries currently only supports filtering on a View's own properties. Currently, users cannot filter on related types in aggregate queries. See the below example: If you refer to the below images you can see that the Actor is related to the Movie. You can see that the relation data is being populated in the Data Management as well.However, when you try to filter out any of the related types in aggregate queries you will get an error message that the Field is not defined. The reason is that the aggregate endpoint is serviced by a different backend(elastic search) compared to querying data in FDM and this service cannot/does not normalize any relationships and only supports aggregations of data in a single cont

1432
Anita Hæhre
Seasoned Practitioner
3 months ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to package a custom extractor as a Docker image [Cognite Official]

Docker is one of the most common ways to deploy an extractor because of its ease of use. Our off-the-shelf extractors have a Docker image you can download and run. But when it comes to custom extractors, you have to build the Docker image yourself (if you intend to run it with Docker, of course). In this article, we’ll see how to do that. We’ll use the example of the REST extractor, from a previous article (https://hub.cognite.com/developer-and-user-community-134/rest-extractor-1232). It is a custom extractor, made with our extractor-utils package, hence it does not have a pre-built Docker image.The content of this article will be specific to the REST extractors made with the extractor-utils package, but is still applicable to other kinds of extractors. You simply have to make sure to adapt environment variables, arguments etc. to what your extractor needs. First, make sure that you have Docker installed, up and running. Open the python project with the extractor in your code editor. T

3
P
Committed
5 months ago
Everton Colling
Practitioner
Everton CollingPractitioner
posted in How-To Guides

Streaming data from Cognite Data Fusion to Power BI [Cognite Official]

IntroductionIn this tutorial, we'll explore one method to stream data from Cognite Data Fusion (CDF) to Power BI. While Cognite's Power BI connector serves its purpose, it has limitations, especially in terms of live data streaming and handling complex queries​​. Here I am presenting an alternative approach using "push" or "streaming" datasets in Power BI, bypassing the limitations of Cognite's Power BI connector by utilizing a scheduled job to fetch data from the CDF REST API and push it to the Power BI REST API.1) Understanding the Limitations of Import mode ConnectorsCognite's Power BI connector it's based on OData (Open Data Protocol) and uses Power BI's Import mode. Import mode is the most common mode used to develop semantic models. This mode delivers fast performance to end users thanks to in-memory querying. It also offers design flexibility to modelers, and support for specific Power BI service features (Q&A, Quick Insights, etc.). Because of these strengths, it's the defa

0
Everton Colling
Practitioner
5 months ago
M
Seasoned Practitioner
Mithila Jayalath ExternalSeasoned Practitioner
posted in How-To Guides

How to Migrate Projects from an Azure tenant to a new Azure tenant [Cognite Official]

Requirements Access to your CDF project. Access to your applications and services. A new Azure Active Directory tenant.  Follow the steps carefully to plan and prepare for a successful migration:Step 1: Collect project data Step 2: Configure new IdP identity and access management Step 2.1: Groups and users Step 2.2: Register applications and service accounts Step 2.3: Register the new IdP for the existing domain. Step 3: Access Management Step 4: Enable new OIDC config for the project Step 5: Update CDF transformations, Functions, Extractors, Grafana, Custom Apps, etc. Step 1: Collect project dataCollect available data for all the items that you need to migrate: Applications: Make a list of Cognite applications (i.e.: Remote, Infield, …) used by your CDF project. Make a list of third-party applications (i.e.: Grafana, Power BI, Azure Data Factory, …) used by your CDF project. Services: Make a list of scheduled functions. Make a list of the extractors in use.

3
Anita Hæhre
Seasoned Practitioner
6 months ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to run the OPC UA extractor with Docker [Cognite Official]

OPC UA is an open source data standard for communication. It is one of the most used in the industry. OPC UA can for example be used to send data from a machine and its sensors to a computer (or to a cloud service such as Cognite Data Fusion).In this article, we’ll see how to extract data from an OPC UA data source to Cognite Data Fusion (CDF). For this purpose, we have an off-the-shelf extractor that we simply need to configure and deploy. This extractor is maintained by Cognite. You can find it in CDF under Integrate > Extract data (where you can also find our other extractors: DB, PI etc., you can also create custom extractors).  One of the most common and easiest ways to deploy such an extractor is via a Docker image. The fact that it is platform agnostic makes it pretty easy to deploy: you just need docker installed to make it run. Also, everything you need is packaged in the image, you do not need to install or build anything. The Docker images are available on Docker Hub (htt

4
Andre Alves
Seasoned
6 months ago
Vishnuvallabha Iyengar
Committed
Vishnuvallabha IyengarCommitted
asked in How-To Guides

How to connect a private Cognite project to Python DSK [Community Contributed]

I am new to learning how to utilize Cognite Data Fusion. I apologize if this question has already been answered; if it has, please inform me!Currently, I plan to use Cognite to establish a connection with a digital twin and utilize the Python SDK to import and extract data using custom-built models. However, I have a question regarding connecting the SDK to a private project. In the case of Open Industrial Data, we were provided with a Tenant ID, Client ID, and Base_URL.Hence, I would like to inquire whether there are any specific steps or requirements to connect to a private access project using the Python SDK. I would greatly appreciate any assistance you can provide. Thank you!

3
Dilini Fernando
Seasoned Practitioner
9 months ago
Y
Committed
Yanzhu YuElkem
asked in How-To Guides

Coloring last reading time in Grafana [Community Contributed]

Hi! I want to display the latest time when new data come in to a time-series and color it based on the following rule:-Latest value within 48 hours, shown latest time in green-Latest value more than 48 hours, shown latest time in red(With that rule, the numbers should be in green, green and red in my picture) I struggle to find the correct thresholds to make this 48 hours rule work. It seems to be related to unix time. Have thought about calculate the time difference between current time and latest updated time, but didn’t make it work in time series custom query. Any suggestion would be appreciated. :)

3
Dilini Fernando
Seasoned Practitioner
9 months ago
Olav Alstad
Active
Olav AlstadActive
asked in How-To Guides

Timestamp for first datapoint in timeseries? [Community Contributed]

I have many thousand timeseries where I need to find the date of the first datapoints in each series. For each timeseries, I have no idea if the first datapoint is from this year, or from 20 years ago. Fetching data for several decades is not effective.  Any ideas how to get the first datapoint? Getting the last datapoint would also be great.  Thanks!

7
Dilini Fernando
Seasoned Practitioner
9 months ago
crgomez13
Committed
crgomez13Accenture
asked in How-To Guides

Cognite Charts: IF/ELSE Functions for Calculations [Community Contributed]

On Behalf of Charts super users at Clear Lake:I would like the ability to incorporate “If/Else” statements as a function in Charts similar to what is capable in Excel. There are some calculations that rely on certain conditional parameters to be met in order to define which mathematical approach must be taken.

6
Dilini Fernando
Seasoned Practitioner
9 months ago
V
Committed
Viswanadha Sai Akhil PujyamCommitted
asked in How-To Guides

Uploading P&ID diagrams to CDF Through Python SDK [Community Contributed]

hello Community,can anyone help me understand how to upload P&ID diagram files to CDF, Using Python SDK.

4
Dilini Fernando
Seasoned Practitioner
9 months ago
E
Practitioner
eashwar11
asked in How-To Guides

SharePoint Online - File content extraction [Cognite Official]

hello Community,Can someone please share the approach (along with code) for extracting files from sharepoint online (xls , with multiple worksheets) and extract the content and load them as RAW tables in CDF.  Is there a direct feature available in Sharepoint file extractor that does this job? Should we use SDK to extend the file extraction and then read the content and insert into tables?.  If we have a large set of files and each containing multiple sheets, it can be hard to process all of them dynamically.  Please advise.

2
Dilini Fernando
Seasoned Practitioner
9 months ago
S
Committed
samirhajCommitted
asked in How-To Guides

detNet RestAPI Cognite SDK example [Community Contributed]

Hello,I am quite new on usinf RestAPI. My company has the license to use CDF. I also inderstood that Cognite has developed dotNet RestApi SDK.I searched a bit but I could not find an example of how we can use this SDK for data retrieval especially for time-series data. How could I get a snippet of code showing an example how to use this?Thanks. 

24
Dilini Fernando
Seasoned Practitioner
9 months ago
A
Active
ArnfinnActive
asked in How-To Guides

How do you delete a (data model) space? [Community Contributed]

I created a test data model in a test space. I finished testing and deleted the data model, but can not find a way to delete the space (which no longer contains any data models)?

3
Dilini Fernando
Seasoned Practitioner
9 months ago
V
Committed
Viswanadha Sai Akhil PujyamCommitted
asked in How-To Guides

.csv file Ingestion into CDF using Python SDK [Community Contributed]

hello community, can anyone tell me is there any way to upload .csv files to CDF using Python SDK.

4
Dilini Fernando
Seasoned Practitioner
9 months ago
K
Seasoned
Kristian NymoenSeasoned
asked in How-To Guides

Filtering assets by geolocation using pythonSDK [Community Contributedl]

How can I filter assets by their geolocation using the python SDK?from cognite.client.data_classes import GeoLocationFilter, GeometryFiltermygeofilter = GeoLocationFilter( relation="within", shape=GeometryFilter( type="Polygon", coordinates=[[[7, 61], [13, 61], [10, 55], [7, 61]]] ))client.assets.list(geo_location=mygeofilter)This approach works for files, but for assets I get:CogniteAPIError: Unexpected value 'within' | code: 400 | X-Request-ID: fd1c5a73-a3d3-9ec1-bd97-039b4b1e0051

2
Dilini Fernando
Seasoned Practitioner
9 months ago
P
Committed
Paramale AratiCommitted
asked in How-To Guides

How to build Asset hierarchy in Power BI [Community Contributed]

Hi Team,I am facing one challenge in creating Asset hierarchy in Power BI. Format of data is not as per the required format for hierarchy creation i.e. i do not get parent-child relation in different columns. However i see all nodes appear in single column. I do have ParentID but unaware how this will help me to form hierarchy. Any inputs will be appreciated. Thanks!Regards,Arati

6
Dilini Fernando
Seasoned Practitioner
9 months ago
S
Seasoned
Sonali Vishal PatilSeasoned
asked in How-To Guides

How to get list of the columns from raw table which is in number & text format? [Community Contributed]

(RAW Table : As above which has total 179 columns #41 contains numeric data 138 contains text data)I have ingested data in raw table….now I want to get only data of  41 columns which has number data type ?how this can be done in the transformation?similar query I want to run for remained 138 columns which has “Text” data type?

4
Dilini Fernando
Seasoned Practitioner
9 months ago
S
Seasoned
Sonali Vishal PatilSeasoned
asked in How-To Guides

How to unpivot any existing raw table? [Community Contributed]

How to unpivot any existing raw table?I want to transpose columns to rowshere I am aware about stack can be used, but i dont want to give hard coded columns names... how this can be done run time (Without passing any hardcoded value of column name in stack command?)How to get run time columns count & how this columns can be passed in stack? Table1:id p1 p2 p3 1 2.0 3.0 4.0 2 10.0 15.0 20.0 select id,   stack(3, 'p1', p1, 'p2', p2, 'p3', p3) as (property,value)from `db1`.`Table1`

4
Dilini Fernando
Seasoned Practitioner
9 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings