Groups
- - Knowledge Base
Product Updates
Product Ideas
Docs
Events
Academy
Partners

Home
Community overview
How-To's

How-To's

Explore user resources including how-to guides and video tutorials

47 Topics

Recently active

Most replies Most views Newest first

Neerajkumar BhatewaraSchlumberger

asked in How to Contribute

How to contribute code to cognite open source repository - cdf-sdk-java

Hello,I want to make code changes to open source cognite repository - https://github.com/cognitedata/cdf-sdk-java.Can you guide me on the process to do the same ? How I can open the pull request on this repo ?Regards,Neeraj B

295

13 days ago

Andreea OpreaSeasoned Practitioner

posted in How-To Guides

How to create faster and more efficient RCAs with Industrial Canvas

Hi everyone,I would like to share with you how to build faster your RCAs with Industrial Canvas. We have seen many of our users spend more time dealing with tools and data instead of focusing on the cause and analyzing the issues. We can facilitate this process and speed up your time to value. Now, in Industrial Canvas, you can easily create cause maps in the same workspace where you have all the data you need as evidence for approving/rejecting a hypothesis. The built-in cause map comes with a few essential features: Easily create a tree with cards. Option to add more connected cards by clicking on the “+” left, right, above, and below. The boxes are automatically connected. Arrow pointing right to left. Auto alignment: The user does not need to take actions for alignment. If you move the root card, the entire tree also moves. Boxes can be connected with other data in the canvas by using connection lines. Multiple trees can be added to the same canvas and connected. Apply statuses o

490

1 month ago

Pierre PernotSeasoned Practitioner

posted in How-To Guides

How to make API calls with the Python SDK [Cognite Official]

Hello,I’m Pierre, data engineer at Cognite. I’ll be posting, on a regular basis, some useful tips about Cognite Data Fusion and its use. If you have any questions or remarks (technical or not), don’t hesitate to reach out to me or to create a new topic in Cognite Hub. Some of the features of the Cognite API are not available in the Python SDK. You can still use those features by making a call to the desired endpoint. The SDK facilitates this process thanks to the get, post, put and delete methods of the Cognite client (see: https://cognite-docs.readthedocs-hosted.com/projects/cognite-sdk-python/en/latest/cognite.html#cogniteclient). Once your client is set up, all the auth process is taken care of by itself, it is very straight forward. Example:payload = {...}endpoint_url = “/api/v1/projects/my-project/context/diagram/detect/”response = client.post(endpoint_url, payload) When a payload is needed, you can refer to the documentation to get insights on its structure. (Cognite API documen

1692

2 months ago

Mithila JayalathSeasoned Practitioner

posted in How-To Guides

How to Extract All Data Available from SAP System Using OData API Extraction Process

OverviewWhen extracting data from an SAP system using the OData API extraction process, you may encounter situations where not all available data is being retrieved. This issue can be resolved by utilizing server-side or client-side pagination, depending on the capabilities of your SAP endpoint.Solution1. Server-Side PaginationFrom SAP Extractor version 2.2.0 onward, server-side pagination is supported for full data loads using the OData $skiptoken parameter. This approach allows the extraction process to handle large data sets more efficiently by managing pagination on the server side.When to Use:Your SAP OData endpoint supports the $skiptoken parameter.Sample Configuration:# full load - server pagination - source_name: s4h_dev name: equipment destination: type: raw database: test_sap_extractor table: equipment_full_server sap_service: API_EQUIPMENT sap_entity: Equipment sap_key: [Equipment] pagination_type: server schedule: type: interval expression:

150

2 months ago

Madina KholmatovaCognite Academy Instructor

posted in How to Contribute

How to contribute?

Your expertise can make a difference! By sharing your best practices and insights on Cognite Data Fusion, you position yourself as a leader in the community. Your contributions bring new perspectives, address gaps with unique use cases, and collectively grow our knowledge center. To share your best practices, simply click the + Create Topic button. Provide a brief description of your topic and its value, and include specific insights or tips—whether through a video, article, infographic, or interactive content. Keep it concise and focused on a single topic. When sharing, remember to select the How-to Guides category and add relevant tags for better categorization. All approved contributions will be marked as "Community Contributed" to acknowledge your valuable input. If your content aligns well with our educational goals and quality standards, we'll add and publish it in our Quick Videos Library on Cognite Academy. Contributors of published modules will be awarded points on the Cogni

150

3 months ago

Ankit KumarSenior Cloud Architect

posted in How-To Guides

How to build a connector for CDF in Power Apps [Cognite Official]

You can load data from Cognite Data Fusion (CDF) into Microsoft Power Apps, where you can build custom apps for your business needs.This article explains how to create and connect a custom Power Apps connector with the Cognite API and build a canvas app.Before you startTo perform the steps below, you need to be an administrator of Azure AD. Make sure you have registered the Cognite API and the CDF portal application in Azure AD and set up Azure AD and CDF groups to control access to CDF data.You'll also need to register a custom web app in Azure AD to give Power Apps access to the CDF data. Make sure to register the redirect URI, https://global.consent.azure-apim.net/redirect under Authentication > Add a platform > Web.Step 1: Create and authenticate a custom connector Sign in to Microsoft Power Apps with your Microsoft account. Start from Dataverse > Custom Connectors. Select New custom connector > Create from blank. On the General information tab: Host: Enter http

62213

6 months ago

Mithila JayalathSeasoned Practitioner

posted in How-To Guides

How to run multiple Instances of the Cognite DB Extractor on the same VM [Cognite Official]

1.0 Introduction 2.0 Steps 2.1 Create a System DSN Entry for the MS SQL Server Database Used as the Source for the Extractors. 2.2 Installing the First Instance of the Windows Service 2.3 Installing the Second Instance of the Windows Service. 2.4 Start the Services 1.0 IntroductionThe purpose of this document is to show how to install multiple instances of the Cognite DB Extractor as a Windows service, where each instance is using a different config.yml file. 2.0 Steps2.1 Create a System DSN Entry for the MS SQL Server Database Used as the Source for the Extractors. Run the “ODBC Data Sources (32-bit)” program as an administrator. Click on the “System DSN” tab and click on the “Add” button to create a new SQL Server DSN. Use the wizard to name and configure the new System DSN entry. When finished with the configuration, click on the “Test Data Source” button to confirm the entry is working; the “Test Results” popup window shown to the right should open to confirm the test was

941

7 months ago

Frank DanielsenPractitioner

published in How-To Guides

How to use extraction pipeline configuration to configure Cognite functions [Cognite Official]

IntroductionCo-author: @Jan Inge Bergseth Cognite Functions provide a run-time environment for hosting and running Python code, similar to Azure Functions, Google Cloud Functions, or Amazon Lambda. One of the benefits of utilizing the built in Functions capability of CDF is that it is tightly coupled with CDF and gives you, as a developer, an implicit Cognite client allowing you to seamlessly interact with your CDF data and data pipeline. CDF Extraction Pipelines allow you to monitor the data flow and gain visibility into the run history and receive notifications from data integration events that need to be paid attention to, all of which is an important part of the resulting data quality. CDF Extraction Pipelines offer a great feature for storing extractor configuration settings, allowing you to remotely configure the extractor. This can be extremely helpful as it avoids having to locally manage extractor configuration. This article will explore the combination of Extraction Pipelines

3184

7 months ago

Johan Løwig Rambech DahlPractitioner

posted in How-To Guides

How to run and test Cognite DB-extractor v3

Purpose:Test v3 of the db-extractor as easily as possible with as little local set-up as needed. Outline:This example will use the Docker version of the db-extractor, and also use Docker to make a temporary PostgreSQL database. If you use another version of the extractor (like a Windows service) and/or want to use a database already installed on your local computer, it should still be fairly simple to use this example and make your own changes where applicable. Pre-requisites:Docker (can be avoided if you run a non-containerized extractor and have access to a database) A CDF-project with IdP-authentication Tenant Client Secret Capabilities connected to the client used in the extractor: Raw: List, Read, Write Extraction Pipelines Runs: Write One Extraction pipeline Steps:Create a containerized PostgreSQL database Open a new terminal Get docker image for the postgreSQL database (only needed once). Run: docker pull postgres Create and run Postgres Container (only neede

240

7 months ago

sudheerchonateActive

posted in How-To Guides

How to show edge source properties stored in separate type using GraphQL query [Community Contributed]

In the below GraphQL schema actors [Person] edge properties (role, compensation) are captured in ActedIn type. How to get edge properties of the Actor stored in separate type using GraphQL query. type Movie {title: Stringactors: [Person]@relation(edgeSource: "ActedIn"type: { space: "imdb", externalId: "HAS_ACTOR" }direction: OUTWARDS)director: Person}type Person {name: String!}type ActedIn @edge {role: Stringcompensation: Floatmetadata: JSONObjectlines: [String]}

210

8 months ago

Pierre PernotSeasoned Practitioner

posted in How-To Guides

How to use the MQTT hosted extractor [Cognite Official]

MQTT is one of the standards for IoT communication in the industry. It is a lightweight, publish-subscribe, machine to machine network protocol for message queue/message queuing service. As it is widely used, it made sense to have a dedicated extractor for that protocol. At Cognite, we have a hosted extractor for MQTT, that facilitates the extraction of data from a broker. “Hosted” means it runs on our infrastructure and you don’t have to deploy anything on yours to extract data: you just have to configure it. In this article, we will see how to configure and use it. An example scenario involving MQTT could be sensors installed on industrial machines publishing timeseries data like temperature, pression etc., to a broker. Data would be published to different topics. When extracting data from the broker, with the Cognite extractor, we can have different behaviors based on the topic. Here, we will publish data to four different topics, on an MQTT broker. Those topics are named as follow

1870

8 months ago

kelvinSeasoned Practitioner

posted in How-To Guides

How to rapidly build data and workflow apps with Cognite Data Fusion and Retool [Cognite Official]

The steps in this post assume that you have access to an instance of Retool and Cognite Data Fusion. It also assumes basic knowledge of Cognite resource types like Assets and Events.A thing that customers of Cognite have realized is that it’s incredibly hard to predict today, which business processes and use cases you need to solve in the future, or even tomorrow (literally). Thus, one of the philosophies behind Cognite Data Fusion, is to enable a solid and flexible data foundation that makes building solutions and processes for new problems faster, and more scalable.One way of speeding up the building of new apps and workflows, is to make data readily available in open, well defined and performant APIs. This lets builders use a wide range of tools, and select the best tool for the task at hand.Here we’ll show how to build a simple mobile-friendly application with Retool and Cognite Data Fusion, to manually log production downtime events, and attach them to an Asset in Cognite Data Fus

3632

8 months ago

gregerSeasoned Practitioner

posted in How-To Guides

Technical deep dive of the Cognite Data Fusion Toolkit beta

530

9 months ago

gregerSeasoned Practitioner

posted in How-To Guides

Quick demo of Cognite Data Fusion Toolkit beta for project administration

450

9 months ago

Shashan UdawattePractitioner

published in How-To Guides

How to query Aggregate using the filter on related types [Cognite Official]

In Fusion Data Modeling(FDM) when you try to use aggregate queries currently only supports filtering on a View's own properties. Currently, users cannot filter on related types in aggregate queries. See the below example: In Fusion Data Modeling(FDM) when you try to use aggregate queries currently only supports filtering on a View's own properties. Currently, users cannot filter on related types in aggregate queries. See the below example: If you refer to the below images you can see that the Actor is related to the Movie. You can see that the relation data is being populated in the Data Management as well.However, when you try to filter out any of the related types in aggregate queries you will get an error message that the Field is not defined. The reason is that the aggregate endpoint is serviced by a different backend(elastic search) compared to querying data in FDM and this service cannot/does not normalize any relationships and only supports aggregations of data in a single cont

1532

9 months ago

Pierre PernotSeasoned Practitioner

posted in How-To Guides

How to package a custom extractor as a Docker image [Cognite Official]

Docker is one of the most common ways to deploy an extractor because of its ease of use. Our off-the-shelf extractors have a Docker image you can download and run. But when it comes to custom extractors, you have to build the Docker image yourself (if you intend to run it with Docker, of course). In this article, we’ll see how to do that. We’ll use the example of the REST extractor, from a previous article (https://hub.cognite.com/developer-and-user-community-134/rest-extractor-1232). It is a custom extractor, made with our extractor-utils package, hence it does not have a pre-built Docker image.The content of this article will be specific to the REST extractors made with the extractor-utils package, but is still applicable to other kinds of extractors. You simply have to make sure to adapt environment variables, arguments etc. to what your extractor needs. First, make sure that you have Docker installed, up and running. Open the python project with the extractor in your code editor. T

2443

11 months ago

Everton CollingPractitioner

posted in How-To Guides

Streaming data from Cognite Data Fusion to Power BI [Cognite Official]

IntroductionIn this tutorial, we'll explore one method to stream data from Cognite Data Fusion (CDF) to Power BI. While Cognite's Power BI connector serves its purpose, it has limitations, especially in terms of live data streaming and handling complex queries. Here I am presenting an alternative approach using "push" or "streaming" datasets in Power BI, bypassing the limitations of Cognite's Power BI connector by utilizing a scheduled job to fetch data from the CDF REST API and push it to the Power BI REST API.1) Understanding the Limitations of Import mode ConnectorsCognite's Power BI connector it's based on OData (Open Data Protocol) and uses Power BI's Import mode. Import mode is the most common mode used to develop semantic models. This mode delivers fast performance to end users thanks to in-memory querying. It also offers design flexibility to modelers, and support for specific Power BI service features (Q&A, Quick Insights, etc.). Because of these strengths, it's the defa

4750

11 months ago

Mithila JayalathSeasoned Practitioner

posted in How-To Guides

How to Migrate Projects from an Azure tenant to a new Azure tenant [Cognite Official]

Requirements Access to your CDF project. Access to your applications and services. A new Azure Active Directory tenant. Follow the steps carefully to plan and prepare for a successful migration:Step 1: Collect project data Step 2: Configure new IdP identity and access management Step 2.1: Groups and users Step 2.2: Register applications and service accounts Step 2.3: Register the new IdP for the existing domain. Step 3: Access Management Step 4: Enable new OIDC config for the project Step 5: Update CDF transformations, Functions, Extractors, Grafana, Custom Apps, etc. Step 1: Collect project dataCollect available data for all the items that you need to migrate: Applications: Make a list of Cognite applications (i.e.: Remote, Infield, …) used by your CDF project. Make a list of third-party applications (i.e.: Grafana, Power BI, Azure Data Factory, …) used by your CDF project. Services: Make a list of scheduled functions. Make a list of the extractors in use.

1623

11 months ago

Pierre PernotSeasoned Practitioner

posted in How-To Guides

How to run the OPC UA extractor with Docker [Cognite Official]

OPC UA is an open source data standard for communication. It is one of the most used in the industry. OPC UA can for example be used to send data from a machine and its sensors to a computer (or to a cloud service such as Cognite Data Fusion).In this article, we’ll see how to extract data from an OPC UA data source to Cognite Data Fusion (CDF). For this purpose, we have an off-the-shelf extractor that we simply need to configure and deploy. This extractor is maintained by Cognite. You can find it in CDF under Integrate > Extract data (where you can also find our other extractors: DB, PI etc., you can also create custom extractors). One of the most common and easiest ways to deploy such an extractor is via a Docker image. The fact that it is platform agnostic makes it pretty easy to deploy: you just need docker installed to make it run. Also, everything you need is packaged in the image, you do not need to install or build anything. The Docker images are available on Docker Hub (htt

4564

1 year ago

Vishnuvallabha IyengarCommitted

asked in How-To Guides

How to connect a private Cognite project to Python DSK [Community Contributed]

I am new to learning how to utilize Cognite Data Fusion. I apologize if this question has already been answered; if it has, please inform me!Currently, I plan to use Cognite to establish a connection with a digital twin and utilize the Python SDK to import and extract data using custom-built models. However, I have a question regarding connecting the SDK to a private project. In the case of Open Industrial Data, we were provided with a Tenant ID, Client ID, and Base_URL.Hence, I would like to inquire whether there are any specific steps or requirements to connect to a private access project using the Python SDK. I would greatly appreciate any assistance you can provide. Thank you!

763

1 year ago

Yanzhu YuElkem

asked in How-To Guides

Coloring last reading time in Grafana [Community Contributed]

Hi! I want to display the latest time when new data come in to a time-series and color it based on the following rule:-Latest value within 48 hours, shown latest time in green-Latest value more than 48 hours, shown latest time in red(With that rule, the numbers should be in green, green and red in my picture) I struggle to find the correct thresholds to make this 48 hours rule work. It seems to be related to unix time. Have thought about calculate the time difference between current time and latest updated time, but didn’t make it work in time series custom query. Any suggestion would be appreciated. :)

383

1 year ago

Olav AlstadCommitted

asked in How-To Guides

Timestamp for first datapoint in timeseries? [Community Contributed]

I have many thousand timeseries where I need to find the date of the first datapoints in each series. For each timeseries, I have no idea if the first datapoint is from this year, or from 20 years ago. Fetching data for several decades is not effective. Any ideas how to get the first datapoint? Getting the last datapoint would also be great. Thanks!

1347

1 year ago

crgomez13Accenture

asked in How-To Guides

Cognite Charts: IF/ELSE Functions for Calculations [Community Contributed]

On Behalf of Charts super users at Clear Lake:I would like the ability to incorporate “If/Else” statements as a function in Charts similar to what is capable in Excel. There are some calculations that rely on certain conditional parameters to be met in order to define which mathematical approach must be taken.

826

1 year ago

Viswanadha Sai Akhil PujyamCommitted

asked in How-To Guides

Uploading P&ID diagrams to CDF Through Python SDK [Community Contributed]

hello Community,can anyone help me understand how to upload P&ID diagram files to CDF, Using Python SDK.

1224

1 year ago

eashwar11

asked in How-To Guides

SharePoint Online - File content extraction [Cognite Official]

hello Community,Can someone please share the approach (along with code) for extracting files from sharepoint online (xls , with multiple worksheets) and extract the content and load them as RAW tables in CDF. Is there a direct feature available in Sharepoint file extractor that does this job? Should we use SDK to extend the file extraction and then read the content and insert into tables?. If we have a large set of files and each containing multiple sheets, it can be hard to process all of them dynamically. Please advise.

982

1 year ago

Page 1 / 2

Generate client secret for OID

Please generate a client secret once every 180 days.

Choose the application:

Badge winners

Andre Alveshas earned the badge Cognite Maintain Practitioner
Alexandre Pinellihas earned the badge Cognite Data Fusion Fundamentals
Zacarias Santanahas earned the badge Cognite Data Engineer Basics - Transform and Contextualize
rafael.fiorihas earned the badge Cognite Data Engineer Basics - Transform and Contextualize
BASKER SUBRAMANIhas earned the badge Cognite Data Fusion Fundamentals