User Resources

Explore user resources including how-to guides and video tutorials

43 Topics

M
Seasoned Practitioner
Mithila Jayalath ExternalSeasoned Practitioner
posted in How-To Guides

How to run multiple Instances of the Cognite DB Extractor on the same VM [Cognite Official]

1.0 Introduction 2.0 Steps 2.1 Create a System DSN Entry for the MS SQL Server Database Used as the Source for the Extractors. 2.2 Installing the First Instance of the Windows Service 2.3 Installing the Second Instance of the Windows Service. 2.4 Start the Services 1.0 IntroductionThe purpose of this document is to show how  to install multiple instances of the Cognite DB Extractor as a Windows service, where each instance is using a different config.yml file. 2.0 Steps2.1 Create a System DSN Entry for the MS SQL Server Database Used as the Source for the Extractors. Run the “ODBC Data Sources (32-bit)” program as an administrator.   Click on the “System DSN” tab and click on the “Add” button to create a new SQL Server DSN.   Use the wizard to name and configure the new System DSN entry.   When finished with the configuration, click on the “Test Data Source” button to confirm the entry is working; the “Test Results” popup window shown to the right should open to confirm the test was

1
M
Seasoned Practitioner
2 months ago
Pedro Lindeza
Practitioner
Pedro LindezaPractitioner
posted in How-To Guides

How to connect Cognite Data Fusion to Postgres [Cognite Official]

This guide describes how to run the Cognite dB Extractor in a separate Docker container to fetch data from Postgres server to Cognite Data Fusion.  Prerequisites Running instance of PostgreSQL server and valid credentials to SELECT rows from the table in question Docker host that can access the PostgreSQL server  Running CDF project with an Azure AD service principal with capabilities to write either to CDF. 1 - Prepare Docker ImageCognite provides the database extractor as a Docker image published to Docker Hub, requiring just the addition of an ODBC driver. Since we are connecting to PostgreSQL we will install drivers provided by PostgreSQL for Debian 11, using this Dockerfile: Note! Go to https://hub.docker.com/r/cognite/db-extractor-base/tags to see the latest version of the Docker imageFROM cognite/db-extractor-base:2.5.0-beta5RUN apt-get install odbc-postgresqlBuild the image: docker build -t cognite-db-postgres . 2 - Setup extractor configThe extractor will read a local conf

0
Pedro Lindeza
Practitioner
1 year ago
Everton Colling
Practitioner
Everton CollingPractitioner
posted in How-To Guides

How to use R with Cognite Data Fusion [Cognite Official]

R is a programming language and environment for statistical computing and graphics. It was developed in the 1990s by statisticians Ross Ihaka and Robert Gentleman and has since become one of the most widely used programming languages for data science. R is an open-source project, which means that anyone can contribute to its development and use it for free.R is particularly popular among data scientists because of its powerful built-in tools for data manipulation, analysis, and visualization. R has a wide range of packages that can be used to perform various tasks in data science, such as data cleaning, statistical modelling, machine learning, and data visualization. R also has a highly active and supportive community, which means that there are many resources available online for learning and troubleshooting.Fetching live data from Cognite Data Fusion in R can provide data scientists and analysts with a powerful tool for analysing and visualizing industrial data. It can help them iden

0
Everton Colling
Practitioner
1 year ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to validate data with pydantic [Cognite Official]

Users are often required to fill in information: YAML, forms, that is then processed by a script. Other developers might also have to send data to an API you created. Nevertheless, there is no guarantee that the user fills in the correct type of information. You might want two things: First, make sure the data you process is of the correct type If it is not, provide the user some meaningful errors  Pydantic is a library that helps with data validation. Indeed, thanks to type hints and validators, you can enforce data types, from simple to complex, and apply validation rules to data. It uses python classes, and is quite easy to use. As an example, let’s say we have a machine. This machine has a name, a make, and an ID. We can create a specific class for it. from pydantic import BaseModel  class Machine(BaseModel):   id: int   name: str   make: str external_data = {   "id": "123",   "name": "Fantastic Machine",   "make": "Machine Manufacturer",} my_machine = Machine(**external_data) pr

0
Pierre Pernot
Practitioner
1 year ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to test Python code involving SDK client calls [Cognite Official]

Automated tests are considered a good practice in software development. They can apply to every language or use case. Here we will focus on Python and testing code that involves the Python SDK’s CogniteClient object. Below are listed some of the benefits of testing.  First, they are a good way to test your code in a fast and repeated manner. With automated tests, you avoid testing manually your code: you can write each test case programmatically and run them on request. When they fail, running them in a debugger really helps understand the code, each object’s structure, and solve the problem quite fast. Then, it helps make the code maintainable. Indeed, when the code needs to be changed, running your tests ensures that the tested code will still work as expected. When a bug is discovered, it is good to create a test case for that bug to avoid having it again when updating the code. It also helps collaborate on the same repository. When several people work on the same code, it might hap

0
Pierre Pernot
Practitioner
1 year ago
ishita1983
Practitioner
ishita1983Practitioner
posted in How-To Guides

How to assess rotating equipment health from sensor data [Cognite Official]

IN SHORT This article aims to showcase how you can get value from your sensor data by continuously monitoring the health condition of a rotating equipment. We will present details of a health monitoring method that is being used by multiple operators across the globe to monitor their high value equipment. Failure of a rotating equipment, such as a turbine, pump or a compressor can cost an operator millions of dollars in revenue and repair. These assets generally contain a large number of sensors that capture the vibration, pressure, temperature and other useful metrics in a live data stream. When the data coming from the sensors go through appropriate analytics, it can correctly categorize the “health risk level” of an equipment. Operators can monitor a live dashboard that continuously updates the asset risks or subscribe to alerts that send a notification whenever there is concern in the operations.  METHODThe asset health risk is based on the percentage of anomalies in the incoming d

0
ishita1983
Practitioner
1 year ago
greger
Seasoned Practitioner
gregerSeasoned Practitioner
posted in How-To Video Guides

Quick demo of Cognite Data Fusion Toolkit beta for project administration

 

0
greger
Seasoned Practitioner
4 months ago
greger
Seasoned Practitioner
gregerSeasoned Practitioner
posted in How-To Video Guides

Technical deep dive of the Cognite Data Fusion Toolkit beta

 

0
greger
Seasoned Practitioner
4 months ago
Everton Colling
Practitioner
Everton CollingPractitioner
posted in How-To Guides

Streaming data from Cognite Data Fusion to Power BI [Cognite Official]

IntroductionIn this tutorial, we'll explore one method to stream data from Cognite Data Fusion (CDF) to Power BI. While Cognite's Power BI connector serves its purpose, it has limitations, especially in terms of live data streaming and handling complex queries​​. Here I am presenting an alternative approach using "push" or "streaming" datasets in Power BI, bypassing the limitations of Cognite's Power BI connector by utilizing a scheduled job to fetch data from the CDF REST API and push it to the Power BI REST API.1) Understanding the Limitations of Import mode ConnectorsCognite's Power BI connector it's based on OData (Open Data Protocol) and uses Power BI's Import mode. Import mode is the most common mode used to develop semantic models. This mode delivers fast performance to end users thanks to in-memory querying. It also offers design flexibility to modelers, and support for specific Power BI service features (Q&A, Quick Insights, etc.). Because of these strengths, it's the defa

0
Everton Colling
Practitioner
6 months ago
A
Practitioner
Arild EidePractitioner
posted in How-To Guides

How to connect Cognite Data Fusion to Microsoft SQL [Cognite Official]

This guide describes how to run the Cognite dB Extractor in a separate Docker container to fetch data from Microsoft SQL server to Cognite Data Fusion. Prerequisites Running instance of MS SQL server and valid credentials to SELECT rows from the table in question Docker host that can access the MS SQL server  Running CDF project with an Azure AD service principal with capabilities to write either to CDF.  1 - Prepare Docker ImageCognite provides the database extractor as a Docker image published to Docker Hub, requiring just the addition of an ODBC driver. Since we are connecting to MS SQL we will install drivers provided by Microsoft for Debian 11, using this Dockerfile: Note! Go to https://hub.docker.com/r/cognite/db-extractor-base/tags to see the latest version of the Docker imageFROM cognite/db-extractor-base:2.5.0-beta5RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \ curl https://packages.microsoft.com/config/debian/11/prod.list > /

0
A
Practitioner
1 year ago
S
Active
sudheerchonateActive
posted in How-To Guides

How to show edge source properties stored in separate type using GraphQL query [Community Contributed]

In the below GraphQL schema actors [Person] edge properties (role, compensation) are captured in ActedIn type. How to get edge properties of the Actor stored in separate type using GraphQL query. type Movie {title: Stringactors: [Person]@relation(edgeSource: "ActedIn"type: { space: "imdb", externalId: "HAS_ACTOR" }direction: OUTWARDS)director: Person}type Person {name: String!}type ActedIn @edge {role: Stringcompensation: Floatmetadata: JSONObjectlines: [String]}

0
S
Active
3 months ago
S
Practitioner
Shashan UdawattePractitioner
posted in How-To Guides

How to run Documentum Extractor with dfc.jar. [Cognite Official]

With the introduction of the Documentum Extractor version 2.0.0 onwards, it’s required to add the dfc.jar file into the lib folder. So to be able to run as a jar file user must add the dfc.jar file into the lib folder. The newly introduced Documentum Extractor is built against version 7.2.0180.0170 of DFC.Issue users might face when running the jar file with the below commandjava -jar path/to/documentum-extractor-<version>.jar path/to/config.yamlIf a user tried to run the Documentum Extractor with the dfc.jar file with the above command as mentioned in this guide, the user will receive an error message similar to the:Main  - Documentum Extractor failed:java.lang.NoClassDefFoundError: com/documentum/fc/common/DfExceptionThis is due to the fact that the Documentum Extractor jar is not identifying the main classpath.Workaround to run the Documentum Extractor with the dfc.jar file.As a workaround users could specify the classpath to include the extractor and run it as shown below. Wi

0
S
Practitioner
1 year ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to make API calls with the Python SDK [Cognite Official]

Hello,I’m Pierre, data engineer at Cognite. I’ll be posting, on a regular basis, some useful tips about Cognite Data Fusion and its use. If you have any questions or remarks (technical or not), don’t hesitate to reach out to me or to create a new topic in Cognite Hub.  Some of the features of the Cognite API are not available in the Python SDK. You can still use those features by making a call to the desired endpoint. The SDK facilitates this process thanks to the get, post, put and delete methods of the Cognite client (see: https://cognite-docs.readthedocs-hosted.com/projects/cognite-sdk-python/en/latest/cognite.html#cogniteclient). Once your client is set up, all the auth process is taken care of by itself, it is very straight forward. Example:payload = {...}endpoint_url = “https://api.cognitedata.com/api/v1/projects/my-project/context/diagram/detect/”response = client.post(endpoint_url, payload) When a payload is needed, you can refer to the documentation to get insights on its stru

0
Pierre Pernot
Practitioner
1 year ago
C
Practitioner
Chyrus RameshPractitioner
posted in How-To Guides

How to setup Asynchronous functions using “wait=False” parameter in Cognite Data Fusion [Cognite Official]

Introduction to Asynchronous Programming in Python SDKAsynchronous programming in Python is a powerful way to handle multiple tasks at once, without blocking the execution of the main program.First, let's start by understanding what asynchronous programming is and how it works. In traditional synchronous programming, a program will execute one task at a time, waiting for each task to complete before moving on to the next. This can be very slow and inefficient when dealing with large amounts of data or slow-performing tasks.On the other hand, asynchronous programming allows a program to execute multiple tasks at the same time, without waiting for each task to complete before moving on to the next. This is achieved by using a technique called "non-blocking I/O," which allows the program to continue executing other tasks while waiting for input or output. The below image gives a better understanding of the difference between synchronous and asynchronous functions. In this Hub post, we wil

0
C
Practitioner
1 year ago
V
Practitioner
Viraj DhanushaPractitioner
published in How-To Guides

How to create a .HAR file using different web browsers [Cognite Official]

As Support team we can sometimes ask you to share a .HAR file when you report an issue in a web application.HAR is the short form for HTTP ARchive format, which tracks all the logging of web browser's interaction with a site.It will save time if you already attach this file when you reach out to us initially.  If you are experiencing any of the below issues, a .HAR file will be useful: Performance issues: slow page loading, timeouts when performing certain task Page rendering issues: incorrect page format, missing information Issues in the UI    If you want to know how to generate a file like that, keep on reading! Generating a .HAR file for Google Chrome Open Google Chrome and go to the page where the issue is occurring. Look for the vertical ellipsis button () and select More Tools > Developer Tools. From the panel opened, select the Network tab. Look for a round record button ( 🔴 ) in the upper left corner of the tab, and make sure it is red. If it is grey, click it on

6250
V
Practitioner
1 year ago
Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to use the MQTT hosted extractor [Cognite Official]

MQTT is one of the standards for IoT communication in the industry. It is a lightweight, publish-subscribe, machine to machine network protocol for message queue/message queuing service. As it is widely used, it made sense to have a dedicated extractor for that protocol. At Cognite, we have a hosted extractor for MQTT, that facilitates the extraction of data from a broker. “Hosted” means it runs on our infrastructure and you don’t have to deploy anything on yours to extract data: you just have to configure it. In this article, we will see how to configure and use it. An example scenario involving MQTT could be sensors installed on industrial machines  publishing timeseries data like temperature, pression etc., to a broker. Data would be published to different topics. When extracting data from the broker, with the Cognite extractor, we can have different behaviors based on the topic. Here, we will publish data to four different topics, on an MQTT broker. Those topics are named as follow

0
Pierre Pernot
Practitioner
3 months ago
J
Practitioner
Johan Løwig Rambech DahlPractitioner
posted in How-To Guides

How to run and test Cognite DB-extractor v3

Purpose:Test v3 of the db-extractor as easily as possible with as little local set-up as needed. Outline:This example will use the Docker version of the db-extractor, and also use Docker to make a temporary PostgreSQL database. If you use another version of the extractor (like a Windows service) and/or want to use a database already installed on your local computer, it should still be fairly simple to use this example and make your own changes where applicable. Pre-requisites:Docker (can be avoided if you run a non-containerized extractor and have access to a database) A CDF-project with IdP-authentication Tenant Client Secret Capabilities connected to the client used in the extractor: Raw: List, Read, Write Extraction Pipelines Runs: Write One Extraction pipeline  Steps:Create a containerized PostgreSQL database Open a new terminal Get docker image for the postgreSQL database (only needed once). Run: docker pull postgres Create and run Postgres Container (only neede

0
J
Practitioner
2 months ago
G
Seasoned Practitioner
Gaetan HelnessSeasoned Practitioner
posted in How-To Guides

How to get started with a sandbox using Cognite replicator [Cognite Official]

If you have a sandbox that you would like to populate with data to get started with Cognite Data Fusion, you can use Open Industrial Data. We have built a Cognite replicator for that purpose. Disclaimer: This component is Community content and not officially supported by Cognite. Bugs and changes will be fixed on a best effort basis. Feel free to open issues and pull requests, we will review them as soon as we can.  How to use the Cognite replicator? You have detailed instructions on Github. You have information on how to run it as python package, docker container or a standalone script and authentication via user impersonation and client credentials. How do I authenticate to my source and destination project? What permissions do I need? You have 2 ways of authenticating to your source and your destination:via user impersonation. the script will be run on behalf of your user, you should therefore have the required permissions (read/write on the resources you want to write to/from + gro

0
G
Seasoned Practitioner
1 year ago

Badge winners

  • Cognite Data Engineer Basics - Integrate
    Sai Meher Tejahas earned the badge Cognite Data Engineer Basics - Integrate
  • Cognite Domain Expert Basics
    Flavio Costahas earned the badge Cognite Domain Expert Basics
  • Cognite Domain Expert Basics
    Joe Wuhas earned the badge Cognite Domain Expert Basics
  • Cognite Data Fusion Fundamentals
    Jeevisha Sehgalhas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    Brien Abellahas earned the badge Cognite Data Fusion Fundamentals
Show all badges
Terms & ConditionsCookie Settings