COGNITE HUB

Developer and User Community

Join the discussions about delivering apps and data-driven decisions powered by Cognite Data Fusion. Click the + CREATE TOPIC button in the menu bar to start the conversation.

  • 389 Topics
  • 1,207 Replies

389 Topics

David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

What is Flexible Data Modeling

This document outlines a concept that CDF has been under development for the last 2 years. As mentioned during the disclaimer above, please use this Cognite Hub group and share any feedback you have around flexible data modeling.This document is reflecting some very early thinking from the App Dev Journey team, and is a mental model that will likely change overtime with your feedback!What is a Data Model​A data model enables users to customize the shape, structure their expectation of data. It plays a crucial part in building solutions (like data science models, mobile and web apps). But it is also is the core of an ontology, knowledge graphs, or industry standard.There are some crucial reasons why data models are effective for the industrial space. Data modeling enables explicit language, flexible customization, governed iteration, and enhanced accessibility towards data. Let's dive further into each of these qualities of data modeling. Data Model is Explicit​A data model needs to be

0
David Liu
Seasoned Practitioner
1 year ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

Getting started with Flexible Data Modeling

The best way to play around with Flexible Data Modeling in CDF is through Templates.  Please be aware of the following before getting started! Currently, the App Dev Journey team at Cognite is working on a new Schema Service that will replace Templates towards the end of 2022. For more details on the differences between Schema Service and Templates, go here.There will need to be a migration between Templates and Schema Service in the second half of 2022. The migration will be simple for the data modeling features and querying features as they will be both be using a GraphQL interface. However, moving the data could be a more difficult migration.Hence we highly recommend continuing with Templates for POCs and early experimentation. You can also productionalize with Templates, but please reach out further clarification around quota and SLAs on this Cognite Hub group. Click to see the official documentation on Getting Started with Templates

1
A
Practitoner
11 months ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

Frequently Asked Questions (FAQ) - Charts

Below we have outlined several frequently asked questions and their corresponding answers.Don’t see the answer to your question? Post as a reply in this thread and we’ll be sure to answer and/or add it to the FAQ list below! Protip: Use [cmd+f] or [cntrl+f] to search for keywords related to your question. FAQsHow do I create a new chart?You can easily create a new chart by clicking the + New Chart button on the landing page after entering How do I edit the name of my chart? From the landing page, click the vertical “•••” to the right of the chart’s current name to reveal a drop-down options menu. Click the “Rename” option to open the text box. When finished, you can save you change by clicking the “Ok” button or by pressing “enter” or “return” on your keyboard to save the name. To cancel, click the “Cancel” button or press “escape” on your keyboard. While viewing the chart itself, mouse over the chart’s name in the top menu bar to reveal the  icon.   Remember, it’s important to choose

4438
P
Active
4 months ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

Charts: I have a feature request. What do I do?

Have a feature request or idea for a new and improved functionality?Then we want to hear all about it! We’re constantly working to improve the Charts product experience.You can choose to either create a dedicated post (topic) in the Charts group by clicking the Create topic button OR simply post a reply below in this thread. Remember to say whether your feature request is Nice to have, Important, or Critical to you and why. Screenshots, sketches, or explanatory videos are also encouraged. 

43935
Anita Hæhre
Seasoned Practitioner
27 days ago
P
Practitoner
Patrick MishimaProduct Specialist
posted in Developer and User Community

Azure with Cognite Data Fusion - Data Integration Part 1

Hello All! As this is my first time writing, I will do a quick introduction. My name is Patrick Mishima, and I am one of the Cognite Product Specialists. My main specialization is data integration and governance. I've worked with data since 2011 with different roles, projects, and job titles all related to data management. I also have experience with various ETL and BI tools, mainly from Microsoft and SAP and a few others.I lead the Product Specialists team, and we'll soon have more articles to share our knowledge and experience with you.Today and in my following articles, I will focus on the different options that Microsoft Azure and Cognite offer on data integration between our platforms and tools. But first, let's talk about a few essential and fundamental points when working with cloud providers. Using a cloud provider, most of the time, we think about saving on infrastructure costs. That is indeed true, but it's not the only truth. When you decide to move to a cloud provider, you

0
P
Practitoner
1 year ago
Gabe Prado
Practitoner
Gabe PradoPractitoner
published in Developer and User Community

Learn more about The Cognite Live Product Tour 2023

The Cognite Live Product Tour 2023 is just over a week away! This event is our annual showcase of Cognite Data Fusion functionalities – both those that will be released in our upcoming December 6th release, and  looking ahead at what is to come throughout the new year. We’ll be driving a discussion with you – our community – about Industrial DataOps and the value of empowering Subject Matter Experts in the industrial organization. We hope you’re looking forward to the event as much as we are! Below is an outline of some of the exciting topics we’ll be discussing. WATCH THE COGNITE LIVE PRODUCT TOUR 2023 RECORDING   INDUSTRIAL DATAOPS AND THE SUBJECT MATTER EXPERTIndustrial DataOps is about breaking down silos and optimizing the broad availability and usability of industrial data. Learn about why we are focusing on enabling the Subject Matter Expert throughout 2023 and the two main challenges that SMEs in every industrial sector face. COGNITE DATA FUSION: HIGHLIGHTS OF OUR DECEMBER 2022

3700
Gabe Prado
Practitoner
6 months ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

What is Templates, Schema Service and ITG?

During the last 2 years, Cognite has taken a very iterative approach to enable flexible data modeling. The key highlight towards this effort can be outlined through 3 key initiatives - Templates, Schema Service and ITG. In this article we will hope to outline the key difference between the 3. Templates Templates is a data modeling capability built on the concepts of GraphQL. Through building a GraphQL data model, you can define the data model that is intended for your solution. Additionally, you can load data into the data model via referencing existing CDF resources or loading in values directly for primitives like String, Boolean, and Float. Lastly, you get the ability to query the data via a GraphQL endpoint, supporting pagination and complex logical filters (with ANDs and ORs).Here is a list of features doable via TemplatesModeling data modelPrimitives (String, Boolean etc.) Relationships between them (1-1, 1-N, N-M). References to Asset, TS, Sequence, File VersioningQueryingBasic

0
David Liu
Seasoned Practitioner
1 year ago
B
Practitoner
Bernt ReinhardtSenior Data Engineer
posted in Developer and User Community

Migrating a CDF project from legacy to OIDC authentication using Azure as Identity Provider (idP)

This article is intended for customers that are planning to migrate, or are in the process of migrating, to Cognite Data Fusion’s (CDF) new authentication system which will be based on Open ID Connect (OIDC). It will cover in detail all aspects of this migration process, and hopefully help making this migration process as simple as possible.What is OIDC? OpenID Connect is an industry-standard protocol supported by many popular identity providers (IdP), such as Azure Active Directory (Azure AD), Microsoft's cloud-based identity and access management service.For more information about OIDC, please visit the official website. What are the benefits of using OIDC over CDF’s legacy authentication? The greatest benefit of using OIDC over CDF’s legacy authentication, is that the user’s identity will now have a single source of truth – the Identity Provider (idP). This simplifies identity management greatly, and makes it more secure by leveraging the idP’s strength – namely identity management.

0
B
Practitoner
1 year ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

Frequently Asked Questions (FAQ) - Flexible Data Modeling

Below we have outlined several frequently asked questions and their corresponding answers.Don’t see the answer to your question? Post as a reply in this thread and we’ll be sure to answer and/or add it to the FAQ list below!Protip: Use [cmd+f] or [cntrl+f] to search for keywords related to your question.Be sure to also let us know what else we can help answer here in the comments below! Q: What is the plans with Templates going forward?Please check out the documentation hereor https://pr-1023.docs.preview.cogniteapp.com/dev/guides/data-modeling/get-started#templates-future Q: Will there be new features built on top of Templates?Yes and no. We will not be doing any development of backend features (like new capabilities) into Templates. However, we will actually be building a lot of additional tooling on top of Templates (better UI, visualization layers, code generation, CLI etc.). Stay tuned in this Cognite Hub to see how you can use these tooling. 

2
Anders Hafreager
Practitoner
5 months ago
Anders Hafreager
Practitoner
Anders HafreagerPractitoner
posted in Developer and User Community

Flexible data modeling has reached Beta

We proudly announce that Flexible Data Modeling has reached public beta. Flexible data modelling in CDF enables you to model industrial data the way you understand it in a self-documented way so that domain experts can find, recognize and understand the data without a third-party manual.pdf. Your data is stored in one single knowledge graph, and empowers users to search, filter and aggregate data according to their needs through various data models on top of the knowledge graph. Flexible data modeling enables rapid scaling, whether it is building an app, data science model, dashboard, or other use cases, we hope that this capability can be a great toolkit in helping you to express data in the most intuitive way.In the Beta version you can Create and manage data models in Fusion and using a cli tool Ingest data into the data model using CDF transformations Query data using GraphQL with powerful search and filtering in Fusion or your own applications with even more powerful features

0
Anders Hafreager
Practitoner
7 months ago
S
Committed
samirhajCommitted
asked in Developer and User Community

detNet RestAPI Cognite SDK example

Hello,I am quite new on usinf RestAPI. My company has the license to use CDF. I also inderstood that Cognite has developed dotNet RestApi SDK.I searched a bit but I could not find an example of how we can use this SDK for data retrieval especially for time-series data. How could I get a snippet of code showing an example how to use this?Thanks. 

23
A
Seasoned
12 days ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community

Charts updates & new features (see post for video walkthrough)

 Hello Charts Early Adopter Community,There have been several new features and functionalities released lately, with some being released just today. I’ve recorded a video (below) to explain several of these core features in detail, namely – calculations running on individual data points, not aggregates.You can also scroll down to read the written details.Please do leave comments below with questions and feedback!DetailsCharts UI/UX Calculations are now running on individual data points, not aggregates In the past, calculations have been running on time series aggregates. This meant that, although calculations were approximately calculations were often not correct his is a very important new feature that greatly increases the accuracy and trustworthiness of calculation results in Charts. Watch the above video or see the slides, below, for more info. If you receive a warning that looks like this (see image, below), then it means that the results of the data has been downsampled to perfo

9
O
Committed
10 months ago
A
Seasoned
Andreas KimsåsSeasoned
asked in Developer and User Community

Timeseries aggregates gives strange results for "average"

Aksels ts is Original data , ts average at 04:00 = 0.125, can only be explained by interpolated values The example above shows that 1h aggregates of type average computed at time-point 03:00 takes into considereation the interpolated values in the time-span 03:00 to 04:00. Is it always like this as long as i choose granualarity 1h ? I could not find the answer to my question here:Aggregation | Cognite documentation

4
A
Seasoned
2 years ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Developer and User Community

transformations-cli with FDM

Hi,I am trying to run transformations-cli locally with below command:transformations-cli deploy . (The current directory has manifest.yaml and transformation.sql files) I am getting below error:Deploying transformations...Failed to parse transformation config, please check that you conform required fields and format: Invalid config: can not match type "dict" to any type of "destination" union: typing.Union[cognite.transformations_cli.commands.deploy.transformation_types.DestinationType, cognite.transformations_cli.commands.deploy.transformation_types.DestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.RawDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.SequenceRowsDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.AlphaDMIDestinationConfig] Here is destination section of manifest file:destination:  viewSpaceExternalId: my-model-space-id  viewExternalId: CDFTimeSeries  viewVersion: 0_2  i

17
S
Seasoned
1 month ago
Juan Alvarado
Committed
Juan AlvaradoCommitted
asked in Developer and User Community

any other means of populating FDM besides transforms?

Using transformations to create/update data can be cumbersome especially for complex FDM models. I understand there are plans to generate APIs from models, but are there any alternatives in the short term, such as implementing GraphQL mutations? Are the APIs used by transformations internal to CDF?

8
T
Practitoner
3 months ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

About the Data Science Toolbox Functions

Why are algorithms and data-driven models only available for the few domain experts who also are fluent in advanced software coding? We certainly don’t believe this should be the case.We are making them available to the rest of the world — particularly to non-coding domain experts.For many years  industrial data scientists have been building smart algorithms to solve  complex industrial problems. They are now available in a no-code drag-and-drop intuitive interface. Liberate your data and empower domain experts the tools to drive impact every day.Cognite Charts includes several data science toolboxes that provide subject matter experts (SMEs) out-of-the-box algorithms to process and manipulate data, conduct root cause analysis (RCA) and develop solutions without having to code.The toolboxes cover basic operations, statistical methods, data transformation, and advanced models. They work out-of-the-box with Cognite Charts, and we will continuously add new algorithms, features, and functi

20910
K
Seasoned Practitioner
3 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Developer and User Community

Flexible data modeling - transformation to JsonObject type

Hi, I have created one FDM model having JsonObject field. In the transformation query I referred that field , example query:SELECT Series_Title as externalId,       Series_Title as name,       Overview as description,       int(Gross) as gross,       float(IMDB_Rating) as imdbRating,       int(Runtime) as runTime,       int(Released_Year) as releasedYear,       '{"key1":"value1","key2":{"key3":12,"key4":"hello","key5":{"key6":10}}}' as dataFROM movies.moviesFDM model is:type Movie {  name: String!  description: String  watchedIt: Boolean  imdbRating: Float  releasedYear: Int  runTime: Int  gross: Int  actors: [Actor]  data: JSONObject}  The transformation is failing with this error:Unknown property type Json. May be I am missing something, Can you please help?

10
D
Practitoner
2 months ago
A
Seasoned
Andreas KimsåsSeasoned
asked in Developer and User Community

How to overcome metadata key length and max keys limitation

On serveral occations we have encountered limitations in metadata key length, most frequently when flattening JSON-formatted strings from our event stream data source systems. When the source system presents nested structure of as many as 4 levels we frequently encounter metadata keys that require more than 128 bytes. Up to now we have “solved” the issue by abbreviating the metadatakeys at the price of higher maintenance cost of the code and more importantly, that end-users get the perception that we have transformed the data or even dont understand what it represents. We now consider moving towards a solution where we simply put the entire JSON-formatted string into one single metadata value field, and leave to front-end teams and end users to flatten the structure. We have a similar issue with max number of metadata keys for timesseries (16). Question 1) Could you please suggest other, better options for handling these metadata limitations? Question 2) Will templates come to the resc

11
T
Practitoner
8 months ago
Sunil Krishnamoorthy
Seasoned Practitioner
Sunil KrishnamoorthyProduct Manager
posted in Developer and User Community

Try out the new Python SDK for Transformations

This post is a hands-on introduction to the features supported in the Transformations Python SDK.Prerequisites Use Case 1: Triggering Transformations Step 1 - Create RAW Tables Step 2 - Uploading data to RAW using Postgres Gateway Step 3 - Create new SQL Transformations Step 4 - Trigger the transformation from Azure Data Factory Use Case 2: Orchestrating Transformations Step 1 - Create RAW Tables Step 2 - Create new SQL Transformations Step 3 - Orchestrate Transformations in sequence PrerequisitesKnowledge: Basic knowledge of Azure Functions and Azure Data Factory Basic knowledge of Cognite Data Fusion RAW and SQL Transformations Prior experience with Python, Postgres and SQL Required Datasets:Download and Unzip the attached hub.zip file, you should find the below structure Use Case 1 : asset-hierarchy.csv UseCase 2: OID-Asset-hirerachy.csv OID-Timeseries.csv OID-Datapoints.csv Use Case 1: Triggering TransformationsData is extracted from source systems and

6
Ben Brandt
Seasoned
9 months ago
P
Practitoner
Pedro LindezaPractitoner
posted in Developer and User Community

Cognite Data Fusion Bootcamp - 1st Edition

December 15th 2022 - SLB Offices @ Pune, IndiaOn the week of Dec 12th to Dec 16th, Maureen Byrne (Software Engineer), Maria Fonseca (Technical Support Engineer) and me (Data Engineer) travelled to Pune, India, to deliver the first Cognite Data Fusion Bootcamp for eighteen participants from SLB and Infosys. This training delivery marks a milestone for customer and partners of Cognite: this is the first training where Cognite instructors guide them through an end-to-end deployment of Cognite Data Fusion, including data integrations and solution-building.The bootcamp is an intensive 5-day training program that focuses on a complete deployment, bringing the different steps together with DevOps best practices from CDF bootstrapping to solution deployment, monitoring, and operations. Congratulations to the participants of the first bootcamp for gaining their Cognite Data Fusion Bootcamp Certificate!The three of us were very impressed by how fast the participants adopted DataOps tools and pra

1
hescalona
Practitioner
5 months ago
P
Practitoner
Patrick MishimaProduct Specialist
posted in Developer and User Community

Azure with Cognite Data Fusion - Data Integration Part 2

 It's been a while, but as promised, here is the follow-up post to my previous article. If you haven't read the previous article, you can find it here.In the first part, I shared the basic concepts about data integration. Today I will continue on the same theme, but the focus will be on latency and frequency. They are related concepts, but not necessarily the same. Why latency and frequency? They are crucial to defining your data pipeline, and they influence which Azure resources you use to cover your needs.What is data latency and frequency? Data latency is how fast/slow data can be retrieved or stored. Low latency means that the data is available in real-time or close to real-time and is vital in use cases where you need to respond quickly to information. Examples are alerts for critical events on machinery and equipment and online games based on real-time experiences. This leads us to the data frequency concept. Data frequency is how many times in a specific period the data should b

0
P
Practitoner
1 year ago
K
Seasoned Practitioner
Knut VidveiSeasoned Practitioner
posted in Developer and User Community

Redirect from charts.cogniteapp.com to fusion.cognite.com/charts (starting February 7th 2023)

As communicated in this post in December, Cognite Charts is now integrated as a part of fusion.cognite.com. We are planning to retire the stand-alone url charts.cogniteapp.com to have a uniform product experience and single login. Starting February 7th 2023 you will find a message if you access the stand-alone url with instructions on how to login to fusion.cognite.com. You fill find all your charts and data intact, and Charts working as you are used to. There will be a grace period for 2 weeks after February 7th giving room for users to still access their charts from the stand-alone url if you need support to login to fusion.cognite.com.  For those of you who are linking to Charts in your third party applications, Microsoft Teams, etc., this will still function as expected. We will be working on an automatic redirect for coming releases, however still recommend you update the hyperlinks to point to your-organization-name.fusion.cognite.com already now. If you have questions on this, d

1
K
Seasoned Practitioner
4 months ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community

Charts updates & new features

Hello Charts Early Adopter Community,I know it’s been a bit quiet in this group lately, but rest assured, the team and I have been hard at work. We have just released several new features and fixes to production. You can scroll down to read about each of them in detail.Please do leave comments below with questions and feedback!What’s coming next?We’re in the middle of our development cycle in preparation for the next major release of Cognite Data Fusion in December 2022. There are the two major features our team is working on that you can look forward to having available:Moving Charts (charts.cogniteapp.com) into Cognite Data Fusion (fusion.cognite.com) We’ve received plenty of feedback that our product experience can and need to be more tightly integrated and we’ve heard you loud and clear. As you all know, Charts is currently available on charts.cogniteapp.com, which is separate from Cognite Data Fusion (fusion.cognite.com). Despite starting out on its standalone URL, Charts has alw

1
R
Committed
7 months ago
C
Seasoned Practitioner
Crystal Connor RichardsSeasoned Practitioner
asked in Developer and User Community

Charts: Help with 12 month BTU Rolling Average - says too long time frame

 Hello  @brendan Buckbee at Celanese has created a chart for the 12 month, but gets an error message. Is there any way that this can be created into a chart. I have attached a screenshot of the calcs. cc: @Kylie R   

11
Simon Funke
Practitoner
11 months ago
T
Active
Trond SaureActive
asked in Developer and User Community

Using a FDM Data model in transformations

I am currently working on a concept for representing tag data in a simple graph (subject - predicate - object - style)An instance of the triple could be something like this D-20VA00001 - hasDryWeight - 5000In parallell I have instanciated another graph holding ontology data and mapping (like CFIHOS, mapping between CFIHOS class and function code and sap referance data etc.) Lets call it the meta-graph.What I want to do is to take a “traditional” tag as input, map the function code on the tag to corresponding CFIHOS-class, and get required attributes in return from the meta-graph. Further I then want to dynamically instanciate every attribute as relation in the triple’ish graph. In order to do this I need to be able to query a data model (FDM) in a transformation- is that far away?Further, now I am faking a labeled property graph in FDM - is LPG functionality far away? 

7
C
Practitoner
5 months ago

Badge winners

  • Cognite Data Fusion Fundamentals
    Hesam Salehihas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    Dzmitry Holubhas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    poshas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Bootcamp
    wkhalidhas earned the badge Cognite Data Fusion Bootcamp
  • Cognite Data Fusion Fundamentals
    Varvara Vlasyukhas earned the badge Cognite Data Fusion Fundamentals
Show all badges
Terms & ConditionsCookie Settings