Developer and User Community

Join the discussions about delivering apps and data-driven decisions powered by Cognite Data Fusion. Click the + CREATE TOPIC button in the menu bar to start the conversation.

  • 552 Topics
  • 1,771 Replies

552 Topics

David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

What is Flexible Data Modeling

This document outlines a concept that CDF has been under development for the last 2 years. As mentioned during the disclaimer above, please use this Cognite Hub group and share any feedback you have around flexible data modeling.This document is reflecting some very early thinking from the App Dev Journey team, and is a mental model that will likely change overtime with your feedback!What is a Data Model​A data model enables users to customize the shape, structure their expectation of data. It plays a crucial part in building solutions (like data science models, mobile and web apps). But it is also is the core of an ontology, knowledge graphs, or industry standard.There are some crucial reasons why data models are effective for the industrial space. Data modeling enables explicit language, flexible customization, governed iteration, and enhanced accessibility towards data. Let's dive further into each of these qualities of data modeling. Data Model is Explicit​A data model needs to be

0
David Liu
Seasoned Practitioner
1 year ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

Getting started with Flexible Data Modeling

The best way to play around with Flexible Data Modeling in CDF is through Templates.  Please be aware of the following before getting started! Currently, the App Dev Journey team at Cognite is working on a new Schema Service that will replace Templates towards the end of 2022. For more details on the differences between Schema Service and Templates, go here.There will need to be a migration between Templates and Schema Service in the second half of 2022. The migration will be simple for the data modeling features and querying features as they will be both be using a GraphQL interface. However, moving the data could be a more difficult migration.Hence we highly recommend continuing with Templates for POCs and early experimentation. You can also productionalize with Templates, but please reach out further clarification around quota and SLAs on this Cognite Hub group. Click to see the official documentation on Getting Started with Templates

1
A
Practitioner
1 year ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

Frequently Asked Questions (FAQ) - Charts

Below we have outlined several frequently asked questions and their corresponding answers.Don’t see the answer to your question? Post as a reply in this thread and we’ll be sure to answer and/or add it to the FAQ list below! Protip: Use [cmd+f] or [cntrl+f] to search for keywords related to your question. FAQsHow do I create a new chart?You can easily create a new chart by clicking the + New Chart button on the landing page after entering How do I edit the name of my chart? From the landing page, click the vertical “•••” to the right of the chart’s current name to reveal a drop-down options menu. Click the “Rename” option to open the text box. When finished, you can save you change by clicking the “Ok” button or by pressing “enter” or “return” on your keyboard to save the name. To cancel, click the “Cancel” button or press “escape” on your keyboard. While viewing the chart itself, mouse over the chart’s name in the top menu bar to reveal the  icon.   Remember, it’s important to choose

5288
P
Active
10 months ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

Charts: I have a feature request. What do I do?

Have a feature request or idea for a new and improved functionality?Then we want to hear all about it! We’re constantly working to improve the Charts product experience.You can choose to either create a dedicated post (topic) in the Charts group by clicking the Create topic button OR simply post a reply below in this thread. Remember to say whether your feature request is Nice to have, Important, or Critical to you and why. Screenshots, sketches, or explanatory videos are also encouraged. 

45835
Anita Hæhre
Seasoned Practitioner
6 months ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

Frequently Asked Questions (FAQ) - Flexible Data Modeling

Below we have outlined several frequently asked questions and their corresponding answers.Don’t see the answer to your question? Post as a reply in this thread and we’ll be sure to answer and/or add it to the FAQ list below!Protip: Use [cmd+f] or [cntrl+f] to search for keywords related to your question.Be sure to also let us know what else we can help answer here in the comments below! Q: What is the plans with Templates going forward?Please check out the documentation hereor https://pr-1023.docs.preview.cogniteapp.com/dev/guides/data-modeling/get-started#templates-future Q: Will there be new features built on top of Templates?Yes and no. We will not be doing any development of backend features (like new capabilities) into Templates. However, we will actually be building a lot of additional tooling on top of Templates (better UI, visualization layers, code generation, CLI etc.). Stay tuned in this Cognite Hub to see how you can use these tooling. 

2
Anders Hafreager
Practitioner
11 months ago
P
Practitioner
Patrick MishimaProduct Specialist
posted in Developer and User Community

Azure with Cognite Data Fusion - Data Integration Part 1

Hello All! As this is my first time writing, I will do a quick introduction. My name is Patrick Mishima, and I am one of the Cognite Product Specialists. My main specialization is data integration and governance. I've worked with data since 2011 with different roles, projects, and job titles all related to data management. I also have experience with various ETL and BI tools, mainly from Microsoft and SAP and a few others.I lead the Product Specialists team, and we'll soon have more articles to share our knowledge and experience with you.Today and in my following articles, I will focus on the different options that Microsoft Azure and Cognite offer on data integration between our platforms and tools. But first, let's talk about a few essential and fundamental points when working with cloud providers. Using a cloud provider, most of the time, we think about saving on infrastructure costs. That is indeed true, but it's not the only truth. When you decide to move to a cloud provider, you

0
P
Practitioner
2 years ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

What is Templates, Schema Service and ITG?

During the last 2 years, Cognite has taken a very iterative approach to enable flexible data modeling. The key highlight towards this effort can be outlined through 3 key initiatives - Templates, Schema Service and ITG. In this article we will hope to outline the key difference between the 3. Templates Templates is a data modeling capability built on the concepts of GraphQL. Through building a GraphQL data model, you can define the data model that is intended for your solution. Additionally, you can load data into the data model via referencing existing CDF resources or loading in values directly for primitives like String, Boolean, and Float. Lastly, you get the ability to query the data via a GraphQL endpoint, supporting pagination and complex logical filters (with ANDs and ORs).Here is a list of features doable via TemplatesModeling data modelPrimitives (String, Boolean etc.) Relationships between them (1-1, 1-N, N-M). References to Asset, TS, Sequence, File VersioningQueryingBasic

0
David Liu
Seasoned Practitioner
1 year ago
Gabe Prado
Practitioner
Gabe PradoPractitioner
published in Developer and User Community

Learn more about The Cognite Live Product Tour 2023

The Cognite Live Product Tour 2023 is just over a week away! This event is our annual showcase of Cognite Data Fusion functionalities – both those that will be released in our upcoming December 6th release, and  looking ahead at what is to come throughout the new year. We’ll be driving a discussion with you – our community – about Industrial DataOps and the value of empowering Subject Matter Experts in the industrial organization. We hope you’re looking forward to the event as much as we are! Below is an outline of some of the exciting topics we’ll be discussing. WATCH THE COGNITE LIVE PRODUCT TOUR 2023 RECORDING   INDUSTRIAL DATAOPS AND THE SUBJECT MATTER EXPERTIndustrial DataOps is about breaking down silos and optimizing the broad availability and usability of industrial data. Learn about why we are focusing on enabling the Subject Matter Expert throughout 2023 and the two main challenges that SMEs in every industrial sector face. COGNITE DATA FUSION: HIGHLIGHTS OF OUR DECEMBER 2022

3820
Gabe Prado
Practitioner
1 year ago
T
Practitioner
tibor.vanczaPractitioner
asked in Developer and User Community

Streamlit - Async tasks problem

Currently Cognite’s Streamlit apps have issues with for example:with st.spinner('Wait for it...'): time.sleep(5)st.success('Done!')The code itself should wait 5 seconds before writing out Done!, but is executed instantly. Another example is when you try to make your application responsive while waiting for some data to arrive from a remote location. For example, when you want to print out a wait message for the user because sometimes it takes some seconds to load the data, but the app doesn't work as intended. In the following example I am trying to print out a message while waiting for the function to finish, but it fails:call_func_button = st.button("Call function with current parameters", key="button_func_call")if call_func_button: func_call = example_function.call(data=input_data, wait=False) st.info("Waiting for function...") while func_call.status == "Running": func_call.update() if func_call.status == "Failed": st.error("Function failed to run!")

4
T
Practitioner
2 months ago
P
Active
paaleActive
asked in Developer and User Community

ResourceWarning: unclosed SSL socket

I get the following warning every time I create a CogniteClient: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.xx.xx.xx', 49572), raddr=('40.xx.xx.xx', 443)> Is this caused by the connection handling in Cognite SDK? I’m using the following Python modules:cognite-sdk-4.11.0cognite-sdk-core-2.56.1

10
P
Active
1 year ago
B
Practitioner
Bernt ReinhardtSenior Data Engineer
posted in Developer and User Community

Migrating a CDF project from legacy to OIDC authentication using Azure as Identity Provider (idP)

This article is intended for customers that are planning to migrate, or are in the process of migrating, to Cognite Data Fusion’s (CDF) new authentication system which will be based on Open ID Connect (OIDC). It will cover in detail all aspects of this migration process, and hopefully help making this migration process as simple as possible.What is OIDC? OpenID Connect is an industry-standard protocol supported by many popular identity providers (IdP), such as Azure Active Directory (Azure AD), Microsoft's cloud-based identity and access management service.For more information about OIDC, please visit the official website. What are the benefits of using OIDC over CDF’s legacy authentication? The greatest benefit of using OIDC over CDF’s legacy authentication, is that the user’s identity will now have a single source of truth – the Identity Provider (idP). This simplifies identity management greatly, and makes it more secure by leveraging the idP’s strength – namely identity management.

0
B
Practitioner
1 year ago
Anders Hafreager
Practitioner
Anders HafreagerPractitioner
posted in Developer and User Community

Flexible data modeling has reached Beta

We proudly announce that Flexible Data Modeling has reached public beta. Flexible data modelling in CDF enables you to model industrial data the way you understand it in a self-documented way so that domain experts can find, recognize and understand the data without a third-party manual.pdf. Your data is stored in one single knowledge graph, and empowers users to search, filter and aggregate data according to their needs through various data models on top of the knowledge graph. Flexible data modeling enables rapid scaling, whether it is building an app, data science model, dashboard, or other use cases, we hope that this capability can be a great toolkit in helping you to express data in the most intuitive way.In the Beta version you can Create and manage data models in Fusion and using a cli tool Ingest data into the data model using CDF transformations Query data using GraphQL with powerful search and filtering in Fusion or your own applications with even more powerful features

0
Anders Hafreager
Practitioner
1 year ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community

Charts updates & new features (see post for video walkthrough)

 Hello Charts Early Adopter Community,There have been several new features and functionalities released lately, with some being released just today. I’ve recorded a video (below) to explain several of these core features in detail, namely – calculations running on individual data points, not aggregates.You can also scroll down to read the written details.Please do leave comments below with questions and feedback!DetailsCharts UI/UX Calculations are now running on individual data points, not aggregates In the past, calculations have been running on time series aggregates. This meant that, although calculations were approximately calculations were often not correct his is a very important new feature that greatly increases the accuracy and trustworthiness of calculation results in Charts. Watch the above video or see the slides, below, for more info. If you receive a warning that looks like this (see image, below), then it means that the results of the data has been downsampled to perfo

9
O
Committed
1 year ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Developer and User Community

transformations-cli with FDM

Hi,I am trying to run transformations-cli locally with below command:transformations-cli deploy . (The current directory has manifest.yaml and transformation.sql files) I am getting below error:Deploying transformations...Failed to parse transformation config, please check that you conform required fields and format: Invalid config: can not match type "dict" to any type of "destination" union: typing.Union[cognite.transformations_cli.commands.deploy.transformation_types.DestinationType, cognite.transformations_cli.commands.deploy.transformation_types.DestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.RawDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.SequenceRowsDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.AlphaDMIDestinationConfig] Here is destination section of manifest file:destination:  viewSpaceExternalId: my-model-space-id  viewExternalId: CDFTimeSeries  viewVersion: 0_2  i

17
S
Seasoned
7 months ago
Pedro Lindeza
Practitioner
Pedro LindezaPractitioner
posted in Developer and User Community

Cognite Data Fusion Bootcamp - 1st Edition

December 15th 2022 - SLB Offices @ Pune, IndiaOn the week of Dec 12th to Dec 16th, Maureen Byrne (Software Engineer), Maria Fonseca (Technical Support Engineer) and me (Data Engineer) travelled to Pune, India, to deliver the first Cognite Data Fusion Bootcamp for eighteen participants from SLB and Infosys. This training delivery marks a milestone for customer and partners of Cognite: this is the first training where Cognite instructors guide them through an end-to-end deployment of Cognite Data Fusion, including data integrations and solution-building.The bootcamp is an intensive 5-day training program that focuses on a complete deployment, bringing the different steps together with DevOps best practices from CDF bootstrapping to solution deployment, monitoring, and operations. Congratulations to the participants of the first bootcamp for gaining their Cognite Data Fusion Bootcamp Certificate!The three of us were very impressed by how fast the participants adopted DataOps tools and pra

1
hescalona
Practitioner
11 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Developer and User Community

Flexible data modeling - transformation to JsonObject type

Hi, I have created one FDM model having JsonObject field. In the transformation query I referred that field , example query:SELECT Series_Title as externalId,       Series_Title as name,       Overview as description,       int(Gross) as gross,       float(IMDB_Rating) as imdbRating,       int(Runtime) as runTime,       int(Released_Year) as releasedYear,       '{"key1":"value1","key2":{"key3":12,"key4":"hello","key5":{"key6":10}}}' as dataFROM movies.moviesFDM model is:type Movie {  name: String!  description: String  watchedIt: Boolean  imdbRating: Float  releasedYear: Int  runTime: Int  gross: Int  actors: [Actor]  data: JSONObject}  The transformation is failing with this error:Unknown property type Json. May be I am missing something, Can you please help?

10
D
Practitioner
8 months ago
Sunil Krishnamoorthy
Seasoned Practitioner
Sunil KrishnamoorthyProduct Manager
posted in Developer and User Community

Try out the new Python SDK for Transformations

This post is a hands-on introduction to the features supported in the Transformations Python SDK.Prerequisites Use Case 1: Triggering Transformations Step 1 - Create RAW Tables Step 2 - Uploading data to RAW using Postgres Gateway Step 3 - Create new SQL Transformations Step 4 - Trigger the transformation from Azure Data Factory Use Case 2: Orchestrating Transformations Step 1 - Create RAW Tables Step 2 - Create new SQL Transformations Step 3 - Orchestrate Transformations in sequence PrerequisitesKnowledge: Basic knowledge of Azure Functions and Azure Data Factory Basic knowledge of Cognite Data Fusion RAW and SQL Transformations Prior experience with Python, Postgres and SQL Required Datasets:Download and Unzip the attached hub.zip file, you should find the below structure Use Case 1 : asset-hierarchy.csv UseCase 2: OID-Asset-hirerachy.csv OID-Timeseries.csv OID-Datapoints.csv Use Case 1: Triggering TransformationsData is extracted from source systems and

6
Ben Brandt
Seasoned
1 year ago
A
Seasoned
Andreas KimsåsSeasoned
asked in Developer and User Community

Timeseries aggregates gives strange results for "average"

Aksels ts is Original data , ts average at 04:00 = 0.125, can only be explained by interpolated values The example above shows that 1h aggregates of type average computed at time-point 03:00 takes into considereation the interpolated values in the time-span 03:00 to 04:00. Is it always like this as long as i choose granualarity 1h ? I could not find the answer to my question here:Aggregation | Cognite documentation

4
A
Seasoned
2 years ago
V
Committed
vidhi.c.jainCommitted
asked in Developer and User Community

Facing issue while writing test cases using monkey_cognite_patch.

Hi Team, We are working on a POC , where we need to create unit test cases where we are facing different issues .For reference we are using code from Cognite HubBelow attaching our code and output expectations .Please help with the approach to resolve this issue ts_retrieved = [    TimeSeries(    id= 3973245479016710,    external_id= "Brazil:3W:Well-Bore-19B_target_throughput",    name= "Brazil:3W:Well-Bore-19B_target_throughput",    is_string= False,    metadata= {},    unit= "Barrel",    asset_id= 3332326574821917,    is_step= False,    description= "target volume for a oil well bore",    security_categories= [],    data_set_id= 8739280581150161,    created_time= "2023-05-02 09:37:04.944000",    last_updated_time= "2023-05-02 09:37:04.944000")]@pytest.fixturedef cognite_client_mock():    with monkeypatch_cognite_client() as client:          client.assets.retrieve().time_series().get().return_value = ts_retrieved        return client   Method code          ts_data = c.assets.retrieve(

10
C
Practitioner
3 months ago
Juan Alvarado
Committed
Juan AlvaradoCommitted
asked in Developer and User Community

any other means of populating FDM besides transforms?

Using transformations to create/update data can be cumbersome especially for complex FDM models. I understand there are plans to generate APIs from models, but are there any alternatives in the short term, such as implementing GraphQL mutations? Are the APIs used by transformations internal to CDF?

8
T
Practitioner
9 months ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

About the Data Science Toolbox Functions

Why are algorithms and data-driven models only available for the few domain experts who also are fluent in advanced software coding? We certainly don’t believe this should be the case.We are making them available to the rest of the world — particularly to non-coding domain experts.For many years  industrial data scientists have been building smart algorithms to solve  complex industrial problems. They are now available in a no-code drag-and-drop intuitive interface. Liberate your data and empower domain experts the tools to drive impact every day.Cognite Charts includes several data science toolboxes that provide subject matter experts (SMEs) out-of-the-box algorithms to process and manipulate data, conduct root cause analysis (RCA) and develop solutions without having to code.The toolboxes cover basic operations, statistical methods, data transformation, and advanced models. They work out-of-the-box with Cognite Charts, and we will continuously add new algorithms, features, and functi

21910
K
Seasoned Practitioner
8 months ago
A
Seasoned
Andreas KimsåsSeasoned
asked in Developer and User Community

How to overcome metadata key length and max keys limitation

On serveral occations we have encountered limitations in metadata key length, most frequently when flattening JSON-formatted strings from our event stream data source systems. When the source system presents nested structure of as many as 4 levels we frequently encounter metadata keys that require more than 128 bytes. Up to now we have “solved” the issue by abbreviating the metadatakeys at the price of higher maintenance cost of the code and more importantly, that end-users get the perception that we have transformed the data or even dont understand what it represents. We now consider moving towards a solution where we simply put the entire JSON-formatted string into one single metadata value field, and leave to front-end teams and end users to flatten the structure. We have a similar issue with max number of metadata keys for timesseries (16). Question 1) Could you please suggest other, better options for handling these metadata limitations? Question 2) Will templates come to the resc

11
T
Practitioner
1 year ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
published in Developer and User Community

Charts: I found a bug or have a question. What do I do?

Found a bug? Have a question about how something works? We want to hear about it!You can choose to either create a dedicated post (topic) in the Charts group by clicking the Create topic button OR simply post a reply below in this thread. Remember to include a screenshot or video to help the product team best understand what exactly you’re talking about or referring to. 

19413
absqueued
Practitioner
1 month ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community

Charts updates & new features

Hello Charts Early Adopter Community,I know it’s been a bit quiet in this group lately, but rest assured, the team and I have been hard at work. We have just released several new features and fixes to production. You can scroll down to read about each of them in detail.Please do leave comments below with questions and feedback!What’s coming next?We’re in the middle of our development cycle in preparation for the next major release of Cognite Data Fusion in December 2022. There are the two major features our team is working on that you can look forward to having available:Moving Charts (charts.cogniteapp.com) into Cognite Data Fusion (fusion.cognite.com) We’ve received plenty of feedback that our product experience can and need to be more tightly integrated and we’ve heard you loud and clear. As you all know, Charts is currently available on charts.cogniteapp.com, which is separate from Cognite Data Fusion (fusion.cognite.com). Despite starting out on its standalone URL, Charts has alw

1
R
Committed
1 year ago
C
Seasoned Practitioner
Crystal Connor RichardsSeasoned Practitioner
asked in Developer and User Community

Charts: Help with 12 month BTU Rolling Average - says too long time frame

 Hello  @brendan Buckbee at Celanese has created a chart for the 12 month, but gets an error message. Is there any way that this can be created into a chart. I have attached a screenshot of the calcs. cc: @Kylie R  

11
Simon Funke
Practitioner
1 year ago

Badge winners

  • Cognite Data Engineer Basics - T&C
    sumitgawalihas earned the badge Cognite Data Engineer Basics - T&C
  • Cognite Data Engineer Basics - Integrate
    sumitgawalihas earned the badge Cognite Data Engineer Basics - Integrate
  • Cognite Domain Expert Basics
    Mitch Rashhas earned the badge Cognite Domain Expert Basics
  • Cognite Domain Expert Basics
    Rajeshhas earned the badge Cognite Domain Expert Basics
  • Cognite Data Scientist Basics
    Abdulmohsen Alateeqhas earned the badge Cognite Data Scientist Basics
Show all badges
Terms & ConditionsCookie Settings