Community Discussions

Hello!I'm trying to develop a PowerBI dashboard but encontered some situations where it's impossible to not expand new columns or lists from the nodes (foreign key from dimension table is inside the node) and I understand this drops the performance considerably (sometimes even turns impossible to load the table) of the model in the dashboard.Is there a way to query the data straight from the connector? I know there's the possibility of making simple filter queries but I wasn't able to use other type of OData queries that exists. We need a solution that simplifies this since currently it's impossible to load more complex data models into PowerBI without expanding the nodes.Please understand that I want to keep this situation as low code as possible, so an API or SDK solution is not an option.Thanks!

Hey Everyone!Curious about how to boost your rank and unlock some amazing new abilities in the community? 🚀 If you haven’t checked it out yet, we’ve just shared an exciting update about the perks awaiting you when you reach Level 4 (Practitioner). From marking correct answers to unlimited post editing, these benefits are designed to empower you as a community leader.👉 Check out the full details here!👈Ready to take your engagement to the next level? Head over to the original post for all the details and start leveling up today!

Hello !  I am new to this community but would like to ask the following:   Has anybody any experience extracting a list of DCS Process Alerts to Operators or Operator Overrides on an Hourly or Daily basis into CDF ?   This data resides in the OT and is sometimes extracted and made available in reports in the IT domain, e.g., by Yokogawa.   As I see it, this is an important indicator for stable and well managed production operations, and also a good predictor of upcoming threats to production.

dear all I was attempting to perform data aggregation based on the date. I am retrieving online data into CDF, which is updated every 2-3 minutes. I am trying to aggregate the data so that the date is updated every 24 hours instead of every 2-4 minutes.I used this code to obtain the list of columns in my data frame.# Check the structure of the DataFrame, including column names and the first few rowsprint(dp.columns)  # This will show all column namesdp.head()          # This will show the first few rows of the DataFrame however, I got only this after running the code I tried to use another code again the first column is not date column.  

Dear developer community,The Cognite product team building the GraphQL API for our Data Modelling API is planning to upgrade third party libraries used in the implementation of the service. This is part of our normal product lifecyle and maintenance, and would not normally be something we make a dedicated announcement of. The upgrade brings improved validation, performance, and other fixes.However we have seen that this upgrade is liable to cause issues for some clients that make GraphQL requests that are not compliant with the GraphQL specification. Specifically, primitive types in field values will no longer be coerced into the correct type by the GraphQL server. So, for example, a client sending the string ”true” where the boolean value true would be correct will, after the upgrade, receive an error rather than the server quietly accepting and converting the value.This behavior change should not be an issue for anyone using one of the compliant third party GraphQL client libraries o

Hi Everyone,Have you heard about the Cognite Most Valuable Professional (MVP) Program? If not, it’s time to check it out and see how you can take your community engagement to the next level!Being an MVP is more than just a title. It’s a recognition of your skills, contributions, and leadership within the Cognite ecosystem. Whether you're answering questions, posting topics, sharing ideas, or leveling up your expertise through our comprehensive learning path offering – every step gets you closer to becoming a recognized leader.If you're already contributing, keep up the great work – you're closer to MVP than you think! And if you're just getting started, now is the perfect time to dive in. Check out the full post about our MVP program here to learn more about the program and see how you can start your MVP journey today!Let’s motivate each other, engage more, and climb the leaderboard together. Who will be our next MVP? 🌟

Hello All! As this is my first time writing, I will do a quick introduction. My name is Patrick Mishima, and I am one of the Cognite Product Specialists. My main specialization is data integration and governance. I've worked with data since 2011 with different roles, projects, and job titles all related to data management. I also have experience with various ETL and BI tools, mainly from Microsoft and SAP and a few others.I lead the Product Specialists team, and we'll soon have more articles to share our knowledge and experience with you.Today and in my following articles, I will focus on the different options that Microsoft Azure and Cognite offer on data integration between our platforms and tools. But first, let's talk about a few essential and fundamental points when working with cloud providers. Using a cloud provider, most of the time, we think about saving on infrastructure costs. That is indeed true, but it's not the only truth. When you decide to move to a cloud provider, you

Hello,I am trying to understand the difference between Sequences and Events to see what is pertinent for me in a specific use case. For me, both of them allow saving a native CDF object along with a timestamp, enriched with other data. I feel like we can also achieve the same goal by making a data model with timestamps in one column and all the properties we are interested in, in other columns;  though this approach would be harder because it would necessitate using pygen for input output. In my company, we work on Industrial processing lines. At any given moment, a line is operating under a set of conditions, defined by a fixed set of set points, like ‘conveyor speed’ and various temperatures. Let us call the set points a, b, c etc. For set points a, b and c, we have measured values i, j and k.To assess whether the machine is operating normally, we want to compare the aggregated measurements (i-j-k) taken during a period when the set points are a-b-c with a previous instance when the

In some AI demos built on top of Cognite, our approach to real-time inference involves retrieving the latest time series values for the equipment. Since our machine learning models are deployed and running on a cluster outside Cognite, we leverage the SDK to handle real-time inference by retrieving the most recent data as follows:real_time_data_bomb_hfx = cdf_client.time_series.retrieve_latest(id="bomt_hfx_time_series_id")Is anyone working with a different approach to send new data for inference to their machine learning models? I'd be interested in hearing how others are managing this process.We are also exploring more resilient approaches, such as: Streaming Data for Real-Time Inference (Event-Driven Approach): We plan to test Cognite's Kafka extractor as soon as possible to enable more seamless streaming. On-Demand Inference via API: While this approach is synchronous, which has caused challenges for us in the past, we prefer to avoid this method and lean toward more asynchronous

I am making comparisons between time series data in CDF and PI. The reason is that in our tenants the CDF data is not 100% accurate compared to PI.However, from my testing I think that PI performs its aggregations with the timestamps “centered” at the aggregated time periods, while CDF puts the timestamps at the start of each aggregated period. Is it possible to specify how this is done with the Python API? From my study of the docs it appears not to be the case. The same applies to the PI Web API as well: I cannot specify how the timestamps are placed. The agreement with PI becomes significantly better if I place the CDF timestamps at the center of the aggregated time periods.My current workaround is the following:Fetch RAW data from CDF Shift the timestamps by 0.5x of the granularity Resample to the desired granularity Compute mean Interplate any missing valuesThe issue is that fetching raw data is a lot more time consuming than fetching aggregates. I have been playing with fetching

Hi all.We’re working on defining KPIs to track our digitalization journey, and think that a KPI on the use and adoption of CDF as opposed to “the old way of doing things” would be useful. I’m posting here in an attempt to hear if the CDF customer community can provide inspiration for how to ensure success and good adoption of CDF among users. Does anyone have thoughts to share on their own metrics for CDF usage?My thoughts so far are tracking the number of unique users logging in to Fusion/Charts, number of API calls against CDF, etc. Although I have not found any direct way as a customer to obtain this information. The Sessions API endpoint could possibly be a place to start?And as a follow-up question; CDF usage in itself is a quite indirect metric on the digital maturity of the business, and on acheived value. My ideal goal would be to find a more direct higher level metric for the entire company, which is not as indirect as user log-ins, and also not as narrow as value obtained for

Wondering how to create a faster and more reliable root cause analysis? Check our How-to post:  

We're excited to announce the first bootcamp following a major update! This version now includes the powerful CDF Toolkit. About the CDF BootcampThe Cognite Data Fusion Bootcamp is an instructor-led, 4-5 days course that guides participants through a hands-on example to develop production-ready solutions with Cognite Data Fusion (CDF). While it is possible to grant access and ingest, transform, and contextualize data manually in CDF through the User Interface, creating a proper CI/CD pipeline that allows the development team to promote the project through different environments (e.g. test & prod) is essential in creating a production-ready solution. With the introduction of the CDF Toolkit, it is now possible to replace different types of CLIs with one command-line tool to configure and administer Cognite Data Fusion (CDF) projects. The Toolkit also includes pre-built templates to help you quickly start your CDF project.The Bootcamp has been developed by CDF and DataOps experts and

Hello Developers!In June, we launched the new Data Workflows service in Cognite Data Fusion. This service enables you to orchestrate Transformations, Functions, and more, bringing powerful capabilities to your data management processes.To help you get started, we've made some new content available:Cognite Learn: Check out our introductory video that covers the core concepts of data workflows. This is the first in a series of videos and guides we’ll be rolling out in the coming months. Jupyter Notebook in Fusion: Explore an example notebook in your CDF project, demonstrating how to leverage the features of the data workflows service. You can find it by navigating to Jupyter Notebook in the Data Management workspace in Fusion, and then go to the shared folder /Examples/data-workflows.   We encourage you to try out Data Workflows and share your feedback with us either here or in the dedicated Hub group. We're continuously working to enhance the service with new features, so stay tuned for

We have a DM with many direct relations. A processing is directly liked to an asset with the name of the property being ‘equipment’.  list shows the field as either just text or a dictionaryWe use the Cognite data modeling API to generate models and use pygen to run programs on it. I have noticed something strange while using pygen. Properties which are direct relations to other types sometimes are shown as text and other times are shown as NodeId objects.On close inspection, they appear as NodeId objectsIn the former case, while observing them in the Data Model view, the properties appear empty. Actually, they contain the external_id of the instance as a string which can be seen while querying with pygen. DM view showing seemingly empty cells Sometimes, instances spontaneously transition from one type to the ohter (NodeId → string or vice versa). In my pygen programs thus, I have to use the following function to keep the program running. from cognite.client.data_classes.data_modeling

In Cognite Data Fusion data models, instances (nodes and edges) are uniquely identified by their space and external ID.To simplify the user experience, Pygen includes a parameter called default_instance_space. You can set this parameter when generating a new SDK, allowing you to work with nodes and edges without specifying their space, as long as all nodes and edges share the same instance space.In earlier versions, if you didn’t specify the default_instance_space, Pygen would default to using the same space as the data model. Now, Pygen generates an SDK that requires users to specify the space when creating, retrieving, and deleting nodes and edges.The reason for this change is that having the schema (data model, views, containers) and data (nodes and edges) in the same space is considered an anti-pattern. Governance of a data model should be in a separate space, while data ingestion and consumption should occur in different spaces, typically one space per data source.

Hello Everyone! Exciting news! We're hosting Impact 2024, Cognite’s first User Conference, and we want YOUR use cases to shine. You can read more about Impact 2024 here. Why Share your Use Case?Your stories inspire. Share how your projects have driven real business impact, from boosting revenue to streamlining processes. We also welcome discussions on challenges you've faced and unresolved issues. This can spark meaningful conversations and potential solutions during the event.How to Get Involved:Submit your use case in this thread including it's impact. Based on the submissions, we will organize workshops to explore these use cases further. If your use case is selected, you will receive a free ticket to Impact 2024. We are committed to fostering a collaborative environment while respecting confidentiality and competition concerns, ensuring we all become better together. Mark Your Calendars:Submissions Open: April 25, 2024Submission Deadline: June 30, 2024Winners Announced: August 1, 2

pygen version v0.99.30has just been released with a new approach to creating queries. This is experimental and we are looking for feedback. Check the documentation for more information.

Hello Everyone 😊, I'm now working on an endeavour that will integrate our organization's multiple old systems with Cognite Data Fusion (CDF). Although there are numerous advantages to CDF's current architecture, there are certain obstacles we must overcome in order to guarantee smooth data transfer between CDF & our more antiquated systems—especially those that weren't created with contemporary data platform in mind.I would appreciate your thoughts on the following specific queries and worries:Data Connectivity: How can CDF and older technologies that use antiquated communication formats or protocols create and sustain reliable connections with one other?🤔 Have anyone of you run across problems with particular kinds of systems?🤔 If so, tell us about it.Data Consistency and Quality: When importing data into CDF from older systems, how can you maintain and guarantee data consistency?🤔 Exist any particular Cognite ecosystem tools or procedures that support preserving data accuracy

Hi There,We would like to confirm if the indexing mechanism in the CDF operates in the same manner as it does in a relational database. Specifically, we need to understand the trade-offs of using indexes carefully in CDF. In relational databases, indexes occupy space on disk and memory when in use, which can be problematic if space or memory is limited. Additionally, maintaining indexes during data insertions, updates, or deletions can slow down these operations and lock tables (or parts of tables), potentially affecting query performance.Given these considerations, do we need to manage indexes in CDF with the same level of caution as in relational databases?Disadvantages of having an index in a relational database:Space: Requires additional disk/memory space. Write speed: Slows down INSERT, UPDATE, and DELETE operations. 

Pygen will now create fields for reverse direct relations in the generated data classes. In addition, with the generated `.list` method you can retrieve the nodes on the other side of the reverse direct relations by calling it with the parameter `retrieve_connections=”full”`. See the documentation for more details an examples https://cognite-pygen.readthedocs-hosted.com/en/latest/usage/listing_filtering_retrieving.html 

Hi,We use CDF workflow to process the data present in data model. Based on the inputs, we trigger multiple workflow instances to take the advantage of scalability. We observed that, the workflow execution time is increasing after we trigger more workflow instances.  Here is the simple diagram to show how we use workflow:   For example:If we run 1 workflow instance to process 20 wellbores (assume some processing logic divided between F1 to F4) with 20 concurrent tasks,  the execution time of that workflow instance is 5 mins.  Now if we want to process more wellbores say 40, we trigger 2 workflow instances , 1st workflow instance to process 20 wellbores and 2nd workflow instance to process other 20 wellbores and we expect the execution time for all workflows to be approximately same but the total time is increasing if we compare with only 1 workflow instance with 20 tasks. Can you please help me to understand if anything is missing? 

Hello, all!This is my first post after just joining this discussion, so please forgive me and provide kind assistance if I have posted to the wrong subsection! I am new here but a real enthusiast and loving this community so far. I have a background in teaching coding and in education and feel I could help with documentation, at least for starters.As a new member in this forum and wish to share and gain some knowledge. I am looking forward to create my own discussion to resolve my query and gain some knowledge though I have taken part in various discussion which is definitely helped me a lot.Also in what category should be taken depends on what factors?🤔Thank you 😊 in advance.

How can I see the default canvas option in my portal?I have seen this visible option in other users.  

A use case we encounter at Cognite is writing data back to SAP. This can happen in several contexts where the main goal is to create or update data in SAP (ex: work orders, notifications etc.) based on the analysis of data stored in Cognite Data Fusion. The ability to write back to SAP allows to take decisions without going back and forth between applications, which can save a lot of time. It is also less error prone than manually filling fields in SAP based on what you read in Cognite Data Fusion.  This use case, which is all about automating processes, definitely fits Industry 4.0. In addition to that, SAP is one of the most used ERP systems in the industry. As a side note: we are talking today about SAP, but the same would be possible with other ERPs (as long as they have an API we can send requests to). For example, in a maintenance context: when analyzing data of your industrial machinery, you might notice that one of your machines needs maintenance. Instead of going to SAP, looki