Community Discussions

Hi, I am trying to transform data from Raw to timeseries in CDF. I made a query and everything seems ok, but when I use distinct to check the external id uniqueness, I get error message:"Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column metadata is map<string,string>; I don’t know why I get this message, Isn’t it possible to use distinct when wi do MAP or To_Metadata mapping? 

I am the Head of the Services delivery team in Cognite. The team I lead develops and delivers domain use cases based in CDF to our customers in O&G, manufacturing and renewables. I am going to focus here on what are the consequences of corrosion of industrial assets on the economy and the environment, as recently covered in a series of publications (see references section). The economic and environmental cost of corrosion is a significant issue and is much more severe than one could think. While several calculations methods have been used in various studies it is generally accepted that the direct costs of corrosion to the economy are equivalent to roughly 3–4% of a country’s gross domestic product (GDP) [1,2]. The main cost impact is due to the replacement of corroded steel in failing infrastructures (roads, railways, etc.) industrial equipment and buildings. As well, it has been qualitatively estimated that between 25 and 33% of the annual steel production is destroyed once in se

As communicated in this post in December, Cognite Charts is now integrated as a part of fusion.cognite.com. We are planning to retire the stand-alone url charts.cogniteapp.com to have a uniform product experience and single login. Starting February 7th 2023 you will find a message if you access the stand-alone url with instructions on how to login to fusion.cognite.com. You fill find all your charts and data intact, and Charts working as you are used to. There will be a grace period for 2 weeks after February 7th giving room for users to still access their charts from the stand-alone url if you need support to login to fusion.cognite.com.  For those of you who are linking to Charts in your third party applications, Microsoft Teams, etc., this will still function as expected. We will be working on an automatic redirect for coming releases, however still recommend you update the hyperlinks to point to your-organization-name.fusion.cognite.com already now. If you have questions on this, d

When working with CDF or Azure, sometimes I got some errors that were hard to figure out. Experience no doubt will help me figure them out faster, nonetheless this is an important part in enabling usability.

Hey community we are trying to be more active in this cognite hub page and add content consistently from a month to month basis. This will include a series of articles on alerting and monitoring as well as plans regarding where AIR is going into the future.Why Alerting?Automated alerting is an essential part of monitoring. They allow you to spot issues with equipment groups, time series, data quality, pipelines etc.But alerts aren’t always as effective as they could be. In particular, real problems are often lost in a sea of noisy alarms. In short:Alert liberally meaning its ok to spam users rather than not alert at all Make sure that the user has the complete control of how and when they want to be alertedInherent challenges in AlertingSensitivity: Overly sensitive systems cause excessive false positive alerts, while less sensitive systems can miss issues and have false negatives. Determining the correct alerting threshold requires ongoing tuning and refinement. Fatigue: The common ap

 I faced some isuees regarding connection with Azure when working with CDF. I believe there is an area of opportunity there! 

@Savieth Aceros and @Gina Esperanza Jimenez Morales - hope you find the learning paths helpful! Are you taking part in a bootcamp as well?

ContextThis write up describes a basic set up of writing data points to Cognite Data Fusion using Apache NiFi. The source data is power consumption readings from the HAN interface of a power meter in a residential fuse box:The starting point here is an existing MQTT broker that receives data from an existing MQTT client device. We will use Apache NiFi to consume the MQTT messages, extract the readings and write continuously to CDF.The end goal is a live updated time series within CDF with power readings every 2.5 seconds that can be used for analysis or automation within CDF or simply visualization: NiFi flow overviewThe picture above shows the complete NiFi flow. From left to right, consuming MQTT messages, transforming and writing to Cognite Data Fusion. Approximately 120 data points per five minutes, which corresponds to the power meter outputting one reading every 2.5 second. The NiFi flow makes use of the following Processors: Processor  Purpose ConsumeMQTT

As 2022 comes to a close, we thought we’d take a moment to reflect on all the improvements that have been put into InField this year. From small bug fixes to big feature releases, from requests from Hub to user research sessions in person, many of you have contributed to make InField the best it can be for all its users. Throughout the year, InField has been updated with a total of 30 updates packed with user-centered functionality and improvements – and at the same time, the InField team is rebuilding the foundational backend, which sets us up for 2023 and beyond.  Users can expect to see major improvements to performance, much improved 3D- and file-viewing, better scheduling of checklists and more capable templates. While we’re working on that, here are some of the improvement and additions the team has made in 2022: January: The Weekly Summary page was introduced We updated the progress indicator to a new status bar, and made it available on mobile Much improved formatting on de

Below we have outlined several frequently asked questions and their corresponding answers.Don’t see the answer to your question? Post as a reply in this thread and we’ll be sure to answer and/or add it to the FAQ list below!Protip: Use [cmd+f] or [cntrl+f] to search for keywords related to your question.Be sure to also let us know what else we can help answer here in the comments below! Q: What is the plans with Templates going forward?Please check out the documentation hereor https://pr-1023.docs.preview.cogniteapp.com/dev/guides/data-modeling/get-started#templates-future Q: Will there be new features built on top of Templates?Yes and no. We will not be doing any development of backend features (like new capabilities) into Templates. However, we will actually be building a lot of additional tooling on top of Templates (better UI, visualization layers, code generation, CLI etc.). Stay tuned in this Cognite Hub to see how you can use these tooling. 

Cognite Team - thank you for bringing back the ability to overlay events directly on the chart as was previously a capability in ADI. It’s a welcome return and valuable add to the application.After utilizing the app a bit, I have a couple of “low hanging fruit” recommendations:Can we add the event description to the metadata that’s shown on the event? Is there a way to toggle using both start time and end time vs. just using start time? Some of the events that we have run for months and the start time is really the only relevant point for the trendHere are some more complex recommendations:Since charts is now integrated directly in Data Explorer - is there a way to click on the event results and actually go to the event directly in CDF to look at all the metadata? When filtering for events by selecting the asset - is there a way to toggle “select all children” as well? An example is that we have several pumps where work is tagged to the smallest component, but if we filter the events t

Today is the December release of Cognite Data Fusion. As you might have seen in last week’s Product Tour or read in today’s release post, there are several new and exciting Charts-specific updates in the latest release, namely...Charts is now an integral part of Cognite Data Fusion (fusion.cognite.com).A long awaited improvement, charting and no-code analytics finally lives alongside all other industrial DataOps functionalities available in Cognite Data Fusion. No URL switching and additional sign-in required.For those of you who have been using Charts before, all private and public charts that you and your colleagues have created will still be available via Cognite Data Fusion (fusion.cognite.com).Note that charts.cogniteapp.com will still work and be available. This teams, for those of you who are linking to Charts in your third party applications, Microsoft Teams, etc., this will still function exactly as expected. We will be working on an automatic redirect for coming releases, how

Hi, there! At Cognite, our customers are at the center of everything we do. Our goal is to help you to solve the industrial data problems daily. It's all about you & your industrial data.  To make sure we're giving you the best experience possible with the Flexible Data Model (FDM), we'd like to learn about your FDM expectations. Please spend a few minutes to share your thoughts with us in this short survey. The information you give will be confident, so be honest. We really want to know what you think. The survey 👉  https://forms.gle/d8BzpW4jmG4YN9kSA Thanks for your help! Cheers,On behalf of the FDM team   

Hello, I found very interesting that a lot of the conversation around digital has moved away from “AI will change the industry” to much more elaborate analysis on the importance of data in context and data orchestration/DataOps. This realization is making its way all the way to the board room.A good example comes from the recent conversation between Blake Moret (CEO of Rockwell Automation) and Olivier Le Peuch (CEO of SLB). Both companies have independently decided to partner with Cognite and you can find some of the rationale in their exchange below.Two additional notes from the discussion:There is a vision to move from an equipment centric digitalization to a full asset/system view of what digital should encompass - with automation/closing the loop as a key actuator (see Sensia’s control systems’ role),  The discussion around optimization has stopped being solely about production volumes but now also systematically includes carbon. Source: Partial transcript from Rockwell Automation'

We have been discussing with one of our customers the need for a course, similar to the excellent Cognite Data Fusion Fundamentals course, but with less of a technology slant. It would be great to hear your thoughts on this and any topics you think this should cover.

Data operations (DataOps) is essential to providing consumers with business-ready, trusted, high-quality data. But when faced with a somewhat different data source, data type, data quality, and data consumer landscape, what are the defining factors that will significantly catalyze DataOps adoption in industrial companies? Join our upcoming webinar with Forrester, and be a part of the discussion, as we unpack the technology behind the DataOps practice. Dive into the topic of data operations for industry and hear leading analysts’ take on:-What is DataOps today? How has it evolved over the years?-How does DataOps adoption affect industrial companies?-What are the challenges when adopting DataOps?-Where do DataOps and MLOps come together? Register here

Sesam.io is a master data hub that simplifies the process of making up-to-date master data available in a data platform architecture.Cognite CDF, well, you all know what is it, right?And we are going to send some data from Sesam to CDF without creating and deploying any connectors, only by using out-of-box functionality provided by Sesam. What we need:Cognite CDF project configured with an Oauth2 identity provider, such as Azure Active directory  Credentials such as client id, secret, oauth token url and scope with access capabilities that allow us to write to Raw tables Provisioned Sesam node with some data.First login into our Sesam instance and create an endpoint system we will use to send data through (I assume you already have some data you want to send to CDF) by choosing “Systems” in left-side menu and then “New system” on the top bar . We will use REST API and push data by making HTTP requests towards CDF. Sesam provides built in REST connector that supports basic and Oauth2 au

 TL;DRIf Jetfire CLI self contained packages or Jetfire docker image is used in your deployment pipeline. Please refer to the documentation below on how to make the necessary changes to migrate to new Transformations CLI. If not, then no action is required from your end. We have handled it for you! :)) As our services mature, we sometimes need to deprecate the tools and services we made earlier. Last December, we released Transformations API with that came great documentation, support in the Python SDK, and a brand new Python-based Transformations CLI for use in CI CD pipelines.The new Transformations CLI replaces the (old) Jetfire-cli, which will no longer be maintained and will eventually be deprecated at the end of December 2022. The new Transformation CLI supports everything the old Jetfire CLI did and more. It is easier to install, and you can still use your existing Transformations scripts with the new CLI with the legacy mode enabled (Refer migration guide below 👇 ).Like with a

In this conversation I will explain / help how to access the data from the Open Industrial Data Project (OID) CDF Project, through the Cognite Python SDK, with authentication through an interactive login token from Cognite Hub’s Azure Active Directory. The video in the article will step by step show you how you can get access to the data, and also link to the relevant information you will need. Steps:1. Install or update the following python modules/librariespip install cognite-sdkpip install msalpip install pandas2. Download the publicdata.py file and store it in your python environments working directory. this file is prefilled with the OIDC ID’s and CDF project values needed to access the CDF project.3. Open a Jupyter Notebook or python file and import the publicdata.pyfrom publicdata import c4. Access the OID CDF project through the CogniteClient object: c, and when the first method is called on the CogniteClient object it will authenticate and redirect towards the AAD tenant and u

We proudly announce that Flexible Data Modeling has reached public beta. Flexible data modelling in CDF enables you to model industrial data the way you understand it in a self-documented way so that domain experts can find, recognize and understand the data without a third-party manual.pdf. Your data is stored in one single knowledge graph, and empowers users to search, filter and aggregate data according to their needs through various data models on top of the knowledge graph. Flexible data modeling enables rapid scaling, whether it is building an app, data science model, dashboard, or other use cases, we hope that this capability can be a great toolkit in helping you to express data in the most intuitive way.In the Beta version you can Create and manage data models in Fusion and using a cli tool Ingest data into the data model using CDF transformations Query data using GraphQL with powerful search and filtering in Fusion or your own applications with even more powerful features

OpenID Connect has been enabled on Open Industrial Data for three months.We will be removing the option to use API keys for authentication on November 1st, 2022.If you are still using API keys for authentication, please change the Authenticatioon to openID Connect. Learn how to do that in this article.

Developing, tracking and meeting sustainability goals is becoming increasingly important for companies in the public sphere. Many of them are turning to data-driven solutions to help them monitor, report and reduce their environmental impact. Recent regulations in Europe recognises that Data-driven solutions for GHG emissions reductions are contributing to climate change mitigation (see official site from the European Commission - link) At Cognite, we have encountered many innovative solutions that promote sustainability. Examples include:  Automating the recording and reporting of greenhouse gas emissions in industrial processes. Optimizing energy efficiency at the equipment and system level to minimize scope 2 emissions. Using robotics to detect dangerous leaks. Leveraging operational data to derive the environmental footprint of discrete products. Protecting biodiversity with the automatic detection of birds close to onshore windmills  After analysing dozens of those solutio

Hello Charts Early Adopter Community,I know it’s been a bit quiet in this group lately, but rest assured, the team and I have been hard at work. We have just released several new features and fixes to production. You can scroll down to read about each of them in detail.Please do leave comments below with questions and feedback!What’s coming next?We’re in the middle of our development cycle in preparation for the next major release of Cognite Data Fusion in December 2022. There are the two major features our team is working on that you can look forward to having available:Moving Charts (charts.cogniteapp.com) into Cognite Data Fusion (fusion.cognite.com) We’ve received plenty of feedback that our product experience can and need to be more tightly integrated and we’ve heard you loud and clear. As you all know, Charts is currently available on charts.cogniteapp.com, which is separate from Cognite Data Fusion (fusion.cognite.com). Despite starting out on its standalone URL, Charts has alw

Do you know what courses are available for you to explore the data from the Open Industrial Data project?Cognite Academy creates e-learning courses with an emphasis on providing hands-on experiences so you can learn to use Cognite Data Fusion through interacting with real industrial data. In most of our courses, the training data we use comes from the Open Industrial Data (OID) project, a live stream of industrial data from the Valhall oil platform. While the data is from the oil and gas industry, it is relevant for all asset-heavy industries as it provides insights into dynamic industrial processes. In this article, I will share a series of CDF & Power BI courses where you can explore the Open Industrial Data and solve a simple use case in Power BI: Introduction to CDF & Power BI  In this course, you will connect Power BI to CDF, and retrieve the Open Industrial Data to learn how filtering and aggregation work with the connector in Power BI.  CDF & Power BI: Solving

Are you a Power BI user who wants to gain hands-on experience with Cognite Data Fusion? Are you curious to learn how to handle a large set of CDF data in Power BI?The Cognite Power BI connector lets you use a CDF project as a data source in Power BI Desktop to query, transform and visualize data, share insights across your organization, or embed dashboards in your app or website. When working with the Cognite Power BI connector, it is important to know how to refresh the data and apply best practices to deliver accurate results. The newest course from Cognite Academy introduces you to implementing incremental refresh - one of the best practices to get the most out of the Cognite Power BI connector. Our instructors first explain when to use incremental refresh, then show you how to set it up in four steps. In this course, the training data comes from the Open Industrial Data (OID) project, a live stream of industrial data from the Valhall oil platform.After completing this course, you’l