Product User Community

Hi! We’re aiming to monitor all of our CDF operations in the solution we have for monitoring other systems and data flows (Splunk). This would involve monitoring of extractors/extraction pipelines, cognite functions, and general platform availability. I wonder if any other Cognite customers have done this, and if you would be willing to share your experience / architecture for extracting logs from cognite functions etc. into Splunk or a similar service? Cheers,Kristian 

While testing the hosted MQTT extractor and the pre-release front end I have noted a couple of things you might be interested in, but I’m also guessing some of these are quite obvious to you and already in the pipeline. Front end should support creating a new destination with specified dataset. Viewing and editing a stream transformation without deleting and creating a new one would be nice. A way of listing existing transformation mappings and create or draft new ones without setting up an extractor instance for it would be helpful. Transform error-messages might not point at correct line and character in the message it returns. I don't have an example of this right now, I will post one if I see it again. Transformation Mappings entity could have a description-attribute that lets users describe the scenario it is written created for. The endpoint listed in beta docs for creating destinations did not work, I ended up using the same endpoint as the front-end where "hostedextractors" is

Happy to share with the Digitalization community some security news!Since Cognite's start, we have prioritized security and take our security role very seriously as a partner to mission-critical industries. Today, I am happy to announce that Cognite has obtained our SOC 2 Type II certification. The American Institute of Certified Public Accountants (AICPA) bases SOC 2 engagements on its Trust Service Criteria. These criteria ensure we have rigorous security guidelines and stick with them. Type II certifications, in particular, are comprehensive and involve significant work to prepare, so a huge thank you to our team and our partners for their contribution to this achievement. At Cognite, we are investing in more automated ways to support continuous compliance monitoring.  We focus on empowering our end users with industrial data operations without managing the related infrastructure. We believe it's our mission to create frictionless security, and our SOC 2 Type II certification is ano

Dear Power and Renewables community members,We're evolving! The Power and Renewables group is closing, but exciting new groups are on the horizon. We'll be launching early adopter groups for our power and renewable industry domain specific offerings.Don't worry about losing valuable content - everything shared here will be available in our community knowledge base. Stay tuned for invites to our early adopter groups, and in the meantime, we hope you'll continue the discussions with everyone in our broader community. Thank you for being a part of the Cognite Hub Community. Your expertise is essential, and we're eager to continue the journey with you in our community spaces.Best,Ingar Beck Landet, VP Power & Renewables in Cognite

Hi everyone,We are working on a new project related to video analytics and would like to know if there are any features available at Cognite that we can count on as possible components of the solution, such as Cognite Cluster for running object detection/classification, object tracking, partial video events, and overlaying video output.

Hi all!This is an update from the Data Modeling team on an upcoming change.We are making a change in how we treat the absence of data in an object when making a query in a data model graph. Currently, if an instance has no value in an optional property, it will not be included in the response to a query for instances where that property is NOT some value.As an example, consider a field “completionYear” which is not always populated - a query for instances where completionYear is NOT “2023” would return only instances where completionYear has a value, and that value is not 2023. Instances where that property is not set would not be included.We intend to change this behavior such that instances without a value in that property would be included in the query reseponse, as this behavior is more in line with the expected behavior in most cases. This also resolves an inconsistency with the search and aggregation queries where this behavior is already happening.We consider this a bug fix, so

Hi! We have made the following updates available for Workflows.Optional TasksDefining non-critical tasks in a workflow as optional, meaning that in the event of a task failure, the workflow itself should continue, has been a much requested feature. You can now set the policy for how to handle failures and timeouts for each task in a workflow. By default, the workflow will fail if a task fails (after retries) or if the task times out. Alternatively, you can set the parameter to skip the task in the event of a failure or timeout (after retries). This means that the workflow execution will continue even when the task does not complete successfully. The feature is documented in the API specififcation here (navigate items > workflowDefinition > tasks > onFailure). Concurrency Policies for TransformationsIf a job is already running for a given Transformation in CDF, a new job cannot be started concurrently (fails with error message “A job already runs for this transform”). To better

Hi Community Members!We’ve recently published a cloud-based Virtual Flow Metering (VFM) system tailored for a late-life well with high water cuts. This solution employs a hybrid physics-data approach, delivering precise and near-real-time estimates of water production per well in gas fields.Link to the full article: https://www.sciencedirect.com/science/article/pii/S277250812300042X Enhancing Production OptimizationIn the oil and gas production domain, understanding the fluid composition and production rates of multiphase operations is crucial for effective planning and optimization. With wells yielding a diverse mix of oil, gas, and water, often with varying compositions, ensuring accurate measurements is essential for maximizing production efficiency. Ratios change in time, and this gets more and more important as the water cut grows in the late part of a water flooded well life. This article explores how Virtual Flow Meters (VFMs) offer a cost-effective alternative to traditional me

Hello everyone,I'm relatively new to using Grafana, and after reading this post (Cognite Hub) on Cognite Hub, I've come to understand that Alerts in Grafana may not be compatible with the CDF connector. Please feel free to correct me if I'm mistaken.Considering this, I'm wondering if anyone has a good setup for monitoring the data coming from the CDF connector. I have around 900 time series, and listing all of them is not very efficient. Additionally, I'm having trouble figuring out how to write an SQL query to count the time series that have a timestamp older than, let's say, 3 hours. this could be a way, to count data not coming in. Would anyone be willing to share their setup or offer any tips?Thank you in advanced :) 

In September, we embarked on an incredible journey with our partners at CNTXT Academy, bringing Cognite Data Fusion (CDF) training to the heart of Saudi Arabia. We were thrilled to host not one, but two engaging Cognite Data Fusion Fundamentals sessions right at the epicenter of innovation - Saudi Aramco's headquarters in Dhahran. Our participants came from diverse backgrounds, each with their own unique experiences with Cognite Data Fusion. So, we tailored the training to ensure it resonated with everyone, addressing their everyday work challenges that Cognite Data Fusion can brilliantly solve.The first part of the training focused on contextualization, mastering Cognite Charts, and seamlessly connecting CDF with Grafana to tackle a real-world use case. The participants got hands-on experience, putting their newfound knowledge to the test right in CDF. In the second part, our exceptional instructors introduced participants to the Cognite Python SDK, taking their skills to the next lev

Hi! We have made the following updates available for Workflows.Support for Workflows in the Cognite Python SDKThe Cognite Python SDK now has native support for Workflows. Documentation can be found here. We have also updated the example notebook, which you can find attached as a file below (Note: Download the file and remove the .txt postfix to open it as a Jupyter Notebook.)New Access Management CapabilitiesYou can now configure access to Workflows directly in Access Management in CDF. Navigate to “Manage” > “Manage access” and for the relevant Groups add capabilities to read/write Workflows as needed (see screenshot below). This replaces the experimental ACLs which were required initially to interact with the service. The capabilities can also be added to a group using the SDK directly, e.g.:from cognite.client.data_classes import Groupcapabilities = [{"workflowOrchestrationAcl": {"actions": ["READ", "WRITE"], "scope": {"all": { }}}}]group = Group(name="Workflow Orchestration", ca

We are happy to share our newest enhancement to the search experience within Cognite Data Fusion, furthering our ambitions to deliver easy access to complex industrial data. This feature provides user-friendly search and enhanced context for industrial users, such as engineers, technical experts, and operational teams. You can narrow your search to a specific site - like an individual offshore asset or a processing facility - to make the search results more relevant The search results are grouped and presented in an industrial context and can be viewed through a list view and a 3D view with a wide set of filter capabilities across all given data types You can open data in Cognite Charts and Cognite Industry Canvas You can easily change between layered models to see CAD models, 360 images, and point cloud modelsWe are very happy to get feedback, and would love to hear how you experience this new feature. Best, Sofie, Product Manager 

Is there an example available for setting up the metric feature using Prometheus for the OPC-UA extractor, similar to what is shown for the PI extractor in the Cognite Academy? 

Dear CDF UsersWe are thrilled to announce the open beta release of the monitoring and alerting functionality in Charts within CDF. This release marks a significant leap forward in providing you with a comprehensive monitoring and alerting solution, designed to empower you in managing and optimizing your data flows. Key Features of  Release:Native Thresholds on CDF Data: Now, within Charts, you can seamlessly create thresholds on time series data. This native capability allows you to monitor or both your time series data and KPI’s generated from no code calculations, ensuring that you are alerted precisely when needed. Save your no-code calculations in CDF: Now within Charts you have the ability to save your no code calculations within CDF so that they are consumable in other context for example your own dashboards, monitoring etc. This write back is now scoped to a dataset to ensure that the integrity of the data is maintained. Email Notifications: Stay informed in real-time! Our monit

Open Beta Release of Monitoring and Alerting in Charts!Dear CDF User,We are thrilled to announce the open beta release of the monitoring and alerting functionality in Charts within CDF. This release marks a significant leap forward in providing you with a comprehensive monitoring and alerting solution, designed to empower you in managing and optimizing your data flows.Key Features of the Open Beta Release:Native Thresholds on CDF Data: Now, within Charts, you can seamlessly create thresholds on time series data. This native capability allows you to monitor or both your time series data and KPI’s generated from no code calculations, ensuring that you are alerted precisely when needed. Save your no-code calculations in CDF: Now within Charts you have the ability to save your no code calculations within CDF so that they are consumable in other context for example your own dashboards, monitoring etc. This write back is now scoped to a dataset to ensure that the integrity of the data is mai

Hi community!Monitoring asset health, performing root cause analysis, and doing on-the-fly troubleshooting can be a complex process for both seasoned experts and new engineers. In this video I'll be sharing some tips on how to gather and analyze data efficiently, which we believe will boost your confidence in managing asset health.I'd love to hear your thoughts and encourage you to share your experiences and insights to help improve our asset health program.Cheers,Andreea.  

Have you already tried and used the latest features released in our Industrial Canvas? We would like to hear your feedback! 🙌Here is the full list of features and how you can use them:Managing your canvasCanvas visibility can now be set to either private or public. By default, when you create a canvas, it will be “marked” as private. You can change it to public by using the Share option while in the canvas. View live datapoints signalled with traffic lights based on basic rulesPin time series connected to assets in engineering diagrams and visualize their “live” values (the latest value is refreshed every 30s). Add simple rules on these values. For example “if the temperature is higher than 30°C, show the tag in colour red”.Leave comments and tag colleagues in the canvas You can easily add comments while working in the canvas and tag your colleagues. You can find all the comments in the “Comments panel” where you can also search for the content.  General fixes, tweaks and small improv

We are currently exploring potential use cases with our team, and one of them involves implementing data quality techniques. I was wondering if anyone could share their experience in order to identify the best approach within the Cognite Platform for conducting data quality checks to ensure that all fields in critical tables are populated correctly.In the past, we have implemented a similar solution with a different vendor, utilizing a system that can prevent the flow of erroneous data through our data pipelines by employing validation and integrity checks based on declarative quality expectations.Thank you in advance.

We are pleased to announce an update to the Search, Filtering and Aggregations documentation, detailing enhanced search, filtering and aggregation capabilities for our Assets, Documents, Events, Sequences and Time Series APIs.These additional features are already implemented in the April 2023 update to Data Explorer and are available to use natively within the service APIs. The documentation provides guidance on how to implement advanced filtering, sorting, search and aggregation functionality (which is available on both the standard fields and metadata), so that application users can use much more powerful and expressive search queries to find exactly the data they are looking for.Use is recommended for existing, mature projects, and apply to use cases for the Assets, Documents, Events, Sequences and Time Series APIs.  For new projects and use cases, customers are recommended to work with the newer Data Modelling APIs.The updated documentation may be found here Tell us what you think

Update! See this post for an updated example notebook based on the Cognite Python SDK.We want to make it easy for you to interact with CDF Workflows through Python. Therefore we’ve created a Jupyter Notebook that demonstrates the service using Python and a simple example. Native support in the Cognite Python SDK will be released shortly. Try it out, and feel free to leave feedback or questions in the comments! Note: Download the file below and remove the .txt postfix to open the file as a Jupyter Notebook.

Hey there! We are thrilled to announce the Early Adopter Beta release of CDF Workflows.CDF Workflows is a managed process orchestration service within CDF. Using CDF Workflows, you can effectively coordinate the order and timely execution of interdependent processes such as Transformations, Functions, requests to CDF, and dynamic tasks effectively. Key Highlights Declarative Workflow Definition We have designed Workflow Orchestration with a declarative approach. Users need to specify the tasks to be orchestrated, input parameters, and the desired execution order. This intuitive approach we believe reduces complexity and lowers the bar for adaption. Orchestrating four different processes The Workflow Orchestration service supports the orchestration of four crucial types of processes: Transformations, Functions, CDF, and Dynamic Tasks. This versatility ensures that our workflows can leverage various functionalities, optimizing productivity. Versioning Capabilities Managing changes an

Found a bug? Have a question about how something works? We want to hear about it!As an early adopter if you wish to report a bug simply post a reply below in this thread or create a question / thread in this group and someone from the team will get back to you regarding the issue.  

In this post, I want to mention a tool that can help us troubleshoot issues such as authentication or unexpected errors: FiddlerFiddler is a Windows-only tool that logs HTTP(s) traffic. This will help you to understand what you are sending on the payload in the request and what response you are getting. This is a community-trusted free tool.Some applications don't show descriptive error logs in the UI, which makes it challenging to narrow down the issue; however, you can get a better understanding of what is going on if you have visibility on the API calls.When using a Web app in the browser, you can use tools such as the Network Tab in the Developer tools to capture the traffic and inspect the API calls. You can get more details on how to do this in this post:But what do you do when you get errors in applications (.exe) such as Power BI, for example? For scenarios like this, you can use Fiddler.How to use FiddlerDownload and install Fiddler-Classic  Open Fiddler You need to decrypt tr

Hello Cognite Hub Community!We’re working on a new webinar series where we focus on concrete use case and solve it from start to finish with Cognite Data Fusion. We’d like to hear from you – which specific industrial use cases would you like us to solve? Please give us as much detail and context as you can.Do you have a use case solution you’d be interested in co-presenting with us? Fantastic. Please reach out to me directly on Cognite Hub or by email (eric.stein@cognite.com) and I’ll be in touch.Thanks everyone and looking forward to hearing from you!

Hello Cognite Hub Community!We’re working on a new webinar series where we focus on concrete use case and solve it from start to finish with Cognite Data Fusion. We’d like to hear from you – which specific industrial use cases would you like us to solve? Please give us as much detail and context as you can.Do you have a use case solution you’d be interested in co-presenting with us? Fantastic. Please reach out to me directly on Cognite Hub or by email (eric.stein@cognite.com) and I’ll be in touch.Thanks everyone and looking forward to hearing from you!