Cognite Data Fusion Release: June 2022

​This June release has several improvements for the end-user and the developer.  View more details in complex CAD models​To see more details and better understand the facility, we've improved the visual quality when you view large and complex 3D models.Also, we have improved the 3D processing pipeline and the load performance so that you can load geometry while the camera is moving. Note! To take full advantage of the improvements, you need to re-upload any 3D models processed before March 2022.A better user experience signing in to CDF​Signing in to the CDF user interface can be cumbersome if you don't know the authentication setup in your organization. With the new CDF sign-in flow, you only need to enter your organization's domain name and specify which CDF project you are signing in to. Your organization domain name is usually your company name spelled without spaces or an abbreviation of the name.Add your customer logo to the sign-in dialogs for a more personalized experience.Authenticate your CDF solutions​To allow the developer to focus on solution development instead of setting up authentication infrastructure, it's now possible to authenticate for solutions built on top of CDF without writing multiple lines of code. The @cognite/auth-wrapper is an OpenID Connect/OAuth 2.0 wrapper library written in JavaScript. It helps you retrieve an access token from any IdP that meets the OpenID standard, making it easy to sign in with our JavaScript SDK. Features in the authentication wrapper SDK include:NPM supported auth package Support for multiple auth flows Detailed documentation on how to use the package Detailed documentation on how to navigate key IdPs (i.e., Azure AD)Search, filter, and sort data in the CDF Data explorer​We've polished up the CDF Data explorer to make it easier to find the data you are looking for:You can reuse any Search and Filters you have defined. Simply copy the URL and share it with your co-workers. The search criteria you enter in the Search bar are now kept when you toggle between the different tabs in the explorer. On the Files > File details tab, you can scroll up and down the list of metadata to find what you're looking for. We've added a notification to let you know that you cannot upload files greater than 5GB. You can now sort in the metadata tables. We've fixed reported bugs and general styling issues.Increased number of columns for sequences​​Sequences now support up to 400 numeric columns (up from 200) and 200 string columns. Read more.DirectQuery support for Power BI (Early adopter)​​DirectQuery is a feature in Power BI that enables you to fetch data on demand and build Power BI dashboards with hundreds of time series and thousands of data points. With DirectQuery, no data is imported or copied into Power BI Desktop. Instead, Power BI queries CDF for the data every time you interact with the dashboard.Cognite Data Source for Grafana​The Cognite Data Source for Grafana now supports the CDF relationships resource type. You can fetch time series based on relationships in the Time series from asset tab and filter the relationships on data set, labels, and active relationships based on the time filter selected in Grafana. The new Relationships tab allows you to use Grafana's Node Graph visualization plugin or the new (alpha) custom visualization plugin available at cognite-grafana-relationships-visualization.Templates support (Early adopter)​We've added a new Templates tab to support the CDF Templates feature. Using templates with Grafana allows you to scale your dashboards dynamically when you add equipment. You can enable this feature in the data source settings.Read more about Templates and Early adopter.

Related products:Product Releases

Power Market Operations [Beta]

Context and introductionAs liquidity increases in short term markets and market mechanisms evolve towards net zero targets, the sheer volume and complexity of data in relation to power trading activity has outpaced the capacity for prudent, optimal decision-making.With Cognite's product suite consisting of Cognite Data Fusion and Power Market Extensions, along with expert input from selected partners, we’re building the next-generation software capabilities designed for modern, data-driven power trading processes. The first Beta Version is ready for operationalisationWe are very proud to present the first Beta version of Power Market Operations to our partner customers. We are now entering a phase of stabilization before releasing to PROD and with that the opportunity to scale the product. Stay tuned.The product is relevant for market analysts, data scientists, production planners and traders who would like to improve data operations and power trading processes.The Beta version of Power Market Operations is the first step towards a fully integrated value chain, especially linking analysis with trading activities across all physical power markets, adding capabilities around continuously improving analyses, models and processes in real time on the basis of OOB benchmarking. This version of Power Market Operations is focused on improving the quality of day-ahead auction participation and helping production planners make decisions with a high confidence level.   The product generates a wide sample space on the basis of sophisticated scenarios for the bid matrix and can update the bid matrix based on events, such as price and water inflow. Cognite Power Market Operations uses event driven automation and frequent updates. By reducing the number of manual operations, it significantly shortens the day-ahead bidding process, and provides visibility around performance to enable quality improvement. Hence, production managers can use automated, detailed benchmarking to improve the process continuously.

Related products:Other

Cognite InField 10.12

This is an exciting release of InField that introduces the Sections feature for checklists based on templates.Sections make it possible to create neatly structured templates, routine round checklists, and checklist reports, with custom sections and subsections for logically grouped tasks.Checklist on desktop Checklist on mobile You can add sections and subsections to new and existing checklist templates if you're granted extended access rights to create templates. To create a section, click Create sections in the template creation view. Give the section a title, and move and indent checklist items that belong together below the section element. To create a subsection, drag and indent a new section directly below an existing section.Checklist items that don't belong to a group, you keep as is by placing them outside a section. You can format the section titles according to these guidelines. Select the Example button in the toolbar for inspiration for an example template with sections. FURTHER IMPROVEMENTSThis is the first version of creating checklist sections, and we'll continue delivering updates on this functionality based on your feedback. OPC UA time series dataStream OPC UA time series data directly from the control system to InField. On the Trends page and the Trends card on the Overview page, you'll see these time series labeled with OPC UA: View P&IDs on mobile unitsWe've learned that some users on some assets struggle to open P&IDs on mobile units. You may need to install a 3rd party file viewer for a particular P&ID document format on the mobile unit. Contact your organization's IT support for help with this.

Related products:InFieldProduct Releases

Cognite Data Fusion Release: April 2022

 Data governance is key to ensuring high quality throughout your data's lifecycle, and Cognite Data Fusion (CDF) has the tools and features to secure that the data meet your expectations. This release optimizes data management by introducing more granular access to data transformations using data sets.Read more about this and the other improvements for the CDF April 2022 release below. We cover the main features in the five-minute video above.  Scope transformation access to data setsProtect access to CDF SQL transformations from ungoverned actions and visibility by associating a transformation to a data set. Users will only be able to see and run the transformations associated with the data sets they have access to.- As a project admin granting users access to transformations, select the data sets a user can see and run transformations for under Scope in the Transformations capability dialog:        - As a user creating a transformation, select the data set you want to associate to a transformation in the Create new transformation dialog. Only users with transformation access to this data set can see and run the transformation.          This feature is also available in the CDF API, Python SDK, and Transformations CLI. Read more about transformations Ingest data into sequence rowsYou’ll now find Sequence Rows as an option in the Destination type field on the Transformations page in the CDF user interface and no longer need to use other developer tools like APIs and SDKs to ingest sequence rows into the Sequence resource type in CDF.        Develop applications using JavaAlready, you can build solutions using Cognite's SDKs. Now, we introduce the Cognite Java SDK to interact with the CDF v1 APIs. The Java SDK is optimized for data publishing and allows you to write data efficiently with the upsert function. The Java SDK covers core CDF functionality, such as contextualization and data extraction. You’ll find all the info you need in the cdf-sdk-java repo, and we have updated the Cognite API documentation  with code samples on how to perform tasks with the Java SDK. Change isStep setting for ingesting time series to CDFWhen you stream time series data points into the CDF data model, the interpolation between the data points is controlled by the isStep property on the time series object. You can now change the isStep property value without recreating and re-ingesting the entire time series.

Related products:Product Releases

Release 1.22 of Cognite Maintain

​Hello from the Maintain team!We’ve been focusing heavily on the future of scheduling. We’re happy to release significant changes to this feature as well as updates to the workspaces functionality and a standard slew of bug fixes.Scheduling upgradesWe’re introducing several new upgrades to our scheduling component, and laying the foundation for upgrades to come!You can now create operations. These operations are tied to Maintain and are used for ‘noting down’ new operations that are required. Once you have finalized your changes, you can export these and add them to your source system (e.g., SAP). You can adjust the granularity to see your schedule in a day, week, month, or year view - making it easier to view longer-lasting plans. You can drag and drop operations to reorder them to make the most sense to your plan. Drag and drop bars are now more user friendly and easier to do. The trade matrix will now use the number of workers as a baseline instead of the number of work hours each day. This will be the foundation for the teams concept coming up the next half year. We’ve given the page a new design refresh to make the experience cleaner and easier to use. Finally, we have added a selection of filters for you to use specific to the scheduling page. For example, filtering by the type of operation phase (Pre, Post or Exec).Workspace upgradesWe’ve given the stamp tool a redesign. We also have added a base set of P&ID symbols that you can use as stamps. You can upload images to CDF and use as stamps. If a document physically covers another document on the page, you can now move it between layers. You can now swap between pages of a PDF.Integrity and Major Risks DashboardWe have made multiple changes to improve the performance of the IMR-dashboard.Other updatesWe have improved the filtering options selection to be more performant and included the number of options this filter has.Bug fixesFixed issue with rendering suggestions. Comments tool in workspaces now works again. Fixed issue where the material status did not show on work orders. This release also contains bug fixes to improve performance and user experience.Behind the scenesWe’re looking forward to booking meetings with super users from our customers to start validating our roadmap plans for the upcoming months! Several users have raised some exciting ideas that we are developing a concept around. We hope that these will further increase the value of Maintain going forward. Head on over to the Maintain Group on Cognite Hub to learn more about specific features, what’s to come, and post a question if you have one!

Related products:Product ReleasesMaintain

Cognite Data Fusion Release: February 2022

The Product Release Spotlight Webinar recording for February can be found here. Cognite Data Fusion (CDF) streams data into the CDF data model, where the data is normalized and enriched by adding connections between data resources of different types to provide better business decisions. With this release, you can ingest geospatial data into CDF to contextualize and map with other CDF resource types to deliver accurate spatial analysis and statistics.Read more on this and all the other improvements for the CDF February 2022 release below. Geospatial dataWork with geospatial data using the Geospatial API and Python & JavaScript SDKs to handle the positioning and geometry of CDF data. We currently support a multitude of coordinate reference systems (CRS), but you can add other entries from the European Petroleum Survey Group (EPSG) as a custom CRS. You can also transform an input CRS to the defined CRS to improve the search in geometrics.  Read more. Ingest PI Asset Framework data to CDFThe new Cognite PI Asset Framework (AF) extractor is considered an extension to the Cognite PI extractor. The Cognite PI AF extractor retrieves a dump of the asset hierarchy from the Asset Framework on the PI Server and ingests this into Cognite RAW. Subsequently, the extractor regularly checks for updates in the PI asset framework and ingests any changes into CDF. Delete asset hierarchies using CDF TransformationsYou can now delete asset hierarchies in CDF using CDF Transformations. Items are deleted by their internal ID, and non-existing IDs are ignored. Read more about CDF Transformations here. Ingest data into CDF Relationships using PostgreSQL gatewayWe're rolling out the option to use the PostgreSQL gateway and Azure Data Factory (ADF) to ingest data directly into CDF Relationships. Select the public.relationships table in your ADF sink. Learn how here. Ingest time series data points directly into CDF Data points using the Cognite DB extractorThe Cognite DB extractor, version 2.4.1, provides a new option to ingest data points directly into CDF Data points instead of Cognite RAW. Set destination type as TIME_SERIES in your configuration file. Early adopter programsCognite offers several Early adopter programs to our users. We encourage you to take this opportunity to learn about and test our features currently in the making. Navigate to Cognite Hub , open the Group tab, and select Request to join to become an Early adopter candidate.

Related products:Product Releases

Cognite Data Fusion Release: December 2021

The Product Release Spotlight Webinar recording for December can be found here.  ​Trusting the data in Cognite Data Fusion (CDF) is the core of managing, building, and using the Cognite product suite. Data engineers need sufficient insight into the staged data to take the next step on the ingestion journey and the new data profiling report on staged data in Cognite RAW caters for this. This CDF release also introduces the new Cognite Solutions Portal. The Cognite Solutions Portal collects applications, use cases, and other helpful links to solutions your organization has built on top of CDF. Users no longer have to keep track of many different URLs to access the information they need in their daily work.Read more on this and all the other improvements for the CDF December release below: Data profiling on staged dataTo get in-depth knowledge about the data quality, discover patterns, outliers, and see other statistics in the new standard profiling report available on the Profile tab in Cognite RAW. Report your findings to the data owners and use this as a trigger to discuss topics such as the best fit for the primary key column and contextualization and to provide the best support for the end-users of the data. Keep iterating on the data integrations to improve the data quality and prepare the data transformation into the CDF data model. Read more here. ​Get notified about interrupted data flow into CDFCatch any issues that interrupt the data flow into CDF by setting up email notifications on the Extraction pipeline page. You can define the time condition for triggering the email. Learn how to set it up in this article.  New tools for running SQL TransformationsNew Transformations CLIThe new Transformations CLI replaces the old Jetfire CLI offering a better developer experience. Now, you can easily declare and manage variables in the configuration file without any hard-code. You install the CLI using the Python pip package manager. Read more here. APIs and Python SDK for TransformationsThe Transformations APIs and Python SDK are promoted to Version 1. Using the APIs/SDK, data engineers can orchestrate transformations sequentially, making it more reliable and quick. Check out the  API docs and SDK docs for more information.  Access management for SQL TransformationsYou can now manage access to SQL Transformations using the new transformations:read and transformations:write capabilities in your CDF group. We still support transformation access using the transformationsor jetfire groups for existing users to ensure backward compatibility. However, we recommend that you switch to managing access to Transformations using the new capabilities. Introducing the Cognite Solutions PortalAs companies ramp up their digitalization and transformation efforts, the number of applications, dashboards, and use cases increases rapidly. As a result, end-users have to keep track of many different URLs to access the information they need. This is the problem Cognite Solutions Portal sets out to solve. It gathers all your CDF-enabled solutions in one place. Use the Cognite Solutions Portal to add applications, dashboards from analytics and visualization tools, and helpful links into shared spaces.  Explore data in 3DThe CDF Data explorer already helps solution builders and data scientists find, validate, and learn about the data in CDF. Get the full experience by navigating and searching for assets and viewing the details in 3D models on the new 3D tab in the CDF Data explorer.    ChartsCognite Charts runs on top of  CDF and is a powerful tool for engineers and domain experts to explore, trend, and analyze industrial data. For more information, visit the Charts [Early Adopter] Group. This release has the following improvements: Automatic data alignmentCharts now automatically handle data alignment in your calculations. Time series data often have different sampling frequencies and time stamps, and it can be hard to do even the simplest calculations (e.g. timeseries_1 + timeseries_2), especially if you’re not proficient in Python.  Calculation results will be inaccurate and untrustworthy without properly accounting for data alignment.    A new and improved no-code calculation builder experienceThe no-code calculation builder has been redesigned and rebuilt on a new, powerful library.   

Related products:Product Releases

Cognite Data Fusion Release: October 2021

The Product Release Spotlight Webinar recording for October can be found here. This release of Cognite Data Fusion (CDF) contains enhancements to all capabilities for ingesting, managing & building trustworthy industrial data products to be used for data-driven decisions.   Data engineers need observability to inspect, monitor, and debug solutions in production. Therefore, we’re introducing Extraction pipelines where you can monitor the status of data ingestions to make sure reliable and trustworthy data are flowing into the CDF data sets. When you set up extraction pipelines, you can add comprehensive documentation and email alerts. You’ll be instantly notified, can quickly start troubleshooting, and have on-the-spot contact info to data set owners and other stakeholders if ingestion issues occur. You create and view these pipelines either from the Data set lineage page while you are already working with a particular data set, or you can get a bird’s eye view across several data sets via the Monitor extraction pipeline page.  Read more in these articles.If you’re using the PostgreSQL gateway, you can add status reporting to your Azure Data Factory pipeline. If you prefer working more programmatically, the new version of the Python Extractor Utils allows building extractors with status reporting, or you can use the REST API or Python SDK. The Cognite Extractors support status reporting from our next release. Running SQL transformations using APIs and the Python SDK significantly improves the developer experience as data engineers can manage transformations programmatically. Using the APIs/SDK, data engineers can orchestrate transformations sequentially, making it more reliable and quick. Check out the  API docs and SDK docs for more information. Note that the Transformations APIs and Python SDK are in public preview and will be promoted to V1 (production use) in November 2021.  Data discovery by contextualizing engineering diagrams, typically P&IDs, has already shown its value in the market. In this release, we’ve redesigned and enhanced the Create interactive engineering diagrams process to give a delightful user experience. The new sidebar navigation guides you carefully through each step in finding, extracting, and matching tags and linking these to the asset hierarchy and other associated resource groups. New actions that improve the quality of the diagrams include approving and rejecting automatically detected tags, clearing all tags on a diagram, and better name-mapping and use of synonyms in tag detection. When you’re done creating interactive diagrams, you can convert the approved PDF files to SVG if this is your preferred format. Read the How-to guides.  The solution builders and data scientists need transparent and easily accessible documentation and information about data science scripts and models to make correct and informed operational decisions. You’ll find it all in the Cognite Industrial Data Science Library.   Industrial data analytics must be of high quality and Cognite Charts is focused on the nitty-gritty details and to give you a smooth charting experience. You’ll find function descriptions by clicking the Information icon located next to a function. Notice also that named inputs/outputs in Cognite Charts make it clear to see which are required for a function. When you zoom in to a specific point on a time series, the aggregated values automatically switch to showing the raw data. In addition, all measurement units stored on time series in CDF are now displayed in Cognite Charts.  For the solution builders, we’ve enabled client credentials grant flow in Grafana. In some cases, for example, if you're using the Grafana free tier, you can not set up an identity provider for the whole Grafana instance. Instead, you need to set up a client credentials grant flow for each instance of the Cognite Data Source. You’ll find all the information you need to set this up in this article. For the CDF administrators, we’re introducing a new token session service for managing internal sessions and exposing internal and external endpoints for interacting with them. The token session API service allows users and service principals outside the API to run jobs within the API. External principals can use the sessions API to delegate their access to functionality in CDF. You can use this functionality to set up and manage long-lived or timed access for internal services. The access is rooted in the respective principal's external identity provider (IdP), from where it can be controlled or terminated. For example, a user can specify a function to be run on their behalf and terminated if their user account is deleted. 

Related products:Product Releases

Cognite Data Fusion Release: August 2021

Today, we are happy to announce the release of the following improvements to Cognite Data Fusion (CDF). Manage RAW foreign table in the Postgres GatewayYou can now manage the RAW foreign table in the Postgres Gateway using APIs and don’t have to rely on external tools such as pgAdmin or psql CLI to create, list, update or delete the RAW foreign tables.Update labels on assetsTo make it easier to update labels on assets, we have added a set method in the updateAssets operation. The operation accepts a list of labels to set or replace for the resource.Update sequence configurationWith the availability of mutable sequences, you can now update an existing sequence configuration. In addition to adding, removing and updating columns in your sequence, you can also update or clear the assetId, externalId, metadata, and the dataSetId for a sequence. To update a sequence, you need sequencesAcl:WRITE capabilities.Group limitsTo support authentication for users who are members of a large number of Azure AD groups we have increased the default group limit token configuration to 1.000 for new CDF projects with AAD as the IdP from now on.Interactive engineering diagramsWe’re hard at work putting the final touches on our new user experience for working with interactive engineering diagrams. Stay tuned for the upcoming release. Cognite ChartsRelease HighlightsThe main new features in this Cognite Charts release are: Search for time series data by time series ID Highlights matches in search results Recently viewed tags and time series are displayed in the search panel Merge y-axes with the same units Show/hide min/max bands Show/hide gridlines Backend performance improvements Bug fixes & UI/UX improvements Search for time series data by time series IDWe have added a tab in the search panel to search for “Time series ID,” specifically. This tab should be used when you know the name of the time series you are searching for (e.g. VAL_21_PT_1019_04:Z.X.Value), while the “Tag number” tab should be used to search for the asset tag to which the time series of interest is related (e.g. 21PT1019). You will also notice that search results that match your query are now highlighted to make direct matches easy to seeSearch by Time series ID You’ll also see the tags (assets) and time series you have recently viewed (added and/or removed from your chart) in the search panel, if you have not begun to type a search query.Basic Statistics & Tag MetadataUsers can view additional basic statistics (mean, std. dev., etc.) for the a given time series for the time range they’re currently viewing in the chart by clicking on the corresponding information icon. By clicking on the “Metadata” tab in this panel on the right-hand side of the screen, can view the metadata information for that Basic statistics Metadata Merge y-axes with same unitsIt’s now possible to easily compare the values of two or more time series or calculations that have the same unit, without needing to manually match and align the axes individually. Simply click the y-axis button found just above your chart and select the “Merge units” toggle. In order for this feature to work, you must have already assigned the input and output units on the time series or calculation(s) of interest.Show/hide min/max bandsWhile looking at a wider range of time, Charts will show the aggregated values for a given time series. With the min/max bands, it’s possible to visually display the min and max range for a time series.Show/hide gridlinesGridlines can be a valuable way to visually identify time series values, but they could also get in the way and be confusing when you have a lot of time series data on your Chart. Use the show/hide gridlines toggle to decide for yourself whether or not you want them displayed on your chart.

Related products:Product Releases

Cognite Data Fusion - June 2021 update

This release of Cognite Data Fusion (CDF) sees Charts released into general availability for all customers. Also, we're adding a PostgreSQL gateway to ingest data into CDF from your existing ETL solutions. Charts released into general availabilityThis release sees Charts released into general availability. We're gradually rolling out Charts to our customers, and it may not yet be available in your CDF project. We plan to have Charts available for all projects by June 10.Use Charts to explore, trend, and analyze industrial data. Find, understand, and troubleshoot for actionable insights from time series and contextualized engineering diagrams already in the CDF portal application or other Cognite applications, then share the chart for collaboration. Charts runs securely with a single, secure sign-in.Learn more in the Charts documentation, and join the conversation in the Charts community on Cognite Hub. CDF Transformations now support OpenID ConnectYou can now use OpenID Connect and your existing identity provider (IdP) framework to manage access to CDF Transformations securely. We currently support Azure AD, Microsoft's cloud-based identity and access management service.When you have registered CDF Transformations in Azure AD, users can sign in with their organizational ID to transform data in a CDF project.You can update existing transformations that are currently using API keys to use OIDC client credentials instead.NOTE: CDF projects running in Google Cloud Platform need to be added to an allowlist (sometimes called a whitelist) to support scheduling of transformations using OpenID Connect. Contact your Cognite representative to be added to the allowlist.Learn more in Setup and administration with OpenID Connect. PostgreSQL gatewayUse the PostgreSQL gateway to ingest data into CDF from popular ETL solutions. The CDF resource types appear as tables in the PostgreSQL database and you can ingest data directly into CDF resources like assets, events and datapoints and to Cognite RAW.We currently support Azure Data Factory as the ETL tool. The PostgreSQL Gateway is intended for ingestion only. We do not support querying data from CDF using the gateway.Learn more: PostgreSQL gateway Grafana - nested variablesIn Grafana, you can now link variables and use the value of one variable as input to another.Learn more: Nested (chained) variablesSneak peek at the new interactive diagrams experienceWe’re excited to give you a sneak peek of the latest redesign and enhancements to the interactive diagrams experience before it’s released publicly. We encourage all customers to try out the new features and give us feedback to help us make them better! Power BI - templatesWe have added support for Templates in Power BI to early adopters. Contact your Cognite representative to give it a try and share your feedback!

Related products:Product Releases