Cognite Data Fusion Release: October 2021

The Product Release Spotlight Webinar recording for October can be found here.

This release of Cognite Data Fusion (CDF) contains enhancements to all capabilities for ingesting, managing & building trustworthy industrial data products to be used for data-driven decisions.  

Data engineers need observability to inspect, monitor, and debug solutions in production. Therefore, we’re introducing Extraction pipelines where you can monitor the status of data ingestions to make sure reliable and trustworthy data are flowing into the CDF data sets. When you set up extraction pipelines, you can add comprehensive documentation and email alerts. You’ll be instantly notified, can quickly start troubleshooting, and have on-the-spot contact info to data set owners and other stakeholders if ingestion issues occur. You create and view these pipelines either from the Data set lineage page while you are already working with a particular data set, or you can get a bird’s eye view across several data sets via the Monitor extraction pipeline page.  Read more in these articles.

If you’re using the PostgreSQL gateway, you can add status reporting to your Azure Data Factory pipeline. If you prefer working more programmatically, the new version of the Python Extractor Utils allows building extractors with status reporting, or you can use the REST API or Python SDK. The Cognite Extractors support status reporting from our next release.

Running SQL transformations using APIs and the Python SDK significantly improves the developer experience as data engineers can manage transformations programmatically. Using the APIs/SDK, data engineers can orchestrate transformations sequentially, making it more reliable and quick. Check out the  API docs and SDK docs for more information. Note that the Transformations APIs and Python SDK are in public preview and will be promoted to V1 (production use) in November 2021. 

Data discovery by contextualizing engineering diagrams, typically P&IDs, has already shown its value in the market. In this release, we’ve redesigned and enhanced the Create interactive engineering diagrams process to give a delightful user experience. The new sidebar navigation guides you carefully through each step in finding, extracting, and matching tags and linking these to the asset hierarchy and other associated resource groups. New actions that improve the quality of the diagrams include approving and rejecting automatically detected tags, clearing all tags on a diagram, and better name-mapping and use of synonyms in tag detection. When you’re done creating interactive diagrams, you can convert the approved PDF files to SVG if this is your preferred format. Read the How-to guides.

The solution builders and data scientists need transparent and easily accessible documentation and information about data science scripts and models to make correct and informed operational decisions. You’ll find it all in the Cognite Industrial Data Science Library.   

Industrial data analytics must be of high quality and Cognite Charts is focused on the nitty-gritty details and to give you a smooth charting experience. You’ll find function descriptions by clicking the Information icon located next to a function. Notice also that named inputs/outputs in Cognite Charts make it clear to see which are required for a function. When you zoom in to a specific point on a time series, the aggregated values automatically switch to showing the raw data. In addition, all measurement units stored on time series in CDF are now displayed in Cognite Charts. 

For the solution builders, we’ve enabled client credentials grant flow in Grafana. In some cases, for example, if you're using the Grafana free tier, you can not set up an identity provider for the whole Grafana instance. Instead, you need to set up a client credentials grant flow for each instance of the Cognite Data Source. You’ll find all the information you need to set this up in this article.

For the CDF administrators, we’re introducing a new token session service for managing internal sessions and exposing internal and external endpoints for interacting with them. The token session API service allows users and service principals outside the API to run jobs within the API. External principals can use the sessions API to delegate their access to functionality in CDF. You can use this functionality to set up and manage long-lived or timed access for internal services. The access is rooted in the respective principal's external identity provider (IdP), from where it can be controlled or terminated. For example, a user can specify a function to be run on their behalf and terminated if their user account is deleted.