Cognite Data Fusion - October 2020 updates

As the leaves turn orange and red and autumn settles in Norway, we continue our rapid iterations to bring you new enhancements.

• We are introducing a huge enabler to further enrich your industrial data in Cognite Data Fusion (CDF): Relationships - now available in beta.

• We continue our investment to enable users to explore data using domain terms by adding support for labels on the Files resource type.

  Additionally, in a new dedicated directory property, you can now store information about the source system directory where the file originates from. These new features allow data engineers to model data in ways that enable a broad range of solutions.

• For all application developers and data scientists out there, we’re adding support for configuring alignment for synthetic time series.

Our next major release, with a bundle of exciting features, is planned for December 2nd. We’re already doing exciting prototyping and beta testing around data discovery for solution developers, scheduling of data pipelines, and administration and monitoring of ingestion pipelines. Much to look forward to!

More flexible modeling of industrial data through Relationships

With the introduction of the Relationships resource type, we enable far more powerful ways to model industrial data. While previously, you could only model equipment relations as a tree, you can now model data flexibly in a graph.

In this release, we are moving relationships from Playground to Beta.

Data engineers can use relationships to organize your assets into multiple structures, tailored for your organization's purposes. For example, you can organize assets by the process flow or the physical connectivity in a grid structure to let users efficiently locate data for nearby equipment.

And while some users in your organization find it useful and efficient to organize the equipment by its function in a maintenance hierarchy, others may benefit from having them organized by physical location in a location-based hierarchy:

In short, relationships give you the flexibility to organize assets in structures that are meaningful for your organization. See Modeling asset structures with relationships for more example structures.

Beyond structuring assets, you can use relationships to create richer connections between data elements and build a knowledge graph on top of the data you already have.

To learn more about how relationships can represent industrial reality, we recommend that you read the "Building a contextualized power system network model" whitepaper.

Adding labels support for the Files resource type

We continue our investment to enable users to explore data using domain terms. Labels are now supported on Files resource, available through the API, the Python SDK and the JavaScript SDK.

from cognite.client import CogniteClient
from cognite.client.data_classes import LabelFilter

c = CogniteClient()

my_label_filter = LabelFilter(contains_all=["WELL LOG", "VERIFIED"])
file_list = c.files.list(labels=my_label_filter)

As a data engineer, you can now classify files by what kind of file it is. Application developers can build good end-user experiences and let users browse files by category. For example, end users can browse files grouped as P&IDs, logic diagrams, and datasheets.

Enhanced lineage information for files

The new directoryPrefix property allows you to store and filter information about the directory in the source system where the file originates from. The new functionality is available in the API and the Python SDK.

from cognite.client import CogniteClient

c = CogniteClient()

file_list = c.files.list(
  limit=5,
  directory_prefix="/usr/local/share/jupyter/"
)

Configurable aggregation alignment

We are introducing a new configuration option for aggregation in synthetic time series: alignment.

New option enables you to:

• Create weekly aggregates that start on specific days of the week, for example, Saturday, or Monday.
• Create aggregates that align to different time zones.

For example, the chart below shows the daily average chlorine indicator for city water aligned to two different time zones: GMT+0 and GMT+3.

In the above graph, the new alignment option allowed us to align daily averages to timestamps in the desired time zones:

• Daily averages aligned to UTC/GMT+0: ts{externalId='houston.ro.REMOTE_AI[34]', alignment=1599609600000, aggregate='average', granularity='24h'}
• Daily averages aligned to UTC/GMT+3: ts{externalId='houston.ro.REMOTE_AI[34]', alignment=1599620400000, aggregate='average', granularity='24h'}

By default, aggregation with synthetic time series is aligned to Thursday 00:00:00 UTC on 1 January 1970.

The new parameter is available in the Synthetic Time Series API, and can be used as part of query expressions without needing to update your SDK, as well as part of custom queries in Cognite Data Source for Grafana 2.0+.