During the last 2 years, Cognite has taken a very iterative approach to enable flexible data modeling. The key highlight towards this effort can be outlined through 3 key initiatives - Templates, Schema Service and ITG. In this article we will hope to outline the key difference between the 3.
Templates
Templates is a data modeling capability built on the concepts of GraphQL. Through building a GraphQL data model, you can define the data model that is intended for your solution. Additionally, you can load data into the data model via referencing existing CDF resources or loading in values directly for primitives like String, Boolean, and Float. Lastly, you get the ability to query the data via a GraphQL endpoint, supporting pagination and complex logical filters (with AND
s and OR
s).
Here is a list of features doable via Templates
Modeling data model
- Primitives (String, Boolean etc.)
- Relationships between them (1-1, 1-N, N-M).
- References to Asset, TS, Sequence, File
- Versioning
Querying
- Basic filters (numbers comparison, and, ors)
- Nested queries
- Pagination
SDK
- Data Model Management
- Querying
- Add Instances
However, there were core features users asked for that were impossible to handle with Templates based on the initial architecture. These are things like advanced fuzzy search, data mapping / view, role based access control. The team is currently focusing efforts towards a new flexible data modeling service called Schema Service on top of a much more robust storage layer to support everything Templates can do, but also more. A migration will be needed from users of Templates towards the new Schema Service, which will be very easy and the team will provide extensive assistance in the migration.
Templates will no longer have large ongoing feature development, mostly remaining in maintenance mode as of January 2021. However, Templates will not be deprecated until there is a clear migration plan to Schema Service as it is powerful enough for most POCs and use cases.
Due to the ease of migration, we strongly recommend starting with Templates for any flexible data modeling needs, check this guide to get started.
Schema Service
Schema Service is a WIP flexible data modeling service that will replace Templates API towards the end of 2022 H2. It also leverages GraphQL for data model definition and querying. It will share a very similar querying and modeling experience to Templates that makes it very easy to migrate. However, there will be a big change between how data is stored for a Schema Service vs Templates which will be a more difficult migration. Where Templates stores data as a persistent layer tied to the data model, Schema Service will treat data more independently from the data model, which will end up more like “views” on top of the data. Hence the migration will be more difficult for the data “ingestion” into Schema Service.
The key additional feature add of Schema Service will be
- Granular access control
- Reusable data model
- Reusable data via “mapping”
- Single source of truth of data… with different views.
- Data Profiling
- Querying - Search, Sort, Aggregation
- Graph queries
Keep in mind, Schema Service will also grant a significant quota and performance boost to what was in Templates, but all the workflow related to Templates will be extremely similar. Hence any feedback provided for Templates will help make Schema Service better for you when it launches! To get started with Templates, check out this guide
ITG - Cognite EPC teams’s offering
Focusing also on flexible data modeling, this was an additional test bed for Cognite for seeing how to handle cross data model interactions and data ingestion experience. It is another GraphQL interface for data model definition and querying. These learnings are what lead into the creation of Schema Service in addition to feedback of Templates.
Non-EPC customers are highly encouraged to use Templates or wait for Schema Service as ITG is focusing towards very different use cases.