About Containers, Properties and scale in CDF Records

As illustrated in our documentation, the CDF Records feature uses a Data Modeling container as the schema definition for record data. I.e. you have to create a space and container first, before you start loading Records to a Stream. (Note: It is possible to use multiple containers together to define the schema for a single record. This may make sense in the context of, for instance, a Work Order record).

However, at the moment when I write this, using the container based schema represents a somewhat confusing “limitation” when it comes to the size of a container vs the size of a record (number of properties). The way we have implemented this capability in CDF Records at the moment - using the DM containers - comes with a side-effect: The number of properties you can have for a single container is, as of right now, the same as it is in your Data Modeling service.

The limits we're documenting in terms of properties for CDF Records are linked to the properties containing data within a single stream. I.e. if the limit for a stream template is set to 1000 properties, you can have a single container with 1000 properties where they all have data, or you could have 100 containers with 10 properties.

We're still developing the functionality to split the Records container limits from the DM container limits. Once we complete this work prior to GA, we will announce it in this forum. The resulting implications of the upcoming records container property limits will be as follows; We expect to support in excess of 1000 properties per stream "out-of-the-box" for "Record containers" at General Availability (GA). This limit will likely be configurable by Cognite, as we expect to see customers with needs in excess of 1000 properties in a single stream immediately at release. The updated container limits will mean that for customers planning on using records to ingest large numbers of containers with large property counts, there may need to be some container/stream layout planning before ingesting data to CDF Records.

For example, if you'd want to ingest up to 190 000 properties (assuming they're distributed across approximately 1200 tables in your source system), you should consider how to "lay out" your records across streams. With the current maximum of 10 streams per CDF project, those of your source 1200 tables that you expect to retrieve data from frequently, with significant aggregate retrieval and filtering, should perhaps be placed in their own stream. You should not design those streams to contain more than approximately 3000 properties with data, in total per stream.

For source tables where you expect to access data infrequently, you perhaps can consider either using JSON blob based properties to hold the data, or in the case where you will need infrequent filtering for most of those properties you can consider upgrading your per-stream property capability to support up to 10000 properties, and define containers that match your "low access frequency" records in those stream(s).