Cognite Replicator

A common setup that we have at Cognite is having 3 different environments: development, test and production. We have one specific Cognite Data Fusion (CDF) project for each of these environments. This setup allows to develop new features without interfering with production data. Also, it makes the deployment of new features to production much safer, since they are tested before.

Sometimes, you still want to have the data you used during your tests, in your production environment. For example, if you extracted some data from sensors and saved them as time series in CDF, you may want to keep this historical data when you move to production. Same goes with files, assets, datasets, functions etc. Another example could be the replication of the data from a full production CDF project, to one of your customer’s project, containing only a subset of the source one.

At Cognite, we have a python package (available at https://pypi.org/project/cognite-replicator/) whose purpose is to copy the selected resources from one project to another. It is quite easy to use: you only have to fill a config file with some information about authentication, the resources you want to replicate and some additional features. Then it is run thanks to a CI/CD pipeline. The time it takes depends on the data you replicate, but it is generally less than a minute. Having this package makes the lead time between development and production much faster, and allows to continuously add new features to your production environment, at scale.

This package is still getting updated with some requests from customers: if you would like to see a new feature in it, please let us know!

@Gaetan Helness, you also worked with the replicator package: any tips or best practices you would like to share ?