How to manage Cognite Canvas and Atlas AI agents during data modelling version deprecation

If you use Data Modelling (DM), you either have faced this issue already, or you will very soon.

A DM  is versioned, which is a very important feature to avoid breaking existing solutions. However, since there is a limit to the number of Views and versions you are allowed to have in a CDF project, you need to deprecate old versions at some point (in particular in periods with heavy DM development).

Currently it is not possible to extract from the logs which View and which version are in use. This is a fundamental problem that needs to be solved, but we will not discuss that here.

CDF offers three UI features that consume the DM instances of a specified version: Atlas AI agents, Canvas and Charts. It seems they, at the time of the data being added, they use either the latest version of the data model, or the version specified in the location filter (someone in Cognite can confirm this). It means that an evolving Canvas can potentially have instances linked to different versions inside the same Canvas.

Currently there is no way to discover, via the API, the version of the instances used by Charts. However, for Canvas and Atlas AI it is, and this is what this post is about.

I have attached two python notebooks to this post, one for Atlas AI agents and one for Canvas. They run through your cdf project, analysing every agent and every Canvas, check which View/version is used, flag them if they are already deleted, and list all users that created those Canvases/agents. They include some other analysis as well.

One thing to note is an added analysis in the Canvas script, detecting“orhpaned references”. Due to a (reported) bug in Canvas, if you delete a Canvas that contains a reference to a deleted DM instance, that reference will not be deleted and will live in cdf as a  “orhpaned references”, eating from the available instance count. Since you can have multiple “historical versions” of a Canvas, the number of “orhpaned references” can potentially be high for some Canvases.As an example, we had 5k “orhpaned references” in our production cdf project. If you explicitly delete the reference from the Canvas, it is deleted in cdf as well.

The “orhpaned references” can easily be deleted via the API, so you might want to run a “cleaup script” on a regular basis until this bug is resolved.

Note that you have to specify the following information at the start of the notebooks. 
PROJECT = "my-cdf-project"
TENANT_ID = "my-tenant-id"
CLIENT_ID = "my-client-id"
CDF_CLUSTER = "my-cluster”

This information can be a little hard to find. Note that PROJECT and CDF_CLUSTER can be found in the url after you log into cdf

https://my-company.fusion.cognite.com/my-cdf-project?cluster=my-cluster.cognitedata.com&workspace=data-fusion

If you are not able to find the information above, you can always copy the code over to a CDF hosted notebook, and then manually copy the content of the html file into a local file (remove the part of the script that write the actual file)

Be prepared to do some local adjustments

Best of luck

PS: These type of analysis, and tools to migrate Atlas/Canvas/Charts to new versions, should in my opinion be part of the in-build toolkit in CDF. Hopefully they will be at some point