Cognite Toolkit, build/lineage.yaml, deployment access control and multiple teams

Hey!

We are looking into how to improve access control in our GitHub monorepo which deploys using Cognite Toolkit. We use a GitHub CODEOWNERS file to configure which teams approve changes to which code. Here are some options I see and the problems that I see with them:

Option 1: Using different deployment credentials for different sets of Cognite Toolkit modules

This initially seemed to me like the most natural option as we then use the access control system of the target platform, but we suspect that dealing with deployment ordering is then going to be too complicated for us to handle unless there is support for this approach from Cognite.

Option 2: Checking that YAML source files do not go out of the bounds of their modules

We then check that source files in a given module do not, for example, deploy into a CDF space managed by a different team, or reference environment variables containing secrets fetched from another team’s vault.

My concern with this approach is that Cognite Toolkit’s variable expansion might radically change the built resource. A variable referenced at the end of a manifest might contain a YAML document start marker followed by a manifest that is different from the one that was checked.

Option 3: Use build/lineage.yaml to check that built resource files do not go out of the bounds of their modules

After running cdf-tk build, build/lineage.yaml contains a mapping between built resources and source file paths. We can use it to check that built resource files do not go out of the bounds of the source modules or reference environment variables containing secrets from a different team’s vault. This partially addresses my concern with option 2, because we then check the files after expansion of variables defined in config.<env>.yaml files.

However, expansion of environment variable syntax like ${ENV_VAR} only happens when running cdf-tk deploy, and could also potentially alter the semantics of the YAML manifests. Checking that environment variable references are legal also appear to be complicated by that Cognite Toolkit currently sometimes expands environment variable references in code expanded from another environment variable.

To my knowledge, Cognite has not documented the file format of any of the files in the build directory, so their format might not be stable.