Answer

How cdf toolkit resolve dependencies between views ?

Forum|Forum|8 months ago
March 26, 2025
5 replies
59 views

Marwen TALEB
MVP

Hello,

We are using cdf toolkit to deploy our container, views and data models.

We recently added many new views, these views are dependent. i.e: they reference each other in the properties through a source block or via implements block. Here an examples of these view dependencies:

- externalId: View1
  implements: []
  name: View1
  properties:
    test:
      container:
        externalId: Container1
        space: some_space
        type: container
      containerPropertyIdentifier: Test
      name: test
      source:
        externalId: View2
        space: some_space
        type: view
        version: v1
  space: some_space
  version: v1

- externalId: View2
  implements: []
  name: View2
  properties:
    wow:
      container:
        externalId: Container2
        space: some_space
        type: container
      containerPropertyIdentifier: Test
      name: wow
  space: some_space
  version: v1

- externalId: Country
  implements:
    - externalId: CountryAttributes
      space: '{{sp_dm_dap_knowledge_graph}}_wv'
      type: view
      version: '{{dap_version}}'
  name: Country
  properties:
    Wells:
      connectionType: multi_reverse_direct_relation
      name: Wells
      source:
        externalId: Well
        space: '{{sp_dm_dap_knowledge_graph}}_wv'
        type: view
        version: '{{dap_version}}'
      through:
        identifier: Country
        source:
          externalId: Well
          space: '{{sp_dm_dap_knowledge_graph}}_wv'
          type: view
          version: '{{dap_version}}'
  space: '{{sp_dm_dap_knowledge_graph}}_wv'
  version: '{{dap_version}}'

- externalId: CountryAttributes
  implements: []
  name: CountryAttributes
  properties:
    CountryName:
      container:
        externalId: Country
        space: '{{sp_dm_dap_knowledge_graph}}'
        type: container
      containerPropertyIdentifier: CountryName
      name: CountryName
    RegionalEntity:
      container:
        externalId: Country
        space: '{{sp_dm_dap_knowledge_graph}}'
        type: container
      containerPropertyIdentifier: RegionalEntity
      name: RegionalEntity
  space: '{{sp_dm_dap_knowledge_graph}}_wv'
  version: '{{dap_version}}'

We have these errors:

ERROR (ResourceCreationError): Failed to create resource(s). Error: Cannot upsert view 'sp_dm_dap_knowledge_graph_wv:Country/0.0.28'. Implemented view 'sp_dm_dap_knowledge_graph_wv:CountryAttributes/0.0.28' does not exist. | code: 400 | X-Request-ID: 027470c1-3679-9e04-a540-68ab60d31be8 | cluster: westeurope-1
The API Failed to process some items.
Successful (2xx): []
Unknown (5xx): []
Failed (4xx):

IMHO, this is issue in CDF toolkit dependencies resolutions. How CDF toolkit does this? How to avoid this errors?

Thank you

Best answer by Anders Albert

Just so I understand you correctly, you are trying to deploy multiple views which have interdependencies with CogniteToolkit?

Some context here. There is a bug in the API that is on process of being fixed. CogniteToolkit has implemented a workaround from version 0.4.11, that does some analysis of the view dependencies and deploys them sequentially. Thus if you are on an earlier version than 0.4.11, I suggest you upgrade.

Anders Albert
Seasoned Practitioner
Answer
Forum|Forum|8 months ago
March 26, 2025

Just so I understand you correctly, you are trying to deploy multiple views which have interdependencies with CogniteToolkit?

Marwen TALEB
Author
MVP
Forum|Forum|8 months ago
March 27, 2025

Thanks @Anders Albert , this seems to resolve our issue. However we notice another one when deploying files.

In fact we have this error:

Traceback (most recent call last):
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/bin/cdf-tk", line 8, in <module>
    sys.exit(app())
             ~~~^^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf.py", line 116, in app
    _app()
    ~~~~^^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/typer/main.py", line 339, in __call__
    raise e
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/typer/main.py", line 322, in __call__
    return get_command(self)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/typer/core.py", line 740, in main
    return _main(
        self,
    ...<6 lines>...
        **extra,
    )
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/typer/core.py", line 195, in _main
    rv = self.invoke(ctx)
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/typer/main.py", line 697, in wrapper
    return callback(**use_params)
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/apps/_core_app.py", line 280, in deploy
    cmd.run(
    ~~~~~~~^
        lambda: cmd.execute(
        ^^^^^^^^^^^^^^^^^^^^
    ...<9 lines>...
        )
        ^
    )
    ^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/commands/_base.py", line 55, in run
    raise e
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/commands/_base.py", line 52, in run
    result = execute(*args, **kwargs)
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/apps/_core_app.py", line 281, in <lambda>
    lambda: cmd.execute(
            ~~~~~~~~~~~^
        env_vars=env_vars,
        ^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
        verbose=verbose,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/commands/deploy.py", line 165, in execute
    result = self.deploy_resources(
        loader_instance,
    ...<6 lines>...
        verbose=verbose,
    )
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/commands/deploy.py", line 197, in deploy_resources
    return self._deploy_resources(
           ~~~~~~~~~~~~~~~~~~~~~~^
        loader, env_vars, state.read_modules, dry_run, has_done_drop, has_dropped_data, force_update, verbose
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/commands/deploy.py", line 224, in _deploy_resources
    to_create, to_update, to_delete, unchanged, duplicated = worker.load_resources(
                                                             ~~~~~~~~~~~~~~~~~~~~~^
        files,
        ^^^^^^
    ...<3 lines>...
        verbose=verbose,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/loaders/_worker.py", line 165, in load_resources
    cdf_dict = self.loader.dump_resource(cdf_resource, local_dict)
  File "/Users/marwen.taleb/Library/Caches/pypoetry/virtualenvs/drillx-dwdap-cdf-toolkit-fxS2v_xU-py3.13/lib/python3.13/site-packages/cognite_toolkit/_cdf_tk/loaders/_resource_loaders/file_loader.py", line 146, in dump_resource
    if ds_id := dumped.pop("dataSetId"):

When trying to deploy this file conf :


- dataSetExternalId: ds_streamlit_apps
  directory: /streamlit-apps/
  externalId: xxxxxxxx
  metadata:
    name: xxxxxx
    published: true
    theme: Light
  mimeType: application/json
  name: xxx.json
  source: Toolkit

I believe this is a bug since in file conf there is no dataSetId but dataSetExternalId as per your documentation.

Could you please take a look, thanks alot !

Anders Albert
Seasoned Practitioner
Forum|Forum|8 months ago
March 28, 2025

You should never get a traceback like that, thus you are absolutely correct. Logged it as bug, and will let you know once it is fixed.

Anders Albert
Seasoned Practitioner
Forum|Forum|8 months ago
March 31, 2025

@Marwen TALEB This should be fixed in the newly released 0.5.1

Marwen TALEB
Author
MVP
Forum|Forum|8 months ago
April 1, 2025

Thanks a lot @Anders Albert for your quick fix!

Sign up

Welcome to the HUB

Scanning file for viruses.

This file cannot be downloaded