Skip to main content

Cognite Data Fusion Toolkit

  • 38 Topics
  • 103 Replies
Andre Alves
MVP
Andre AlvesMVP
Issue Installing cognite-toolkit (0.4.15) – FileNotFoundError

Hi Cognite Team, I’m encountering an issue when trying to install cognite-toolkit (version 0.4.15) using Poetry. The installation fails with the following error: Installing C:\Users\andre.carvalho\AppData\Local\pypoetry\Cache\virtualenvs\cognite-toolkit-playgroud-gU7xo0oD-py3.11\Lib\site-packages\cognite_toolkit\_builtin_modules\contextualization\cdf_entity_matching\default.config.yaml - Installing cognite-toolkit (0.4.15): Failed FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\andre.carvalho\AppData\Local\pypoetry\Cache\virtualenvs\cognite-toolkit-playgroud-gU7xo0oD-py3.11\Lib\site-packages\cognite_toolkit\_builtin_modules\contextualization\cdf_entity_matching\extraction_pipelines\ctx_files_entity_matcher.ExtractionPipeline.yaml' Before running poetry add cognite-toolkit, I ensured my environment was set up correctly. However, it seems like the package is missing required files. Could you confirm if this is a known issue or if there’s a recommended wo

183
Andre Alves
MVP
S
Active
Silje RaddumActive
Session  IdP refresh failed

We are trying to deploy a new project with cdf toolkit, but get the following error: ERROR (ResourceCreationError): Failed to create resource(s). Error: Session IdP refresh failed | code: 400 | X-Request-ID: 34cc5242-b88e-991e-87ef-c2b00a6df73e | cluster: api.ERRORERROR: build step 0 "cognite/toolkit:0.4.2" failed: step exited with non-zero status: 1

242
S
Active
M
Seasoned
Marwen TALEBSeasoned
Default views (filter, prefrences, locations..) in Industrial search UI

Hello, Could we define default filters, locations, column list in search UI to not redo the same work every time we connect to industrial tools search UI. Could you please transform this into a feature request if this is not available today? Thanks

574
Sofie Svartdal Berge
Seasoned Practitioner
Ayush Daruka
Seasoned
Ayush DarukaSeasoned
Override type for direct relations

I came across this documentation regarding how we can use GQL to override the type of direct relation when using a CDM feature.Is there a similar documentation to highlight the changes needed to be made in the YAMLs when using CDF toolkit to deploy our model? Please note that I tried dumping the working GQL, but the resulting yamls did not work when tried the other way around.

906
Anders Albert
Seasoned Practitioner
J
Seasoned
Jesse JenkenSeasoned
Unable to Deploy IAM Groups: cognite.client.exceptions.CogniteAPIError: Failed to buffer the request body │ code: 400

We have encountered a scenario where we are unable to continue deploying to our production environment. The deployment fails on the IAM groups which indicate an HTTP Error 400 with a note indicating the Cognite API Failed to bugger the request body. Here is the important parts of the error: ...Deploying 19 iam.groups(resource_scoped) to CDF...╭──────────────────────────────────────────────────────────────────────────────╮│ Traceback (most recent call last): ││ File ││ "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/cognit ││ e_toolkit/_cdf_tk/commands/deploy.py", line 423, in _update_resources ││ updated = loader.update(resources) ││ ^^^^^^^^^^^^^^^^^^^^^^^^ ││ File ││ "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/cognit ││ e_toolkit/_cdf_tk/loaders/_resource_loaders/auth_loaders.py", line 402, in ││ update ││ return self._upsert(items) ││ ^^^^^^^^^^^^^^^^^^^ ││ File ││ "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/c

753
J
Seasoned
Ayush Daruka
Seasoned
Ayush DarukaSeasoned
CDF Toolkit Instances deployment using csv file

Can we use a CSV file instead of a Yaml file containing details of the instances we want to populate in a view using CDF toolkit? If yes, is there a limit to it? Edit: What I mean to ask is whether we can use a csv file directly to populate a view in a data model, i.e. without using raw and transformations? Just like we can use a node.yaml file to directly deploy an instance in a view.

939
Ayush Daruka
Seasoned
B
Committed
Basudeba MandalCommitted
Unable to deploy streamlit app using CDF Toolkit (alpha version)

Hi Team, We are attempting to deploy a Streamlit app using the CDF Toolkit (alpha version), but we've encountered the following error during the process. Could you please assist us in resolving this issue so that we can effectively use the toolkit for deploying the app? Below are some additional details for your reference: CDF Toolkit Version: 0.3.0a7Target CDF Project: slb-pdfI've also attached the cdf.tmol file for your review. Error Snippet: from ._auth_app import AuthApp File "/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/cognite_toolkit/_cdf_tk/apps/_auth_app.py", line 6, in <module> from cognite_toolkit._cdf_tk.commands import AuthCommand File "/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/cognite_toolkit/_cdf_tk/commands/__init__.py", line 1, in <module> from .auth import AuthCommand File "/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/cognite_toolkit/_cdf_tk/commands/auth.py", line 34, in <module> from cognite_

465
B
Committed
S
Seasoned
Snehal-JagtapSeasoned
Utility to take dump of CDF project / Convert graphQL model to yaml files for CDF toolkit

Hello Team, Is there any utility to convert GraphQL model in CDF to yaml files which can be used in CDF toolkit for deployment. Currently, even after taking dump using SDK, we need to change the format of YAML as per the supported syntax. Thanks, Snehal,

597
S
Seasoned
P
Practitioner
palronningArchitect
Bullet-proofing Streamlit Apps in Cognite Data Fusion

From Prototype to Production: Developing and Deploying Robust Streamlit Apps in Cognite Data Fusion Streamlit dashboards and applications can easily be built and deployed in Cognite Data Fusion using the built-in, browser-based code editor. It gives instant previews and is easy to prototype with, especially when using Copilot and starting with available templates. Once a Streamlit app or dashboard is starting to get traction and users depend on it in their everyday work, it is time to start thinking about how you manage changes and how you make sure those changes don’t introduce bugs or have unwanted side effects. We want to be able to make improvements, not frustrate users or make them loose trust in the application or the data. Here’s how you can introduce a robust process for developing, testing and deploying changes using Visual Studio Code and Streamlit on your local computer and deploy to Cognite Data Fusion with the Cognite Toolkit. Local setup Our goal is to detect errors and i

510
P
Practitioner
K
Committed
Khilesh SahuCommitted
Challenges with Data Model creation using CDF Toolkit Examples

I tried to create data models using these examples in the repo using CDF Toolkit https://github.com/cognitedata/toolkit/tree/main/tests/data/describe_data/data_models However, I am not getting the expected result. I got following result as data model type Asset { areaId: Int categoryId: Int createdDate: Timestamp description: String documents: [File] isActive: Boolean isCriticalLine: Boolean metrics: [TimeSeries] parent: JSONObject sourceDb: String tag: String updatedDate: Timestamp children: [Asset] } type WorkOrder { actualHours: Int createdDate: Timestamp description: String dueDate: Timestamp durationHours: Int endTime: Timestamp isActive: Boolean isCancelled: Boolean isCompleted: Boolean isSafetyCritical: Boolean percentageProgress: Int plannedStart: Timestamp priorityDescription: String programNumber: String startTime: Timestamp status: String title: String workOrderNumber: String workPackageNumber: String workItems: [WorkItem] linkedAssets: [Asset] } type WorkItem { criticality:

978
K
Committed
M
Seasoned
Marwen TALEBSeasoned
Delete a field from a view without changing data model version

Hello, We recently decided to not show a field X in one view. X is still in the container, since containers are immutable. We just changed the definition of the view. We did this change without bumping the data model version. This resulted into a surprising outcome: deployment was successful using CDF toolkit but the field is still in the view and can be queried. Only when we incremented the version of the data model the field disappeared. We expect one of these two outcomes: Raising an error in cdf tool kit if change will not be considered only when the version changes. Cdf tool kit does this in some cases. IMHO this should part of these cases. Taking into account the field deletion in the current version Happy to discuss. Thanks

262
M
Seasoned
M
Seasoned
Marwen TALEBSeasoned
Confusing Location support in cdf toolkit

Hello, Based in this doc: https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/resource_library/#locations location is managed by cdf toolkit but it does not appear when deploying in the summary and do not seem to be taken into account. I’m using the latest version of cdf toolkit.Please see screenshots. Thanks

273
Anders Albert
Seasoned Practitioner
J
Seasoned
Jesse JenkenSeasoned
Help Deploying Hosted Extractors using Toolkit v0.3.12

I am successfully deploying a Hosted Extractor, Source, Destination, Mapping and Job using the Cognite Toolkit v0.3.12. The deployment succeeds and the hosted extractor appears in the UI, however it reports the following error when connecting to the source (from the Hosted Extractor Insight UI Tab): Failed to read partitions: Management request failed: StatusError {code: StatusCode(401), description: Some("Attempted to perform an unauthorized operation.") } I have provided the same connectivity information as we have working, in a manually-configured Hosted Extractor. I am only trying to “package” this manually configured one into the toolkit, however there’s no feedback provided in the UI or the deployment suggesting why this is failing, and I need more logs to troubleshoot. When setting up hosted extractors manually I find success pasting the connection string and allowing the UI to parse it into the correct fields. My assumption is the SharedAccessKey is the problem with my deployme

463
Mithila Jayalath
Seasoned Practitioner
E
Active
Eirik RatheActive
Creating edges with cdf toolkit

Hey, does anyone know how to create an edge between two nodes using cdf toolkit? I am specifically looking to figure out: How does the yaml need to be structured does the file need a special name? does the file need to be in a special folder?

775
Anders Albert
Seasoned Practitioner
K
MVP
Kristian NymoenMVP
Groups with id-scoped timeseriesAcl

Hi. I would like to set up a time series scoped iam group - that is scoped with read access to specific time series. For other resources (e.g. data sets), the scope can be set with reference to the external id. But for time series, it fails unless the internal id is used. Is external-id referencing of time series IDs for timeseriesAcl on the roadmap? capabilities: - timeSeriesAcl: actions: - READ scope: idScope: ids: - my.external.id ******* ValueError: invalid literal for int() with base 10: 'my.external.id'

465
Glen Sykes
Seasoned Practitioner
A
Committed
Amol PawarCommitted
Integration & transformation of "Sparkplug B-formatted MQTT data" into Cognite Data Fusion

Hi, I am working on one of the project where I am using MQTT extractor to inject data into CDF. Kindly advise best optimized stepwise solution to integrate and transform data into CDF. Note that it is Sparkplug B-formatted MQTT data. Kindly advise. Regards, Amol S Pawar

716
Jan Inge Bergseth
MVP
Andre Alves
MVP
Andre AlvesMVP
Clarifying the Initial Steps for Implementing Cognite QuickStart with the Cognite Toolkit

I've heard various approaches discussed by my team for starting work with Cognite Data Fusion (CDF), and I’d like to clarify a few of them. I want to begin with the basics: For example, if we are starting with a new client using the Cognite QuickStart requirements (deploying and configuring out-of-the-box solutions like InField, Maintain, and InRobot), and we plan to use the Cognite Toolkit, what should the first step be? a) Start by setting up the Cognite Toolkit using InField built-in modules/examples/infield?b) Start from scratch, setting up Cognite with a new enterprise data model?(We have a lot of questions about this one)c) Start by setting up Cognite with another example from the toolkit repo?d) None of the above. Additionally, if the answer is a, is there any need at this stage to create a specific solution model or a custom domain model/enterprise model, or can we leverage Cognite's core models, given that use cases outside of Maintain and InField have not yet been defined?

793
kelvin
Seasoned Practitioner
Andre Alves
MVP
Andre AlvesMVP
Broken Links in CDF Toolkit Documentation

The links to the modules in the GitHub repository are broken. For instance, the link for cdf_apm_base points to https://github.com/cognitedata/toolkit/blob/release/cognite_toolkit/cognite_modules/core/cdf_apm_base, which no longer exists. The documentation containing these broken links can be found here: https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/packages/cdf_demo_infield.

101
Anders Albert
Seasoned Practitioner
R
Seasoned
Rajendra PasupuletiSeasoned
Ability to download graphql result to excel file on schedule

Hi Team, I want to download data from a CDF Data Model (GraphQL) on schedule. I create a Streamlit App but I can’t download and place the data on local machine based on schedule. Please let me know if there is any solution.

222
R
Seasoned
Anders Albert
Seasoned Practitioner
Anders AlbertSeasoned Practitioner
New STABLE release of Cognite Toolkit, 0.3.0!

A few highlights of the changes from the last stable release. Support for new resources: HostedExtractors, CogniteFile, Sequence, Robotics, 3DModel, Asset, and LocationFilter New command cdf modules. This has the subcommand init, add , list and upgrade, and replaces the former cdf-tk init command. The cdf modules init gives you a guided experience to select new modules. The cdf modules add enables you to your existing setup. The cdf modules upgrade automatically upgrades your modules to match the Toolkit CLI version. The cdf modules list enables you to get an overview over the modules you have. The cdf auth has been reworked into cdf auth init and cdf auth verify. This gives you a more guided experience to setup the required service principal required to run Toolkit. Device login - easier and more secure replacement for interactive login. Toolkit is now also released as a Docker Image: docker hub. New command cdf repo with currently one subcommand cdf repo init. This creates default Gi

320
Anders Albert
Seasoned Practitioner
Anders Albert
Seasoned Practitioner
Anders AlbertSeasoned Practitioner
New STABLE release of Cognite Toolkit, 0.3.0!

A few highlights of the changes from the last stable release. Support for new resources: HostedExtractors, CogniteFile, Sequence, Robotics, 3DModel, Asset, and LocationFilter New command cdf modules. This has the subcommand init, add , list and upgrade, and replaces the former cdf-tk init command. The cdf modules init gives you a guided experience to select new modules. The cdf modules add enables you to your existing setup. The cdf modules upgrade automatically upgrades your modules to match the Toolkit CLI version. The cdf modules list enables you to get an overview over the modules you have. The cdf auth has been reworked into cdf auth init and cdf auth verify. This gives you a more guided experience to setup the required service principal required to run Toolkit. Device login - easier and more secure replacement for interactive login. Toolkit is now also released as a Docker Image: docker hub. New command cdf repo with currently one subcommand cdf repo init. This creates default Gi

190
Anders Albert
Seasoned Practitioner
K
MVP
Kristian NymoenMVP
deploy dry-run with read-only access

Hi. We’re administering CDF deployments from Github using github actions and Cognite toolkit. I’m setting up a github action to automatically perform a dry-run for a pull request to main, and post the dry-run output as a comment on the PR to assist the reviewer. I’d preferably like to use a client with read-only access to CDF for this, but it seems cdf-tk requires full write access even for dry runs - is that so? Performing a dry-run locally with read-only credentials results in: ERROR (AuthorizationError): Don't have correct access rights to deploy iam.groups(all_scoped). Missing:GroupsAcl(actions=[<GroupsAcl Action.Create: 'CREATE'>], scope=AllScope()) -GroupsAcl(actions=[<GroupsAcl Action.Delete: 'DELETE'>], scope=AllScope())Please click here to visit the documentation and ensure that you have setup authentication for the CDF toolkit correctly. I would expect to see the same error on my github-action, but it stops without much useful information: Run cdf-tk deploy --env=dev --dry-ru

321
P
Practitioner
N
Seasoned
N G NidhiSeasoned
how can we deploy cdf functions from azure devops using toolkit

Hi All, I need to deploy CDF Functions from azure devops using cdf toolkit. can you share me any example pipeline files which i can take it as reference. Regards, Nidhi N G

663
P
Practitioner
H
Seasoned
Hugo LopesSeasoned
Disappearing Jupyter Notebooks in Cognite

I recently encountered an issue with Jupyter notebooks stored. Notebooks that were saved inside a dataset folder within the Notebooks directory vanished after about a week of inactivity. Despite using these notebooks on different days, confirming they were indeed stored in Cognite and not just locally on my browser, they still disappeared. One of the notebooks was uploaded to the platform and not created there, but just to be sure, should uploaded notebooks also be permanently stored there?

424
Anders Hafreager
Practitioner
K
MVP
Kristian NymoenMVP
Group capabilities YAML indentation level

Hi. I’m getting the following error: ERROR (ToolkitYAMLFormatError): Failed to load 7.<groupname>.writer.Group.yaml with iam.groups(resource_scoped). Error: CogniteAuthError("Don't have correct access rights. Need READ and WRITE on datasetsAcl.").) I’ve noticed that the “actions”-part of my YAML files have varying indentations. Like below, where the READ action is on a higher level than LIST and READ: - extractionPipelinesAcl: actions: - READ scope: datasetScope: ids: - <removed> - <removed>- groupsAcl: actions: - LIST - READ scope: In the YAML-configuration reference, there are examples of both: - dataModelInstancesAcl: actions: - READ scope: spaceIdScope: { spaceIds: [ 'my_space' ] } <...> - rawAcl: actions: - READ - WRITE scope: tableScope: dbsToTables: my_database: tables: [] Does indentation level matter, and if so, which one is correct?

484
K
MVP

About This Group

This group is for announcements, discussions, and sharing on the Cognite Data Fusion Toolkit, aka cdf-tk, and the project templates that come bundled with it.

The tool is used to create and manage configurations of Cognite Data Fusion projects with authoritative configurations in yaml files. Entities like transformations, extraction pipelines, RAW tables, groups, data models, and more can be managed through the tool. You can either manage configurations from the command line, or embed the tool in your DevOps pipelines supporting development, staging, and production environments.

Documentation can be found here, and the Python-based tool itself is self-documented and can be installed using pip install cognite-toolkit.

Members

Anita Hæhre
Seasoned Practitioner
Madina Kholmatova
Seasoned Practitioner
G
MVP
kelvin
Seasoned Practitioner
Peter Arwanitis
Practitioner
Dilini Fernando
Seasoned Practitioner
Jan Inge Bergseth
MVP
greger
Seasoned Practitioner
P
J
Seasoned Practitioner
Mithila Jayalath
Seasoned Practitioner
Everton Colling
Seasoned Practitioner
dinesh.makked
Seasoned
STEWART WALLACE
Seasoned Practitioner
Øystein Aspøy
Committed
Noah Karsky
Practitioner
Anders Albert
Seasoned Practitioner
P
Practitioner
Anders Hafreager
Practitioner
K
MVP
R
Seasoned
B
Practitioner
S
Committed
N
Seasoned
hescalona
Practitioner
Aditya Kotiyal
MVP
T
Seasoned
M
T
Practitioner
S
Seasoned
+ 51
Terms & ConditionsCookie Settings

Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings