Skip to main content
Answer

CDF Toolkit Instances deployment using csv file

  • February 6, 2025
  • 9 replies
  • 117 views

Ayush Daruka
Seasoned

Can we use a CSV file instead of a Yaml file containing details of the instances we want to populate in a view using CDF toolkit? If yes, is there a limit to it? 
 

Edit:  What I mean to ask is whether we can use a csv file directly to populate a view in a data model, i.e. without using raw and transformations? Just like we can use a node.yaml file to directly deploy an instance in a view.

Best answer by Anders Albert

 ​@Ayush Daruka We have release Toolkit `v0.4.7` which has alpha support for populating nodes through a view from a csv or parquet file. 

You enable it in the `cdf.toml` with the following

[alpha_flags]
populate=true

The command is `cdf populate view`. 

9 replies

Ayush Daruka
Seasoned
  • Author
  • Seasoned
  • February 6, 2025

@Snehal-Jagtap 


Forum|alt.badge.img+4

We have some of our instance data in csv files, and it works well. Our files are structured like this;

key name description aliases source ...
external id 1 name 1 ... ... ... ...
external id 2 name 2 ... ... ... ...

The files are deployed to raw via cognite toolkit and then instances are populated with transformations/workflows. 

None of our CSV-files are very large though. I think the largest ones are approximately 300 rows.


Ayush Daruka
Seasoned
  • Author
  • Seasoned
  • February 6, 2025

We have some of our instance data in csv files, and it works well. Our files are structured like this;

key name description aliases source ...
external id 1 name 1 ... ... ... ...
external id 2 name 2 ... ... ... ...

The files are deployed to raw via cognite toolkit and then instances are populated with transformations/workflows. 

None of our CSV-files are very large though. I think the largest ones are approximately 300 rows.

Thanks for the information. Have you also tried using the csv directly to populate a view just like we can with a node.yaml file?


Forum|alt.badge.img+4

No, our node.yaml files only contain node types, no instances.


Anders  Albert
Seasoned Practitioner
Forum|alt.badge.img
  • Seasoned Practitioner
  • February 11, 2025

@Ayush Daruka as ​@Kristian Nymoen is saying only a YAML file is supported and it is intended for either node types or configuration nodes. For populating large amount of nodes transformations/or custom script/an extractor are the only options.

Note I have logged this as a feature request for Toolkit.


Ayush Daruka
Seasoned
  • Author
  • Seasoned
  • February 11, 2025

@Ayush Daruka as ​@Kristian Nymoen is saying only a YAML file is supported and it is intended for either node types or configuration nodes. For populating large amount of nodes transformations/or custom script/an extractor are the only options.

Note I have logged this as a feature request for Toolkit.

 

@Anders Albert Thanks for confirming!!!


Anders  Albert
Seasoned Practitioner
Forum|alt.badge.img
  • Seasoned Practitioner
  • Answer
  • February 13, 2025

 ​@Ayush Daruka We have release Toolkit `v0.4.7` which has alpha support for populating nodes through a view from a csv or parquet file. 

You enable it in the `cdf.toml` with the following

[alpha_flags]
populate=true

The command is `cdf populate view`. 


Ayush Daruka
Seasoned
  • Author
  • Seasoned
  • February 13, 2025

@Anders Albert Thanks for sharing this. Let me try it out and get back if I have any questions.


Ayush Daruka
Seasoned
  • Author
  • Seasoned
  • February 13, 2025

 ​@Ayush Daruka We have release Toolkit `v0.4.7` which has alpha support for populating nodes through a view from a csv or parquet file. 

You enable it in the `cdf.toml` with the following

[alpha_flags]
populate=true

The command is `cdf populate view`. 

@Snehal-Jagtap ​@Neerajkumar Bhatewara