Skip to main content

Is there any way to read CSV files in an incremental manner from a drop folder following a file name pattern and ingest the data into specific RAW tables.

Eg. I have file names which have timeseries data from daily report with columns like asset id, timestamp, data point. With file name pattern as “timeseries_<unique number/date>.csv”. Users are pasting these files in a public sharepoint location in an incremental manner. 

Hi,

If I understand correctly your question, you want to extract data from CSV files (which have a specific name pattern) that are added incrementally to a folder, and create/update timeseries from those files. 

What I would recommend here is to build a CSV extractor using our “extractor-utils” package:

https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/index.html

In the docs there is a guide to build a CSV extractor. Of course you would have to change some logic inside of it, but I think this package has everything you need: reading CSV, upload queue, state store etc. 

I hope that was useful. If you have other questions, please let us know !

 


Hi,

If I understand correctly your question, you want to extract data from CSV files (which have a specific name pattern) that are added incrementally to a folder, and create/update timeseries from those files. 

What I would recommend here is to build a CSV extractor using our “extractor-utils” package:

https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/index.html

In the docs there is a guide to build a CSV extractor. Of course you would have to change some logic inside of it, but I think this package has everything you need: reading CSV, upload queue, state store etc. 

I hope that was useful. If you have other questions, please let us know !

 

Thanks @Pierre Pernot , I will go through the doc!


Just in case, you can find some guides on specific topics here:
https://hub.cognite.com/how-to-guides-225

It could be useful if you want to run your extractor with Docker, or deploy other kinds of extractors for example.


Reply