+3
Is there any way to read CSV files in an incremental manner from a drop folder following a file name pattern and ingest the data into specific RAW tables.
Eg. I have file names which have timeseries data from daily report with columns like asset id, timestamp, data point. With file name pattern as “timeseries_<unique number/date>.csv”. Users are pasting these files in a public sharepoint location in an incremental manner.
Best answer by Pierre Pernot
View original
Hi,
If I understand correctly your question, you want to extract data from CSV files (which have a specific name pattern) that are added incrementally to a folder, and create/update timeseries from those files.
What I would recommend here is to build a CSV extractor using our “extractor-utils” package:
https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/index.html
In the docs there is a guide to build a CSV extractor. Of course you would have to change some logic inside of it, but I think this package has everything you need: reading CSV, upload queue, state store etc.
I hope that was useful. If you have other questions, please let us know !
+3
Hi,
If I understand correctly your question, you want to extract data from CSV files (which have a specific name pattern) that are added incrementally to a folder, and create/update timeseries from those files.
What I would recommend here is to build a CSV extractor using our “extractor-utils” package:
https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/index.html
In the docs there is a guide to build a CSV extractor. Of course you would have to change some logic inside of it, but I think this package has everything you need: reading CSV, upload queue, state store etc.
I hope that was useful. If you have other questions, please let us know !
Thanks
Just in case, you can find some guides on specific topics here:
https://hub.cognite.com/how-to-guides-225
It could be useful if you want to run your extractor with Docker, or deploy other kinds of extractors for example.
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support