Skip to main content
Closed

Cognite File extractor : Addition of Configuration to control the no. of files to be loaded

Related products:Extractors
  • April 11, 2024
  • 3 replies
  • 52 views
H
Seasoned ⭐️⭐️

I am writing to request a new feature for the Cognite File Extractor tool that would enhance its functionality and usability. Specifically, I propose the addition of a configuration option to control the number of files loaded by the extractor.

As a user of the Cognite File Extractor, I have encountered situations where I need to manage large volumes of files during extraction processes. Currently, the tool loads all files present in the specified directory, which can lead to performance issues and inefficiencies when dealing with a high number of files. Therefore, implementing a feature that allows users to specify the maximum number of files to be loaded in a controlled manner would greatly improve the tool's flexibility and performance.

The proposed configuration option would enable users to define a limit on the number of files extracted, through a numerical input . This would give users greater control over the extraction process, allowing them to tailor it to their specific needs and optimize performance based on their system resources and workflow requirements.

By introducing this configuration control, users can effectively manage resource utilization and streamline their extraction workflows, leading to improved efficiency and productivity. 

 

Also consider adding logging for this feature which will help to debug in case of issues.

 

3 replies

Andreea Oprea
Expert ⭐️⭐️⭐️⭐️
Forum|alt.badge.img
  • Andreea Oprea
  • Expert ⭐️⭐️⭐️⭐️
  • April 12, 2024

Hello @Harsha,


We have received your request and our PM responsible for this area @Thomas Sjølshagen will review the request and come back to you soon if he needs more details.

Thank you for your request and all the details provided.
Andreea.  

 

T
Expert ⭐️⭐️⭐️⭐️
Forum|alt.badge.img

Hi @Harsha, and thanks for this proposed solution. 

Other than volume of files representing a challenge for a timely completion of the extraction process, is there an underlying experience / problem you’ve been having with the extractor that would help me understand how to think about issues like how to know which files to include before the limit is reached, or how to sort the files during an initial load vs subsequent loads if the limit is specified, etc?

Also, some classes of performance relate to how the underlying data store for the source system handles bulk requests, etc, so a more succinct description of the use case leading to having performance challenges would be really helpful to us, thanks!

T
Expert ⭐️⭐️⭐️⭐️
Forum|alt.badge.img

Hi again @Harsha,

Wondering if you’ve been able to think about what I’ve asked above and perhaps can help me clarify beyond the fact that you’re seeing performance issues when the file count is high? 

For instance, are the performance that extracting an individual file is slow once the list of files to update/load are over a certain number (any thoughts on where that boundary is for your configuration?), or overall time-to-completion increases disproportionately to the increase of files to extract once the list of files-to-load exceeds a certain (rough) threshold?

Terms & ConditionsAccessibility statement