Cognite Sap extractor : Splitting large data ranges inputs

Thank you for sharing one more your insightful reflection, Harsha! 

We greatly appreciate your contribution and stay tuned for updates on how we plan to proceed.

Dear @Harsha 

I appreciate your feature request and your continued engagement with our platform. Currently, our primary focus is on ensuring the scalability of our product, particularly with regards to SAP writeback functionalities. our request for enhancements to the SAP extractor has been duly noted, and we are adding it to our backlog for future consideration.

Thank you for your support and understanding in this matter.

Best regards,
Elka

Hi, I am not sure if Planned for development is the most appropriate status for things that are in the backlog? We currently don’t review the “Planned for development” items with Cognite product teams, I feel there is a chance something important in backlog - can lose track if tagged as “Planned for development”, without a defined timeline for implementation. 

Hi @Harsha @rsiddha 

The SAP extractor extracts data in two different “modes”: full load or incremental load. This is handled by the “incremental_field” configuration parameter. When using the incremental load, the extractor will extract the data in chunks from SAP OData, and the chunk size is also configurable (see chunk_size parameter here). Hence, when running incremental loads, the extractor splits the data in minor requests towards SAP Gateway (avoiding errors like “Request is too big”).

Can you please test this functionality and let us know your feedback?

Thanks and Regards,

Rogerio Junior
Cognite Engineering