Important updates to documentation and limits on the use of the Parallel Retrieval method

Hello CDF developers and users,

Following feedback about the best practice examples provided in the documentation for how to use Parallel Retrieval / Partitions, Cognite has updated its documentation to make more explicit the limits in place for how many partitions that can be used in a request.  The document now makes explicit that a maximum of 10 partitions is supported.

This guidance is found in the Pagination section of the Developer Guide

In conjunction with the updated guidance, limit enforcement will be put in place on the Assets, Events and Files APIs, to ensure predictability of performance with these APIs, effective from 1st February 2023.  Requests calling for more than 10 partitions will return a HTTP 422 ‘Unprocessable_Entity’ error message.  Similar limits will be rolled out across all CDF backend APIs, and details of these limits will be provided in their respective resource type descriptions.

For any existing client applications that are calling more than 10 partitions, the implementation of limits on the 1st February will not impact requests from these clients.  Instead, a background process will gracefully reformat the requests to conform to the limits.  Developers are requested however to refactor their applications to conform to the new limits at their earliest convenience and preferably before 31st March 2024 (when an internal review of consumption patterns will happen and potential tightening up of how the limits should be applied).

As a final note, please rest assured that Cognite will not implement limits in such a way that would break existing app integration, and we value your feedback on this topic.
 

Summary

• The partition parameter, used for parallel retrieval of large volumes of data will have an enforced maximum of 10 partitions per request for the Assets, Events and Files APIs, effective from the 1st February 2023.  

• In CDF projects created after 1st February, requests that include more than 10 partitions will return a 422: Unprocessable_Entity error.

• In existing CDF projects, requests that include more than 10 partitions will be gracefully and transparently managed to use 5 partitions.  This is a robust non-breaking change that has demonstrably improved performance (and has been running for several months), and customers are encouraged to update their integrations to conform to the best practice recommendations to enjoy further performance benefits.