Skip to main content
Closed

Queue Management for Events/Assets API

Related products:API and SDKs
  • September 25, 2024
  • 2 replies
  • 25 views
H
Committed

Proposal  for queue management to handle bursts of requests during bulk updates in source systems, 

 

Background:
Users performing bulk updates in source systems often encounter 409 (Conflict) and 429 (Too Many Requests) errors when using the Cognite Events/Assets API. This indicates a need for improved handling of request bursts during these operations.

Request:
Implement a queue management system for the Events/Assets API to efficiently handle bursts of requests during bulk updates from source systems.

Proposed Features

1. Request queueing mechanism for bulk operations
2. Intelligent prioritization of queued requests in sequence
3. Automatic retry logic with configurable backoff strategy
 

Expected Benefits:

1. Reduced occurrence of 409 and 429 errors during bulk updates
2. Improved data consistency between source systems and Cognite Data Fusion
3. Enhanced performance and reliability for large-scale data synchronization
4. Simplified error handling for API users during bulk operations

 

This enhancement would significantly improve the reliability and efficiency of bulk update operations from source systems to Cognite Data Fusion, addressing the specific challenges posed by request bursts and reducing the occurrence of 409 and 429 errors.

 

2 replies

T
Seasoned Practitioner
Forum|alt.badge.img

​@Harsha,

Both of these statuses are the result of intentional limiting to protect the services. I will forward and highlight your idea to ​@Elka Sierra, the product manager responsible for these services.

    Elka Sierra
    MVP
    Forum|alt.badge.img

    Hi ​@Harsha, thank you for your request.

    After reviewing your usage pattern, it seems the 429 rate is primarily due to bulk updates where minimal data changes. This approach, while straightforward, can lead to inefficiencies, increased costs, and slower performance. We recommend optimizing updates to send only necessary changes, which will improve performance and reduce costs.

    For bulk updates, you might consider implementing a queuing system on your side to better manage the data flow. However, addressing this directly at the API level doesn't align with our current priorities, particularly for APIs like Assets and Events.

    Thanks and regards

    Elka

      Terms & ConditionsCookie SettingsAccessibility statement