How to Extract All Data Available from SAP System Using OData API Extraction Process [Cognite Official]

Overview

When extracting data from an SAP system using the OData API extraction process, you may encounter situations where not all available data is being retrieved. This issue can be resolved by utilizing server-side or client-side pagination, depending on the capabilities of your SAP endpoint.

Solution

1. Server-Side Pagination

From SAP Extractor version 2.2.0 onward, server-side pagination is supported for full data loads using the OData $skiptoken parameter. This approach allows the extraction process to handle large data sets more efficiently by managing pagination on the server side.

When to Use:

• Your SAP OData endpoint supports the $skiptoken parameter.

Sample Configuration:

# full load - server pagination 
- source_name: s4h_dev 
  name: equipment 
  destination: 
    type: raw 
    database: test_sap_extractor 
    table: equipment_full_server 

  sap_service: API_EQUIPMENT 
  sap_entity: Equipment 
  sap_key: [Equipment] 

  pagination_type: server 

  schedule: 
    type: interval 
    expression: 1m

For more information on how to implement $skiptoken, please refer to the OData documentation.

2. Client-Side Pagination

If your SAP OData endpoint does not support the $skiptoken parameter, you can use client-side pagination to handle the full data load. This method involves managing pagination on the client side to ensure all available data is extracted.

When to Use:

• Your SAP OData endpoint does not support the $skiptoken parameter.

Sample Configuration:

# full load - client pagination 
- source_name: s4h_dev 
  name: equipment 
  destination: 
    type: raw 
    database: test_sap_extractor 
    table: equipment_full_client 

  sap_service: API_EQUIPMENT 
  sap_entity: Equipment 
  sap_key: [Equipment] 

  pagination_type: client 

  schedule: 
    type: interval 
    expression: 1m

If the number of available records is beyond 50,000 it will just pause to upload and continue. By choosing the appropriate pagination method based on your SAP OData endpoint's capabilities, you can ensure that all available data is successfully extracted from the SAP system.