Solved

Query for pagination

S
Rank

Hello Team,

 

Now the Python SDK supporting the cursor parameter or not?

 

Regards,

Ayushi.

icon

Best answer by roman.chesnokov 24 April 2023, 10:58

View original

10 replies

roman.chesnokov
Rank
Userlevel 3

Hi Shah, the Python SDK only supports the cursor parameter to list the transformation schedules, but there are different approaches that you can implement if you need pagination. For example, for datapoints you can use timestamps for pagination. You can also read some additional info about the performance of datapoints retrieval here.

S
Rank

Hello Roman,

Actually, I am trying to retrieve events but there is more than thousands of events and it’s taking more time to give response so i want to do pagination and looking for cursor value using python sdk. Could you please help me?

roman.chesnokov
Rank
Userlevel 3

Hello Shah, 

 

For events specifically you can use that idea:

Iterate over chunks of events to reduce memory load:


from cognite.client import CogniteClient
c = CogniteClient()
for event_list in c.events(chunk_size=2500):
    event_list # do something with the events

You can also make batches based on last_updated_time, start_time or end_time and use the timestamp ranges in events.list() method. 

Another approach is to iterate over assets and get events for each asset.

Let me know if it was helpful.

 

S
Rank

Hello Roman,

No this is not working i tried already so now based on our requirement i need cursor value we want to do pagination.

roman.chesnokov
Rank
Userlevel 3

The first approach uses a cursor under the hood, it uses a generator and makes an API call for each chunk. Could you explain to me the requirements and why you need a cursor specifically? 

roman.chesnokov
Rank
Userlevel 3

Example of time for each API call for different chunk_size:

t1 = datetime.now()
for event_list in client.events(chunk_size=2500):
    print(len(event_list))
    t2 = datetime.now()
    print(t2 - t1)
    t1 = t2

2500
0:00:02.744404
2500
0:00:01.414136
2500
0:00:02.498593
2500
0:00:01.865444
2500
0:00:02.715027

 

t1 = datetime.now()
for event_list in client.events(chunk_size=250):
    print(len(event_list))
    t2 = datetime.now()
    print(t2 - t1)
    t1 = t2

250
0:00:00.368891
250
0:00:00.274805
250
0:00:00.285139
250
0:00:00.332811
250
0:00:00.309854

 

J
Badge

Hi Roman,

the approach of using client.events(chunk_size=250) will not work in our case, because we want to use pagination in our backend to deliver data to our frontend, so we need to know what the nextCursor value is to be able to query for the next chunk if required.

Is there a way to get the cursor value using the SDK ?

If not is Cognite planning to upgrade the SDK to allow the user to work with the cursor value ?

 

roman.chesnokov
Rank
Userlevel 3

Hey Janos, the client.events() method is basically a generator. So you can use something like that for pagination:

events_provider = client.events(chunk_size=250)
next_chunck = next(events_provider)

Then next_chunck is a list of the next 250 events. And request to API happens when you call the next function. So it uses a cursor, but you don’t need to handle it by yourself. Do you think it will work for you? 

J
Badge

Unfortunately it isn’t. 

Consider this scenario: Our UI requests using our backend to list the first 30 Events. Backend using Cognite Python SDK queries the first 30 events from Cognite and returns them to out UI. User wants to see the next page the UI requests the next page from the Backend, but since the backend is stateless will not know how to retrieve the next 30 events from CDF.

This is why we need to have the cursor so that when the UI requests the new page the cursor is there to identify the next page.

 

It is the same as what the Fusion UI does when listing events on the “Explore” page

roman.chesnokov
Rank
Userlevel 3

I see; then probably the best way is to use our REST API directly. You can find the docs here. As far as I know, there are no plans for implementing cursor usage by users via SDK. 

Reply


Terms & ConditionsCookie Settings