Now the Python SDK supporting the cursor parameter or not?
Best answer by roman.chesnokov 24 April 2023, 10:58
Hi Shah, the Python SDK only supports the cursor parameter to list the transformation schedules, but there are different approaches that you can implement if you need pagination. For example, for datapoints you can use timestamps for pagination. You can also read some additional info about the performance of datapoints retrieval here.
Actually, I am trying to retrieve events but there is more than thousands of events and it’s taking more time to give response so i want to do pagination and looking for cursor value using python sdk. Could you please help me?
For events specifically you can use that idea:
Iterate over chunks of events to reduce memory load:
from cognite.client import CogniteClientc = CogniteClient()for event_list in c.events(chunk_size=2500): event_list # do something with the events
You can also make batches based on last_updated_time, start_time or end_time and use the timestamp ranges in events.list() method.
Another approach is to iterate over assets and get events for each asset.
Let me know if it was helpful.
No this is not working i tried already so now based on our requirement i need cursor value we want to do pagination.
The first approach uses a cursor under the hood, it uses a generator and makes an API call for each chunk. Could you explain to me the requirements and why you need a cursor specifically?
Example of time for each API call for different chunk_size:
t1 = datetime.now()for event_list in client.events(chunk_size=2500): print(len(event_list)) t2 = datetime.now() print(t2 - t1) t1 = t225000:00:02.74440425000:00:01.41413625000:00:02.49859325000:00:01.86544425000:00:02.715027
t1 = datetime.now()for event_list in client.events(chunk_size=250): print(len(event_list)) t2 = datetime.now() print(t2 - t1) t1 = t22500:00:00.3688912500:00:00.2748052500:00:00.2851392500:00:00.3328112500:00:00.309854
the approach of using client.events(chunk_size=250) will not work in our case, because we want to use pagination in our backend to deliver data to our frontend, so we need to know what the nextCursor value is to be able to query for the next chunk if required.
Is there a way to get the cursor value using the SDK ?
If not is Cognite planning to upgrade the SDK to allow the user to work with the cursor value ?
Hey Janos, the client.events() method is basically a generator. So you can use something like that for pagination:
events_provider = client.events(chunk_size=250)next_chunck = next(events_provider)
Then next_chunck is a list of the next 250 events. And request to API happens when you call the next function. So it uses a cursor, but you don’t need to handle it by yourself. Do you think it will work for you?
Unfortunately it isn’t.
Consider this scenario: Our UI requests using our backend to list the first 30 Events. Backend using Cognite Python SDK queries the first 30 events from Cognite and returns them to out UI. User wants to see the next page the UI requests the next page from the Backend, but since the backend is stateless will not know how to retrieve the next 30 events from CDF.
This is why we need to have the cursor so that when the UI requests the new page the cursor is there to identify the next page.
It is the same as what the Fusion UI does when listing events on the “Explore” page
I see; then probably the best way is to use our REST API directly. You can find the docs here. As far as I know, there are no plans for implementing cursor usage by users via SDK.
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.