Parallelize restAPI calls with pagination

Forum|Forum|2 years ago
August 15, 2023
5 replies
131 views

Hello, I am trying to use “Filter time series” or “List time series” endpoints. However, because of 1000 limit, I need to parallelize these calls as I would like to retrieve thousands of timeseries. The challenge I am facing is the pagination used in these endpoints which is based on id of timeseries. Is there a way to use index as pagination?

Best answer by Jason Dressel

@Diana Chimnaz Johan

client: CogniteClient = ...

print(f"Max Workers: {client.config.max_workers}") #ensure you have workers >1 :)

timeseries = client.time_series.list(limit=None) #not making use of the workers

timeseries = client.time_series.list(partitions=10, limit=None) #will leverage the workers and run in parallel

-Jason

Glen Sykes

Expert ⭐️⭐️⭐️⭐️

Hi Diana, have you tried using the parallel retrieval method, using the /partition parameters?

Read here for more information

Glen | Cognite Product Management

D

Diana Chimnaz JohanAuthor

Seasoned ⭐️⭐️⭐️

Hi Glen, thank you for getting back to me. Can you tell me more about the parallel retrieval method? It sounds like you may have shared a link, but I do not see it.

Glen Sykes

Expert ⭐️⭐️⭐️⭐️

Hi Diane, sorry about that. Hopefully this link works!

https://developer.cognite.com/dev/concepts/pagination/

Glen | Cognite Product Management

J

Jason DresselAnswer

Expert ⭐️⭐️⭐️⭐️

@Diana Chimnaz Johan

client: CogniteClient = ...

print(f"Max Workers: {client.config.max_workers}") #ensure you have workers >1 :)

timeseries = client.time_series.list(limit=None) #not making use of the workers

timeseries = client.time_series.list(partitions=10, limit=None) #will leverage the workers and run in parallel

-Jason

C

Carin Meems

Expert ⭐️⭐️⭐️⭐️

@Diana Chimnaz Johan did the above help you?

Sign up

Welcome to Cognite Hub