Skip to main content
Solved

Parallelize restAPI calls with pagination


Forum|alt.badge.img+3

Hello, I am trying to use “Filter time series” or “List time series” endpoints. However, because of 1000 limit, I need to parallelize these calls as I would like to retrieve thousands of timeseries. The challenge I am facing is the pagination used in these endpoints which is based on id of timeseries. Is there a way to use index as pagination? 

Best answer by Jason Dressel

@Diana Chimnaz Johan 

client: CogniteClient = ...

print(f"Max Workers: {client.config.max_workers}") #ensure you have workers >1 :)

timeseries = client.time_series.list(limit=None) #not making use of the workers

timeseries = client.time_series.list(partitions=10, limit=None) #will leverage the workers and run in parallel

-Jason

View original
Did this topic help you find an answer to your question?

5 replies

Glen Sykes
Seasoned Practitioner
  • Seasoned Practitioner
  • 123 replies
  • August 16, 2023

Hi Diana, have you tried using the parallel retrieval method, using the /partition parameters?

Read here for more information 

 


Forum|alt.badge.img+3

Hi Glen, thank you for getting back to me. Can you tell me more about the parallel retrieval method? It sounds like you may have shared a link, but I do not see it. 


Glen Sykes
Seasoned Practitioner
  • Seasoned Practitioner
  • 123 replies
  • August 16, 2023

Hi Diane, sorry about that.  Hopefully this link works!

https://developer.cognite.com/dev/concepts/pagination/


Forum|alt.badge.img
  • Seasoned Practitioner
  • 282 replies
  • Answer
  • August 18, 2023

@Diana Chimnaz Johan 

client: CogniteClient = ...

print(f"Max Workers: {client.config.max_workers}") #ensure you have workers >1 :)

timeseries = client.time_series.list(limit=None) #not making use of the workers

timeseries = client.time_series.list(partitions=10, limit=None) #will leverage the workers and run in parallel

-Jason


  • Seasoned Practitioner
  • 223 replies
  • August 23, 2023

@Diana Chimnaz Johan did the above help you? 


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings