Skip to main content
Solved

Is it possible to iterate over a chunk for more than 3 hours?

  • 24 July 2024
  • 3 replies
  • 44 views

A fictitious example, but one that reproduces my problem, is the following code:

 

for event_list in client.events(chunk_size = 500_000, data_set_ids = ids_sites):

        events = event_list.to_pandas()

 

Error message after just over an hour: CogniteAPIError: Unauthorized | code: 401 | X-Request-ID: 9c8702fc-59bf-9f7a-be47-4876d6b433f3

 

In fact, if I iterate in another way (for example, filtering by date), I can run code for more than 8 hours (for as long as I want, actually, because I re-authenticate between iterations, preventing the token from expiring). However, I don't want to filter by date because it doesn't provide consistent data volume like iterating by chunks.

And apparently, I can't authenticate between iterations in the example I provided here, as it seems the chunk persists the initial authentication, which initially makes sense. Does anyone know how to solve this?

 

 

 

3 replies

Userlevel 4
Badge

Hi, and thanks for the bug report.

This should have been fixed some time ago, in version 6.3.2. Can you double-check what version you are using?

If you are running on a more recent version and still experience this, please create an issue on Github, if you would be so kind. Then it’s easier for us to track and we can get fixing this prioritized!

Please also describe how you authenticate, and any non-default settings you are using.

Userlevel 4
Badge +2

Hi @Erivan Jose de Oliveira Junior,

We are following up to see whether you're satisfied with the responses you've received? 

Userlevel 4
Badge

Issue was resolved (afaik): https://github.com/cognitedata/cognite-sdk-python/issues/1862

Reply