Solved

Problem with ingesting datapoint immediately after Timeseries object creation with multiple concurrent jobs.

  • 3 November 2023
  • 4 replies
  • 62 views
D
Rank

We’re attempting to execute 10 concurrent jobs, and each one will create approximately 20 Timeseries objects, immediately ingesting datapoints for these newly created Timeseries objects in a loop.

However, if we try to run 500 similar jobs with 10 concurrently running at a time, about 10% of the jobs fail due to a ‘Timeseries not found’ error.

We theorize that the creation of a Timeseries object can take some time. Therefore, immediately ingesting the datapoint might fail because Cognite doesn’t refresh quickly enough to find the corresponding Timeseries object. This issue only occurs when we attempt to run multiple concurrent jobs.

Has anyone else encountered similar issues?

 

icon

Best answer by Dilini Fernando 30 November 2023, 09:32

View original

4 replies

J
Rank
Userlevel 4
Badge

@Dexter Nguyen 

Is there an potential race condition here?  Would a thread try to upload datapoints before the timeseries is actually created?  Given that you only observe this behavior during concurrent jobs, this seems likely.  Is is possible to have 2 jobs types: First, create all the necessary timeseries objects. You can POST to create 1000 time series objects in a single request.  Second, post to create the datapoints?

-Jason

Glen Sykes
Rank
Userlevel 3

Thanks Jason, I was just about to add:

Did you verify that you got a 201 response upon creation of the Time Series before attempting to ingest the data?  It could be that you’re attempting to write the data to the Time Series before it’s fully created, or if creation failed for some reason.

We do have integration tests doing this operation all of the time, running concurrently without issue.

Glen | Cognite Product Management
Dilini Fernando
Rank
Userlevel 4
Badge +2

Hi @Dexter Nguyen,

We are following up to see whether you're satisfied with the responses you've received.

Best regards,
Dilini 

Dilini Fernando
Rank
Userlevel 4
Badge +2

Hi @Dexter Nguyen,

I hope the above helped. I’m closing this topic now, please feel free to create a new post if you still experience problems. 

Br,

Dilini

Reply


Terms & ConditionsCookie Settings