Skip to main content
Gathering Interest

Cognite Data Fusion: Missing values in a Time Series is reflected in Synthetic Time Series

Related products:API and SDKs
  • January 21, 2022
  • 9 replies
  • 189 views
  • Anita Hæhre
    Anita Hæhre
  • Morten
    Morten
  • Crystal Connor Richards
  • Christoffer Lange
  • Henrik Hogheim Apeland

Problem: If one or more of the source-time series are missing data-points they will also be missing in the synthetic time series output.

Would it be possible to add an option to supply a default “not available” value for missing data-points which could then be used by the API to calculate a partial result?

Source: 

 

9 replies

Anita Hæhre
Seasoned Practitioner
Forum|alt.badge.img+1
  • Head of Academy and Community
  • 590 replies
  • February 4, 2022
Updated idea statusNewGathering Interest

Marcela Young
MVP

@Andreea Pastinaru @Hunter Beck what is the status of this one?


Glen Sykes
Seasoned Practitioner
  • Seasoned Practitioner
  • 123 replies
  • November 8, 2023

Hi Marcela,

This one is with me as the PM for Time Series.  We have begun work on the API design to implement support for Status Codes, using the OPC-UA standard.

This will allow data points to be marked as Good, Bad or Uncertain.

As part of this work, we will also be changing the behaviour of our extractors, so instead of dropping bad data points, we will instead store them with their appropriate status code and we will offer a choice of pre-calculated aggregates based on what to do with these status codes.

It will also be possible to choose not to use Bad data points (or Bad and Uncertain) in the generation of a Synthetic Time Series.

We anticipate that we will have a beta release in the early part of Q1 2024 for this.  I’d be happy to include you in the development feedback process if you are interested to participate?

Kind Regards, Glen


Hi @Glen Sykes ,

Would this new API feature help solving issues with synthetic timeseries where some datapoints are missing in one or more timeseries, as described in the source? 


Glen Sykes
Seasoned Practitioner
  • Seasoned Practitioner
  • 123 replies
  • November 9, 2023
Marie Solvik Lepoutre wrote:

Hi @Glen Sykes ,

Would this new API feature help solving issues with synthetic timeseries where some datapoints are missing in one or more timeseries, as described in the source? 

Hi @Marie Solvik Lepoutre,

So with the new feature, there will no longer be gaps in the time series where the extractor drops bad data points from the source.  Instead, there will be legitimate values, but with a ‘bad’ status indicator.   

So the request here is:  If summing two time series where some data points are bad, you are asking for us to treat the bad data point as if it were ‘zero’, rather than ‘not a number’, and so the synthetic time series would add ‘zero’ to the corresponding data point and produce a number, rather than an ‘undefined’ result.

Have I stated the request correctly?

 


Marcela Young
MVP

@Glen Sykes please keep myself as well as the following informed: @crgomez13 , @ibrahim.alsyed , and @rsiddha 


Marcela Young
MVP
Glen Sykes wrote:
Marie Solvik Lepoutre wrote:

Hi @Glen Sykes ,

Would this new API feature help solving issues with synthetic timeseries where some datapoints are missing in one or more timeseries, as described in the source? 

Hi @Marie Solvik Lepoutre,

So with the new feature, there will no longer be gaps in the time series where the extractor drops bad data points from the source.  Instead, there will be legitimate values, but with a ‘bad’ status indicator.   

So the request here is:  If summing two time series where some data points are bad, you are asking for us to treat the bad data point as if it were ‘zero’, rather than ‘not a number’, and so the synthetic time series would add ‘zero’ to the corresponding data point and produce a number, rather than an ‘undefined’ result.

Have I stated the request correctly?

 

This would help some issues we’ve seen users have with calculations that pull from sources with bad indicators - their assumption would have been that it is treated as zero rather than creating an error message and not computing or having to get rid of the bad source altogether.


Glen Sykes
Seasoned Practitioner
  • Seasoned Practitioner
  • 123 replies
  • November 30, 2023

Hi Marcela,

I will send you a direct message as I would like to share our draft API specification with you for feedback.

Kind Regards, Glen


@Glen Sykes Are there any updates on this? 

As Ola is writing here, we would also like to be able to do a calculation on multiple timeseries where some are not overlapping by padding the timeseries with 0-values. I’m not sure if this feature request will solve that, or if you have another feature request for padding? 


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings