+2To ensure data quality we want to run a filtering algorithm on certain timeseries that are prone to faulty spikes. Typically we want to replace a datapoint that is more than rollig.avg +/-3rolling std.dev with interpolated values.
as part of this, it would also be useful to track the changes on the actual datapoints. Is there a functionality in CDF for this? Ideally, there could be some metadata attached to the datapoint, describing e.g. the original value, when it was changes, etc.
Hi
Thank you for suggesting this product idea. We will follow the traction this idea gets in the community. You can expect updates on this post if we decide to include this in our future roadmap, or if we require more information.
Hi
Spike filtering can be implemented as a Cognite Function. You deploy Python code that runs on a schedule, reads your time series, applies the rolling avg ± 3σ logic, replaces the spike with an interpolated value, and writes the cleaned datapoint back to CDF.
Datapoint-level annotation is supported via OPC UA status codes. When writing back the cleaned value, you attach a status code to it. Something like UncertainSubstituteValue (code 1083244544) or GoodEdited (code 14417920) are good fits for the use case you're describing. Then downstream consumers and visualizations can use to handle it differently.
The API also exposes quality-aware aggregates, like countGood, countUncertain, countBad, so you can track how many datapoints in a time range were flagged. Charts in CDF already shade uncertain data and show gaps for bad data, which gives some visual traceability out of the box.
Tracking changes can be done via Data Point Subscriptions. Subscriptions give you a changelog of all writes to a set of time series, including out-of-order updates, so you can track exactly what changed and when.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
