Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
On the PowerOps project we use simulation models to determine the behaviour of hydro-power plants given time-series values of parameters such as inflow, power prices and water discharge. All of these time-series contain both historical, measured values and forecasts for the future. The forecast part of these time-series changes often (multiple times per day). The problem is that we sometimes want to run our physical models using the time-series values as they were at an arbitrary point in time. In order retain access to outdated forecasts we currently have to duplicate our time-series every time we add a new data-point. However, this is a very inefficient and ugly solution as our project now has 8+ million time-series and counting. It would significantly simplify our work if versioning of data-points was added to CDF.
Specifically, what I want is for us to change CDF data-points from the form “(timestamp, value)” to the form “(timestamp, value, created_time, deleted_time)”. Any time a datapoint is added, the created_time of that datapoint would be set to the current time in millis. Then whenever a datapoint is deleted, instead of removing it completely, we simply set deleted_time equal to the current time. This allows for functions such as:
client.datapoints.retrieve(external_id=..., start="2w-ago", end="now", time="1w-ago")Where we with the “time” parameter set that we want to fetch the datapoints as they were 1 week ago. The function would then return a list of data-points as such:
# This is just inefficient pseudo-code
return [dp for dp in datapoints if dp.created_time <= time < dp.deleted_time]
The ability to fetch time-series as they were in a specific point in time can be really useful for a bunch of other use-cases than our specific example. For example:
- Machine learning - When training a machine learning model to create forecasts it is important to train it only on data you which you will have available at the time of forecasting. With versioned data-points it is simple to find which data was available at a previous point in time.
- Data-replication - Currently, if you want to continually replicate time-series data from one project to another project you have to copy over every data-point in every time-series each time as there is no way to know which data-points have been added since you last replicated the data. This is not feasible with 100k+ time-series in a project. However, if we had versioned data-points then you could simply check which data-points have been created since you last did replication and copy over only the new data-points.
- Analysing performance of forecasts
