Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1354 Ideas

Edvard HolenPractitioner ⭐️⭐️⭐️

Cognite Data Fusion: Versioning of datapointsGathering Interest

On the PowerOps project we use simulation models to determine the behaviour of hydro-power plants given time-series values of parameters such as inflow, power prices and water discharge. All of these time-series contain both historical, measured values and forecasts for the future. The forecast part of these time-series changes often (multiple times per day). The problem is that we sometimes want to run our physical models using the time-series values as they were at an arbitrary point in time. In order retain access to outdated forecasts we currently have to duplicate our time-series every time we add a new data-point. However, this is a very inefficient and ugly solution as our project now has 8+ million time-series and counting. It would significantly simplify our work if versioning of data-points was added to CDF. Specifically, what I want is for us to change CDF data-points from the form “(timestamp, value)” to the form “(timestamp, value, created_time, deleted_time)”. Any time a datapoint is added, the created_time of that datapoint would be set to the current time in millis. Then whenever a datapoint is deleted, instead of removing it completely, we simply set deleted_time equal to the current time. This allows for functions such as:client.datapoints.retrieve(external_id=..., start="2w-ago", end="now", time="1w-ago")Where we with the “time” parameter set that we want to fetch the datapoints as they were 1 week ago. The function would then return a list of data-points as such:# This is just inefficient pseudo-codereturn [dp for dp in datapoints if dp.created_time <= time < dp.deleted_time] The ability to fetch time-series as they were in a specific point in time can be really useful for a bunch of other use-cases than our specific example. For example: Machine learning - When training a machine learning model to create forecasts it is important to train it only on data you which you will have available at the time of forecasting. With versioned data-points it is simple to find which data was available at a previous point in time. Data-replication - Currently, if you want to continually replicate time-series data from one project to another project you have to copy over every data-point in every time-series each time as there is no way to know which data-points have been added since you last replicated the data. This is not feasible with 100k+ time-series in a project. However, if we had versioned data-points then you could simply check which data-points have been created since you last did replication and copy over only the new data-points. Analysing performance of forecasts