We need an option for hard deletes

Currently data modelling in CDF only support “soft deletes”, which in day to day operations works well and support the necessary sync features. However we have scenarios where we need to delete vast amounts of instances, ideally in a very short timeframe.

1. During testing we sometimes need to delete and recreate a data model
2. We are currently migrating data models and have two living in parallel for a time, eventually we will delete the old one, which has more than 50 million instances, with the current limit of 10 million soft deletes this would 15 days due to soft delete retention period.
3. Cleanup of old test data
4. Decommissioning of data models (or oil rigs for that matter)

This is an ongoing issue in one of test environments where we have a need to do a full wipe of all instances within the CDF project.

The feature I am looking for is a way to skip the soft delete default, and simply do a hard delete right away. In the scenarios I’ve listed the features of soft deletes are not needed as we do planned and coordinated complete deletes rather than a regular update from a source system. This is the first time we are facing this issue, but we see that this will be a recurring problem in the time to come for us.

Markus Pettersen
Aker BP - Data Platform Architect