Skip to main content
Markus Pettersen
MVP ⭐️⭐️⭐️⭐️⭐️
Markus Pettersen
MVP ⭐️⭐️⭐️⭐️⭐️
August 4, 2026
Gathering Interest

We need an option for hard deletes

Related products:Data Modeling
  • August 4, 2026
  • 3 replies
  • 42 views

Currently data modelling in CDF only support “soft deletes”, which in day to day operations works well and support the necessary sync features. However we have scenarios where we need to delete vast amounts of instances, ideally in a very short timeframe.

  1. During testing we sometimes need to delete and recreate a data model
  2. We are currently migrating data models and have two living in parallel for a time, eventually we will delete the old one, which has more than 50 million instances, with the current limit of 10 million soft deletes this would 15 days due to soft delete retention period.
  3. Cleanup of old test data
  4. Decommissioning of data models (or oil rigs for that matter)

This is an ongoing issue in one of test environments where we have a need to do a full wipe of all instances within the CDF project.

The feature I am looking for is a way to skip the soft delete default, and simply do a hard delete right away. In the scenarios I’ve listed the features of soft deletes are not needed as we do planned and coordinated complete deletes rather than a regular update from a source system. This is the first time we are facing this issue, but we see that this will be a recurring problem in the time to come for us.

 

Markus Pettersen
Aker BP - Data Platform Architect

    3 replies

    Jørgen Lund
    Expert ⭐️⭐️⭐️⭐️
    Jørgen Lund
    Product Manager
    August 5, 2026

    Hi ​@Markus Pettersen!

    Thank you for suggesting this product idea. We will follow the traction this idea gets in the community. You can expect updates on this post if we decide to include this in our future roadmap, or if we require more information.

    Markus Pettersen
    MVP ⭐️⭐️⭐️⭐️⭐️
    Markus PettersenAuthor
    MVP ⭐️⭐️⭐️⭐️⭐️
    August 6, 2026

    It’s nice that if will be considered for traction, but what is Cognite’s suggested approach to dealing with this given the current setup?

    What is the best approach to deleting vast amounts of instances? It’s not something we will do often, but it is something we occasionally need to do. It taking 9 days to delete 30M instances because we are limited to delete 10M every 3 days seems unnecessarily long. But that is a theoretical limit, because if we actually delete 10M then the regular pipelines will fail when try to delete a few instances here and there as part of daily operations. In other words, we need to always ensure that there is a buffer available, so it will take even longer.

    We understand the practical use case for soft deletes, but it only serves a purpose for daily operations, not necessary cleanup tasks. At least we like an admin feature that gives the option to clear the soft deletes either all or a selection. We will soon have a data model 50-70M instances that we have to delete and it taking 15-21 days and taking up nearly all soft deletes in all environments during that time is by no means ideal.

    Arild Eide
    Expert ⭐️⭐️⭐️⭐️
    Arild Eide
    Expert ⭐️⭐️⭐️⭐️
    August 6, 2026

    Hi ​@Markus Pettersen 

     

    The soft delete stage is there to propagate deletes to downstream systems and internal CDF components. This is a fundamental part of the system design and vital for data consistency. It is therefore not trivial to fulfill your request of skipping soft deletes entirely, without jeopardizing data consistency. 

    Presence of a large amount of deletes would be detrimental to query performance, which is why there is a separate limit to control deletes. 

    The engineering team is looking to increase the soft delete limit for the project in question, aiming to provide a meaningful improvement while still imposing a reasonable limit.

    Hope this helps for now. 

     

    Regards,

    Arild Eide

    Terms & ConditionsAccessibility statement