We request the introduction of an auto-reconciliation feature for the Cognite DB Extractor. Currently, we face significant issues with data synchronization between our source systems and Cognite. When physical deletes occur in the source tables, these deletions are not reflected in Cognite, leading to outdated or incorrect data residing in our Cognite environment. This discrepancy is resulting in data integrity issues and misinformed decision-making.
To address this, we propose a feature that automatically detects and reconciles physical deletes from the source system, ensuring that these changes are promptly reflected in Cognite. This enhancement should operate with minimal latency, ideally processing updates within minutes, to ensure that our data remains accurate and reliable. Implementing this feature will streamline our data management processes and maintain consistency across systems.