Skip to main content
Question

CDC extractors

  • 19 September 2024
  • 3 replies
  • 43 views

Hello, 

 

Does CDF provide CDC connectors/extractors for common SQL databases ? 

 

Thanks

3 replies

Forum|alt.badge.img

(I’m presuming you mean “Change Data Capture” by CDC?) 

For CDC like behavior, we have a hosted extractor for Kafka available from the Integrate -> “Extractors” page in Fusion you may be able to use with a bit of rigging on the source side?

We don’t explicitly have a CDC connector for SQL per se, perhaps you could craft a query statement against any SQL friendly change tables? Assuming the DB supports/uses that rather than expecting the reader to understand transaction logs… 🤷


  • Author
  • Seasoned
  • 13 replies
  • September 27, 2024

(I’m presuming you mean “Change Data Capture” by CDC?) 

Yes

I think crafting SQL queries for this need is not very optimal and reaches its limit very fast. 

Given that there are stable OSS solutions out there for this (exp: https://debezium.io) , maybe it is something you could add it to your backlog/road map soon :) 


Andre Alves
MVP
Forum|alt.badge.img+11
  • MVP
  • 115 replies
  • September 27, 2024

Hi @Marwen TALEB 
 

Since CDF (Cognite Data Fusion) is not yet included in the ADF (Azure Data Factory) connector list with native change data capture (CDC) support,  I recommend exploring the auto-incremental extraction option. With this approach, you simply need to specify the incremental column to capture data changes. Once set up, the system will take care of the rest, including generating dynamic queries for delta loading and managing checkpoints for each activity run.

I understand that native change data capture is generally a more efficient and seamless solution, but sometimes we need to work with alternatives while waiting for a more robust solution to become available.

Do you think this could work for your needs, or does it not make sense for your use case?


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings