Skip to main content
Closed

Ability to do standard transformations in-database

Related products:Transformations and RAW
  • October 9, 2024
  • 1 reply
  • 18 views

We are doing quite a few transformations on larger FDM-tables, and are experiencing them to be a lot slower than what is to be expected. 

From my understanding CDF is reading  some rows a at a time from the database, doing a standard database operation (with Spark), and writing back to the database. The read operation is very slow (for one transform I am running now read speed is 1600 rows/s). 

Is it possible to write a transformation (with standard SQL) that will be run in-database, that avoids the slow read? 

This would be especially beneficial for FDM to FDM transformations, as this would reduce time from minutes to milliseconds for some of our transformations. 

Jørgen Lund
Seasoned Practitioner

Hi ​@Håkon Beckstrøm, thanks for submitting this product idea. 

Certain operations are pushed down to the Data Modeling service, but compute is mainly performed in Transformations. It’s not possible to run jobs like described within the Data Modeling service (DMS), and no such capabilities are planned. Instead, we will focus on improving performance on reads and writes to and from DMS, including change detection when a Transformation writes to DMS. 

 



Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings