We are doing quite a few transformations on larger FDM-tables, and are experiencing them to be a lot slower than what is to be expected.
From my understanding CDF is reading some rows a at a time from the database, doing a standard database operation (with Spark), and writing back to the database. The read operation is very slow (for one transform I am running now read speed is 1600 rows/s).
Is it possible to write a transformation (with standard SQL) that will be run in-database, that avoids the slow read?
This would be especially beneficial for FDM to FDM transformations, as this would reduce time from minutes to milliseconds for some of our transformations.