We are doing quite a few transformations on larger FDM-tables, and are experiencing them to be a lot slower than what is to be expected.
From my understanding CDF is reading some rows a at a time from the database, doing a standard database operation (with Spark), and writing back to the database. The read operation is very slow (for one transform I am running now read speed is 1600 rows/s).
Is it possible to write a transformation (with standard SQL) that will be run in-database, that avoids the slow read?
This would be especially beneficial for FDM to FDM transformations, as this would reduce time from minutes to milliseconds for some of our transformations.
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support