Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
When writing a transformation, the logic is slowly built up. Sometimes, you join different tables, and have different sub queries. Each subquery needs a correct logic, so a Data Engineer often spends time identifying that logic is correct. Other times, you want to confirm the behaviour of data in CDF, so you use transformations to profile some specific data.
Since we do not have views/temporary tables, you often need to create a database in CDF (a temporary one), get permission to write to this database (hard to do in production), create a table, then run your transformation so it writes to this table, and then write another table to query your previous temporary table. If your original query was wrong/you want to test something else, you clear the table and start over.
it would be great to build a functionality to help with this iterative debugging process
Transformations testing
Transformations and RAW
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Hi@Gaetan Helness ,
I get the access permissions you will need to create temp DB/tables in production environments. How about using Common Table Expressions to store results of sub query temporarily. Does that work for you?
Thanks,
Sunil
We use WITH clauses quite a lot when building up queries. However, when developing, I may want to investigate that sub table. I cannot do this with the preview button (which caps out at some runtime/some row limit). As such, I need to run the transformation on the full source tables to see the results of what I am building up.
When running the transformation to view the quality of the transformation/an sub table, it needs to write to somewhere.