Hi Team,
We have a request from Aker Solutions that Transformations populating data models takes much longer than populating raw table. We request for a feature to improve the performance of transformation that are populating data on Data Models using transformation.
+13I don’t have all the details, but based on what we're discussing, this sounds like a heavy workload. You could consider a parallel processing approach to move data from the staging area to the silver/gold layers (data model), using techniques similar to those available in Apache Airflow.
This approach is likely also supported in Cognite Workflows, where you can run Cognite Functions in parallel to process chunks of data independently for each task.
If you can share more details, we’d be happy to suggest a solution that better fits your specific scenario.
This is ideal for ETL/ELT pipelines where:
You can split the dataset into chunks.
Each chunk is processed independently (e.g., loading different partitions, handling different tables or time intervals).
The results are later aggregated or moved to the silver/gold layers.
Regards,
André
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
