Performance improvement on populating data on Data Models using transformation

Hi ​@Shashan Udawatte 

I don’t have all the details, but based on what we're discussing, this sounds like a heavy workload. You could consider a parallel processing approach to move data from the staging area to the silver/gold layers (data model), using techniques similar to those available in Apache Airflow.

This approach is likely also supported in Cognite Workflows, where you can run Cognite Functions in parallel to process chunks of data independently for each task.

If you can share more details, we’d be happy to suggest a solution that better fits your specific scenario.

This is ideal for ETL/ELT pipelines where:

• You can split the dataset into chunks.

• Each chunk is processed independently (e.g., loading different partitions, handling different tables or time intervals).

• The results are later aggregated or moved to the silver/gold layers.

Regards,
André

FYI ​@sebastianheibo.akso