Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
Dynamic Tasks processing in sequence
I have created 1 workflow , in which I am creating dynamic tasks depending on input, it creates batch of ids and create tasks out of it. Below is workflow definition
WorkflowVersionUpsert(
workflow_external_id="test_dynamic-0729",
version="1",
workflow_definition=WorkflowDefinitionUpsert(
description="This workflow has two steps",
tasks=[
WorkflowTask(
external_id="test_sub_tasks",
parameters=FunctionTaskParameters(
external_id="test_sub_tasks",
data="${workflow.input}"
),
retries=1,
timeout=3600,
depends_on=[],
on_failure = "abortWorkflow",
),
WorkflowTask(
external_id="test_create_sub",
parameters=DynamicTaskParameters(
tasks="${test_sub_tasks.output.response.tasks}"
),
name="Dynamic Task",
description="Executes a list of workflow tasks for subscription creation",
retries=0,
timeout=3600,
depends_on=["test_sub_tasks"],
on_failure = "abortWorkflow",
)
]
)
As part of this workflow, I have some task that are needed to be executed in parallel and are expected to finish in around similar time if running parallel. However, dynamic tasks are not getting executed in parallel, I have 6 dynamic tasks, but they are finishing up sequentially. Sharing snapshot. If we can see time difference between first and last task is ~6m. Not able to take advantage of parallel execution with dynamic tasks.
Impact : Dynamic Execution taking sequential time to process tasks, further delaying completion of workflow
