Skip to main content
Question

concurrent executions in workflows

  • 18 July 2024
  • 9 replies
  • 122 views

Hi,

We use CDF workflow to process the data present in data model. Based on the inputs, we trigger multiple workflow instances to take the advantage of scalability. We observed that, the workflow execution time is increasing after we trigger more workflow instances. 

 

Here is the simple diagram to show how we use workflow:

 

 

 

For example:

If we run 1 workflow instance to process 20 wellbores (assume some processing logic divided between F1 to F4) with 20 concurrent tasks,  the execution time of that workflow instance is 5 mins.  

Now if we want to process more wellbores say 40, we trigger 2 workflow instances , 1st workflow instance to process 20 wellbores and 2nd workflow instance to process other 20 wellbores and we expect the execution time for all workflows to be approximately same but the total time is increasing if we compare with only 1 workflow instance with 20 tasks. 

Can you please help me to understand if anything is missing? 

9 replies

Bert Verstraete
Rank
Userlevel 2

Hi @Niranjan Madhukar Karvekar. We have a concurrent execution limit of tasks per project of 50. Meaning that only 50 tasks will be concurrently executing at a given time, the rest will be waiting until there’s a slot available. This explains the increasing execution times. I’ll raise this within the team and see if we can find ways to optimize the current limit behavior and the possibility for expanding the limit.

Out of interest, how many tasks are scheduled at maximum at a given time?

N
Rank
Userlevel 2

Thanks @Bert Verstraete . The idea is to check how much historical data we can process in parallel through multiple workflow instances and multiple tasks. If we have 1 workflow instance with 40 concurrent tasks v/s  2 workflow instances with 20 concurrent tasks in each, is there any difference in performance? Is there any recommendations around this?

It will be also good to understand if the current limit can be expanded easily based on use cases.  

For incremental daily updates, the workflows will be scheduled once in a day, but for historical data we expect the workflow to run only once to get all historical data which will be time taking where mostly concurrency matters.

 

Bert Verstraete
Rank
Userlevel 2

Hey @Niranjan Madhukar Karvekar . There is no difference between running one workflow with 40 concurrent tasks versus two workflows with 20 concurrent tasks each. Important to keep in mind is the 100 task per workflow execution limit and 50 concurrent tasks execution limit per project. 

A few follow-up questions for you:

  • How many tasks are running at concurrently at maximum for the current use case implementation? 
  • Which cluster are you operating on and which project?
  • What type of tasks do the workflows consist of?
  • What’s the operational consequence for workflow being throttled because of the limit?

If there is a significant operational consequence, we can work with you to improve this. In the meantime, you can work on spacing out the workflow executions in time to work within bounds of the mentioned 50 concurrent tasks execution limit. I.e., you can achieve this by running workflows more often per day on less data.

N
Rank
Userlevel 2

 

Hi @Bert Verstraete 

  • How many tasks are running at concurrently at maximum for the current use case implementation? 
    • Because of 50 concurrent executions, we are now running 5 workflow instances with 10 tasks in each workflow to process 50 wellbores. As we wanted to process around 10k-15k wellbores, we have to run it in batches. 
  • Which cluster are you operating on and which project?
    • cluster: westeurope-1, project: slb-uds-dev
  • What type of tasks do the workflows consist of?
    • We have 2 major workflows, we expect historical workflow to run only once and incremental to run once in a day:
      • Historical workflow - Read all timeseries from multiple datasets, get the instances from data model based on entity metadata present in each timeseries, create the payload as per destination schema, ingest the payload using REST API in destination. 
      • Incremental workflow - Read timeseries from all subscriptions, get the instances from data model based on entity metadata present in each timeseries, create the payload as per destination schema, ingest the payload using REST API in destination. 
  • What’s the operational consequence for workflow being throttled because of the limit?
    • Data size that we are considering : 15k wellbores, 26 properties for each wellbore, 20 years of daily aggregated data. (0.4 million timeseries, 7.8 million data points) 
    • When we deploy the workflow for the first time, historical workflow would take more time and to process all data, the workflow can be scheduled to tun on subset of data. Even if historical workflow takes more time, then that should be fine. 
    • But for incremental workflow, if we receive more timeseries in subscriptions , (>50) then current limit is not sufficient. We can still run workflow more often on less data but then we have to keep the track of which data is processed in each run. In each run, we may have to schedule the workflow again to process unprocessed data. It will be good to know if the current limits can be increased to process more incremental data.

 

N
Rank
Userlevel 2

@Bert Verstraete 

  1. Can you please confirm if 50 limit of concurrent executions in a project is soft/hard limit?
  2. If it is the soft limit, what is the turnaround time to increase it if the use case is justified. 

 

 

Bert Verstraete
Rank
Userlevel 2

@Niranjan Madhukar Karvekar
It is indeed a soft limit, it can be bumped (per project). Bumping the limit to 100 should provide you a better experience. I’ll be discussing this with the team the rest of the team next week and communicate back how we’ll help.

N
Rank
Userlevel 2

@Bert Verstraete Can you please update if it is possible to increase the limit in our projects slb-uds-dev and slb-uds-qa?

 

 

Jørgen Lund
Rank
Userlevel 4

Hi @Niranjan Madhukar Karvekar. Thanks for the detailed information about your use case. We’ll discuss this issue within the team today and get back to you (today) on how we plan to proceed.

Jørgen Lund
Rank
Userlevel 4

Hi @Niranjan Madhukar Karvekar. We’ll follow up on this thread by email. 

Reply


Terms & ConditionsCookie Settings