concurrent executions in workflows

Hi @Niranjan Madhukar Karvekar. We have a concurrent execution limit of tasks per project of 50. Meaning that only 50 tasks will be concurrently executing at a given time, the rest will be waiting until there’s a slot available. This explains the increasing execution times. I’ll raise this within the team and see if we can find ways to optimize the current limit behavior and the possibility for expanding the limit.

Out of interest, how many tasks are scheduled at maximum at a given time?

Thanks @Bert Verstraete . The idea is to check how much historical data we can process in parallel through multiple workflow instances and multiple tasks. If we have 1 workflow instance with 40 concurrent tasks v/s  2 workflow instances with 20 concurrent tasks in each, is there any difference in performance? Is there any recommendations around this?

It will be also good to understand if the current limit can be expanded easily based on use cases.  

For incremental daily updates, the workflows will be scheduled once in a day, but for historical data we expect the workflow to run only once to get all historical data which will be time taking where mostly concurrency matters.

Hey @Niranjan Madhukar Karvekar . There is no difference between running one workflow with 40 concurrent tasks versus two workflows with 20 concurrent tasks each. Important to keep in mind is the 100 task per workflow execution limit and 50 concurrent tasks execution limit per project. 

A few follow-up questions for you:

• How many tasks are running at concurrently at maximum for the current use case implementation? 
• Which cluster are you operating on and which project?
• What type of tasks do the workflows consist of?

• What’s the operational consequence for workflow being throttled because of the limit?

If there is a significant operational consequence, we can work with you to improve this. In the meantime, you can work on spacing out the workflow executions in time to work within bounds of the mentioned 50 concurrent tasks execution limit. I.e., you can achieve this by running workflows more often per day on less data.

Hi @Bert Verstraete 

• How many tasks are running at concurrently at maximum for the current use case implementation? 
  • Because of 50 concurrent executions, we are now running 5 workflow instances with 10 tasks in each workflow to process 50 wellbores. As we wanted to process around 10k-15k wellbores, we have to run it in batches. 
• Which cluster are you operating on and which project?
  • cluster: westeurope-1, project: slb-uds-dev
• What type of tasks do the workflows consist of?
  • We have 2 major workflows, we expect historical workflow to run only once and incremental to run once in a day:
    • Historical workflow - Read all timeseries from multiple datasets, get the instances from data model based on entity metadata present in each timeseries, create the payload as per destination schema, ingest the payload using REST API in destination. 
    • Incremental workflow - Read timeseries from all subscriptions, get the instances from data model based on entity metadata present in each timeseries, create the payload as per destination schema, ingest the payload using REST API in destination. 

• What’s the operational consequence for workflow being throttled because of the limit?
  • Data size that we are considering : 15k wellbores, 26 properties for each wellbore, 20 years of daily aggregated data. (0.4 million timeseries, 7.8 million data points) 
  • When we deploy the workflow for the first time, historical workflow would take more time and to process all data, the workflow can be scheduled to tun on subset of data. Even if historical workflow takes more time, then that should be fine. 
  • But for incremental workflow, if we receive more timeseries in subscriptions , (>50) then current limit is not sufficient. We can still run workflow more often on less data but then we have to keep the track of which data is processed in each run. In each run, we may have to schedule the workflow again to process unprocessed data. It will be good to know if the current limits can be increased to process more incremental data.

@Bert Verstraete 

1. Can you please confirm if 50 limit of concurrent executions in a project is soft/hard limit?
2. If it is the soft limit, what is the turnaround time to increase it if the use case is justified. 

@Niranjan Madhukar Karvekar
It is indeed a soft limit, it can be bumped (per project). Bumping the limit to 100 should provide you a better experience. I’ll be discussing this with the team the rest of the team next week and communicate back how we’ll help.

@Bert Verstraete Can you please update if it is possible to increase the limit in our projects slb-uds-dev and slb-uds-qa?

Hi @Niranjan Madhukar Karvekar. Thanks for the detailed information about your use case. We’ll discuss this issue within the team today and get back to you (today) on how we plan to proceed.

Hi @Niranjan Madhukar Karvekar. We’ll follow up on this thread by email.