Limit Issue with running workflow in project

Issue with limit, In recent scenario, I logged number of workflow created, limit has reached 38, still it gave limit reached error:

Workflow count: 2024-04-26 05:17 : 38 2024-04-26 05:17 : Traceback (most recent call last): 2024-04-26 05:17 : File "/home/site/wwwroot/function/handler.py", line 51, in handle 2024-04-26 05:17 : run = client.workflows.executions.trigger( 2024-04-26 05:17 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api/workflows.py", line 208, in trigger 2024-04-26 05:17 : response = self._post(url_path=f"/workflows/{workflow_external_id}/versions/{version}/run", json=body) 2024-04-26 05:17 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 172, in _post 2024-04-26 05:17 : return self._do_request( 2024-04-26 05:17 : ^^^^^^^^^^^^^^^^^ 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 235, in _do_request 2024-04-26 05:17 : self._raise_api_error(res, payload=json_payload) 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 1201, in _raise_api_error 2024-04-26 05:17 : raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra) 2024-04-26 05:17 : cognite.client.exceptions.CogniteAPIError: The number of running workflows for this project has reached the limit of 50 | code: 429 | X-Request-ID: 9b8ecdd2-f0bf-9253-b2ac-7990e61707cb 2024-04-26 05:17 : undefined

2nd scenario:

Workflow count: 2024-04-26 03:59 : 33 2024-04-26 03:59 : Traceback (most recent call last): 2024-04-26 03:59 : File "/home/site/wwwroot/function/handler.py", line 51, in handle 2024-04-26 03:59 : run = client.workflows.executions.trigger( 2024-04-26 03:59 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api/workflows.py", line 208, in trigger 2024-04-26 03:59 : response = self._post(url_path=f"/workflows/{workflow_external_id}/versions/{version}/run", json=body) 2024-04-26 03:59 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 172, in _post 2024-04-26 03:59 : return self._do_request( 2024-04-26 03:59 : ^^^^^^^^^^^^^^^^^ 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 235, in _do_request 2024-04-26 03:59 : self._raise_api_error(res, payload=json_payload) 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 1201, in _raise_api_error 2024-04-26 03:59 : raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra) 2024-04-26 03:59 : cognite.client.exceptions.CogniteAPIError: The number of running workflows for this project has reached the limit of 50 | code: 429 | X-Request-ID: fa3a2207-5bfe-9774-9438-8d18555101ac

Hi @Rimmi Anand . 

I’ll discuss bumping the project limit with the team, will get back to you latest on Monday. May require some more information before we grant the increase. Thanks for the report on the second part, I’ll let you know when it should be fixed.
 

@Rimmi Anand I’ve deployed an attempt at fixing the reported issue. Could you verify that you can actually run 50 concurrent executions now?

@Bert Verstraete , I re-checked, still facing same issue and no other workflows in project are running apart from this

2024-04-26 11:27 : Workflow count: 2024-04-26 11:27 : 29

2024-04-26 11:27 : Traceback (most recent call last):

2024-04-26 11:27 : File "/home/site/wwwroot/function/handler.py", line 52, in handle

2024-04-26 11:27 : run = client.workflows.executions.trigger( 2024-04-26 11:27 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api/workflows.py", line 208, in trigger

2024-04-26 11:27 : response = self._post(url_path=f"/workflows/{workflow_external_id}/versions/{version}/run", json=body)

2024-04-26 11:27 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 173, in _post

2024-04-26 11:27 : return self._do_request( 2024-04-26 11:27 : ^^^^^^^^^^^^^^^^^ 2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 236, in _do_request 2024-04-26 11:27 : self._raise_api_error(res, payload=json_payload)

2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 1202, in _raise_api_error

2024-04-26 11:27 : raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra) 2024-04-26 11:27 : cognite.client.exceptions.CogniteAPIError: The number of running workflows for this project has reached the limit of 50 | code: 429 | X-Request-ID: c3770831-2847-9152-abdd-a7de1b3ccdd1

Will need a further investigation, could you share the project name and cluster that this is occurring on?

project=slb-uds-dev

cluster=westeurope-1.cognitedata.com

Hey Rimmi. We’re investigating the issue and working on a fix.

Thanks @Bert Verstraete , any ETA you can provide?
We are a bit blocked because of this.

Appreciate your support.

@Aditya Kotiyal Be assured that we have this issue as our highest priority. We’ve debugged today and found a prospective solution, but it will take a few days to get it in with proper testing as it is not trivial.

@Rimmi Anand @Aditya Kotiyal We’ve deployed an attempted fix today. Could you verify that you can run 50 workflows concurrently now? 

We’ve identified an additional issue that may be contributing to the limit not being consistent. Working on a fix today.

The last fix is deployed. We had an issue where we considered dynamic tasks and subworkflows that are part of a workflow as additional running workflows. This is no longer the case. Would appreciate it if you can re-test to verify. @Aditya Kotiyal @Rimmi Anand 

Thanks! 

@Rimmi Anand Has this issue been resolved with the latest fix?

Thank you @Bert Verstraete , I have tested this running from function locally, 50 workflows are being created and running. However, running from UI,  It must be related to session issue, so couldn’t test to trigger workflows , Raised issue for same .

Hi @Rimmi Anand,

We are following up to see whether you're satisfied with the responses you've received?