Question

Limit Issue with running workflow in project

R
Rank
Userlevel 1

Is it possible to increase the limit on execution of workflow instances per project or limit on execution workflow instance itself instead of project ? as we require to schedule workflow, on that it depends how much data we get to run workflows instances, so it can be more than 50 

 

14 replies

Bert Verstraete
Rank
Userlevel 2

We’ve identified an additional issue that may be contributing to the limit not being consistent. Working on a fix today.

R
Rank
Userlevel 1

Thank you @Bert Verstraete , I have tested this running from function locally, 50 workflows are being created and running. However, running from UI,  It must be related to session issue, so couldn’t test to trigger workflows , Raised issue for same .

 

Bert Verstraete
Rank
Userlevel 2

@Aditya Kotiyal Be assured that we have this issue as our highest priority. We’ve debugged today and found a prospective solution, but it will take a few days to get it in with proper testing as it is not trivial.

Bert Verstraete
Rank
Userlevel 2

@Rimmi Anand Has this issue been resolved with the latest fix?

R
Rank
Userlevel 1

Issue with limit, In recent scenario, I logged number of workflow created, limit has reached 38, still it gave limit reached error:

 

Workflow count: 2024-04-26 05:17 : 38 2024-04-26 05:17 : Traceback (most recent call last): 2024-04-26 05:17 : File "/home/site/wwwroot/function/handler.py", line 51, in handle 2024-04-26 05:17 : run = client.workflows.executions.trigger( 2024-04-26 05:17 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api/workflows.py", line 208, in trigger 2024-04-26 05:17 : response = self._post(url_path=f"/workflows/{workflow_external_id}/versions/{version}/run", json=body) 2024-04-26 05:17 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 172, in _post 2024-04-26 05:17 : return self._do_request( 2024-04-26 05:17 : ^^^^^^^^^^^^^^^^^ 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 235, in _do_request 2024-04-26 05:17 : self._raise_api_error(res, payload=json_payload) 2024-04-26 05:17 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 1201, in _raise_api_error 2024-04-26 05:17 : raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra) 2024-04-26 05:17 : cognite.client.exceptions.CogniteAPIError: The number of running workflows for this project has reached the limit of 50 | code: 429 | X-Request-ID: 9b8ecdd2-f0bf-9253-b2ac-7990e61707cb 2024-04-26 05:17 : undefined

 

2nd scenario:

Workflow count: 2024-04-26 03:59 : 33 2024-04-26 03:59 : Traceback (most recent call last): 2024-04-26 03:59 : File "/home/site/wwwroot/function/handler.py", line 51, in handle 2024-04-26 03:59 : run = client.workflows.executions.trigger( 2024-04-26 03:59 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api/workflows.py", line 208, in trigger 2024-04-26 03:59 : response = self._post(url_path=f"/workflows/{workflow_external_id}/versions/{version}/run", json=body) 2024-04-26 03:59 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 172, in _post 2024-04-26 03:59 : return self._do_request( 2024-04-26 03:59 : ^^^^^^^^^^^^^^^^^ 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 235, in _do_request 2024-04-26 03:59 : self._raise_api_error(res, payload=json_payload) 2024-04-26 03:59 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 1201, in _raise_api_error 2024-04-26 03:59 : raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra) 2024-04-26 03:59 : cognite.client.exceptions.CogniteAPIError: The number of running workflows for this project has reached the limit of 50 | code: 429 | X-Request-ID: fa3a2207-5bfe-9774-9438-8d18555101ac

Bert Verstraete
Rank
Userlevel 2

Hi @Rimmi Anand . 

I’ll discuss bumping the project limit with the team, will get back to you latest on Monday. May require some more information before we grant the increase. Thanks for the report on the second part, I’ll let you know when it should be fixed.
 

Bert Verstraete
Rank
Userlevel 2

@Rimmi Anand I’ve deployed an attempt at fixing the reported issue. Could you verify that you can actually run 50 concurrent executions now?

R
Rank
Userlevel 1

@Bert Verstraete , I re-checked, still facing same issue and no other workflows in project are running apart from this

2024-04-26 11:27 : Workflow count: 2024-04-26 11:27 : 29

2024-04-26 11:27 : Traceback (most recent call last):

2024-04-26 11:27 : File "/home/site/wwwroot/function/handler.py", line 52, in handle

2024-04-26 11:27 : run = client.workflows.executions.trigger( 2024-04-26 11:27 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api/workflows.py", line 208, in trigger

2024-04-26 11:27 : response = self._post(url_path=f"/workflows/{workflow_external_id}/versions/{version}/run", json=body)

2024-04-26 11:27 : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 173, in _post

2024-04-26 11:27 : return self._do_request( 2024-04-26 11:27 : ^^^^^^^^^^^^^^^^^ 2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 236, in _do_request 2024-04-26 11:27 : self._raise_api_error(res, payload=json_payload)

2024-04-26 11:27 : File "/home/site/wwwroot/.python_packages/lib/site-packages/cognite/client/_api_client.py", line 1202, in _raise_api_error

2024-04-26 11:27 : raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra) 2024-04-26 11:27 : cognite.client.exceptions.CogniteAPIError: The number of running workflows for this project has reached the limit of 50 | code: 429 | X-Request-ID: c3770831-2847-9152-abdd-a7de1b3ccdd1

 

 

Bert Verstraete
Rank
Userlevel 2

Will need a further investigation, could you share the project name and cluster that this is occurring on?

R
Rank
Userlevel 1

project=slb-uds-dev

cluster=westeurope-1.cognitedata.com

Bert Verstraete
Rank
Userlevel 2

Hey Rimmi. We’re investigating the issue and working on a fix.

A
Rank
Userlevel 2
Badge +3

Thanks @Bert Verstraete , any ETA you can provide?
We are a bit blocked because of this.

Appreciate your support.

Bert Verstraete
Rank
Userlevel 2

@Rimmi Anand @Aditya Kotiyal We’ve deployed an attempted fix today. Could you verify that you can run 50 workflows concurrently now? 

Bert Verstraete
Rank
Userlevel 2

The last fix is deployed. We had an issue where we considered dynamic tasks and subworkflows that are part of a workflow as additional running workflows. This is no longer the case. Would appreciate it if you can re-test to verify. @Aditya Kotiyal @Rimmi Anand 

Thanks! 

Reply


Terms & ConditionsCookie Settings