New features: Optional Tasks and Concurrency Policies for Transformations

Hi! We have made the following updates available for Workflows.

Optional Tasks

Defining non-critical tasks in a workflow as optional, meaning that in the event of a task failure, the workflow itself should continue, has been a much requested feature. 

You can now set the policy for how to handle failures and timeouts for each task in a workflow. By default, the workflow will fail if a task fails (after retries) or if the task times out. Alternatively, you can set the parameter to skip the task in the event of a failure or timeout (after retries). This means that the workflow execution will continue even when the task does not complete successfully. 

The feature is documented in the API specififcation here (navigate items > workflowDefinition > tasks > onFailure). 

Concurrency Policies for Transformations

If a job is already running for a given Transformation in CDF, a new job cannot be started concurrently (fails with error message “A job already runs for this transform”). 

To better handle this scenario for Transformation tasks in workflows and make pipelines of Transformations more robust, you can now define the policy for how to handle concurrent executions of the same Transformation in CDF directly in the workflow definition, with the following options: 

• fail: The task fails if another instance of the Transformation is currently running.
• waitForCurrent: The task will pause and wait for the already running Transformation to complete. Once completed, the task is completed. This mode is useful for preventing redundant Transformation runs.
• restartAfterRunning: The task waits for the ongoing Transformation to finish. After completion, the task restarts the Transformation. This mode ensures that the most recent data can be used by following tasks.

The feature is documented in the API specififcation here (navigate items > workflowDefinition > tasks > parameters > Transformation Task Parameters > concurrencyPolicy).

Try out the new functionality, and let us know in the comments if you have any questions or feedback!