We're excited to announce two powerful new features for Data Workflows that will make automating and managing data pipelines easier: Triggers for Data Workflows and Nested Subworkflows.

1. Triggers for Data Workflows

Automating workflows has been tricky, requiring custom workarounds that make the process time-consuming. Understanding and managing the triggers behind each workflow can be difficult as well.

With our new Triggers feature, you can now:

• Natively define schedule-based (cron) triggers within Data Workflows.
• Configure input parameters directly within each trigger.
• Access detailed run history for all triggers to gain visibility into workflow execution.

This update simplifies automation by embedding trigger definitions directly into workflows, and is designed to evolve to support more complex trigger types in upcoming releases.

How to Get Started

• API Documentation: Triggers API
• Python SDK: Create Triggers
• User Guide: Triggers in Data Workflows

2. Nested Subworkflows

Large workflows with many steps often become difficult to manage, and it’s challenging to reuse common steps between different workflows.

We've introduced Nested Subworkflows, that:

• Allows you to reference another workflow definition in the subworkflow task type.

• Automatically embed this referenced workflow into the main workflow at runtime.

This can be illustrated by an example Workflow A and Workflow B. In the definition of Workflow A, a task of type subworkflow contains a reference to the definiton of Workflow B, which is completely separate from Workflow A.

When executing Workflow A, the definition of Workflow B will be dynamically loaded into and executed as a subworkflow inside workflow A (illustrated below). If Workflow B is executed directly, Workflow A is not impacted.

This enhancement simplifies the management of complex workflows by breaking them into smaller, manageable pieces. It also encourages the reuse of common steps across multiple workflows, improving efficiency and reducing redundancy.

How to Get Started

• Documentation: Subworkflow Task
• API Documentation: Subworkflow Task

Both of these updates are designed to make your data workflows more efficient, scalable, and easier to maintain. We’re excited to see how you’ll use these new capabilities to power your data solutions!

Let us know in the comments if you have any feedback or questions!