Skip to main content
  • Home
  • Product Ideas
  • Sequencing Extractors , Transformations , downstream objects for Enhanced data Workflows

 

Add support for sequencing-specific extractors and transformations and downstream objects. This would enable users to easily use tools for common sequencing data processing tasks.

Key features:

support to trigger data workflows post competition of extraction process.

currently it is happening in silos and the transformation assumes that the extraction process is complete and runs on a predefined frequency. we would like to have the transformation triggered post completion of the extraction process 

 

Automated Workflows in relativity enables users to streamline and automate data processing tasks, particularly for sequencing-specific operations. Key features include:

Sequencing-Specific Extractors and Transformations

The system should support specialized extractors and transformations tailored for sequencing data. This allows users to easily process common sequencing data types without manual intervention.

Downstream Object Support

Automated Workflows can now handle downstream objects, enabling more complex data processing pipelines that depend on earlier extraction and transformation steps.

Triggered Transformations

A crucial improvement is the ability to trigger transformations immediately upon completion of the extraction process. This replaces the previous model where transformations ran on predefined schedules, regardless of extraction status.

Workflow Chaining

Users should be able to  chain multiple workflows together, with each step triggering the next upon completion. This creates a seamless, end-to-end automated process for data handling.

Customizable Triggers

The system offers flexible trigger options, allowing workflows to start based on various conditions such as  completion of other workflows.

 

Hello @Harsha, thanks for submitting this product idea. Much of what you’re describing is already supported today through Data Workflows in CDF or on our roadmap for upcoming releases. Have you already tested data workflows? 

I’m breaking down your post a bit to answer the different parts of it.

 

Support to trigger data workflows post competition of extraction process.

We are working on event-based triggers for data workflows. In the Q4 release you’ll be able to create data workflow triggers that start a workflow based on changes to data modeling instances (user defines the relevant query). A possibility then would be to have the extractor create instances in data modeling signaling the completion of a task, and use this to trigger the data workflow. In Q1 (tentative timeline), we plan on adding native support for extractor events, meaning you’ll be able to define a trigger for a data workflow based on events emitted by an extractor represented in extraction pipelines (a new extraction pipelines service also coming Q1). On roadmap.

 

Sequencing-Specific Extractors and Transformations

A combination of what’s described above and the ability for data workflows to orchestrate various data processing within CDF (Functions, Transformations, Simulators, and more to be added in upcoming releases). Already supported + on roadmap.

 

Triggered Transformations

Described above. On roadmap.

 

Workflow Chaining

Using the Subworkflow task in data workflows you can reference another workflow. The other workflow will then be embedded into the main workflow at runtime. This is not chaining directly, but can be viewed as a substitute feature. I’m noting down the request to have completion of a workflow trigger another workflow. We view this as another type of event-based trigger, similar to what’s described above for extractors. Already supported and extended functionality to be considered for roadmap.

 

Customizable Triggers

Described above. On roadmap.

Terms & ConditionsCookie Settings