Answer

Transformation Schedule on CDF

Forum|Forum|1 year ago
January 18, 2024
2 replies
61 views

+1

Rajendra Pasupuleti
Seasoned

Hi There,

We have multiple CDF transformations running on schedule for every 15 minutes. Sometimes these jobs are taking more than 15 minutes and leading to Transformation failure with an error - ‘A job already runs for this transform ’. Is there any provision for us in CDF to not run a job/transformation while it is running already.

Best answer by Andrian Gasper

Hi :)
To my knowledge, there is no provision for that. If I am wrong, someone please correct me.

However, I would like to come with a suggestion on how to fix this.

My suggestion is to have 2 identical (except the name and id of course) transformations. Scheduled every 30 minutes but starting at different times, such that there is always at least one of them running every 15 minutes.

For example “transformation_1” will be running at 15:00, 15:30, 16:00 … etc. And “transformation_2” will be running at 15:15, 15:45, 16:15 … etc.

Note: This might lead to having both of them running a the same time (overlapping), so one needs to account for that. Or, before starting a transformation, add a step to check whether the other transformation is currently active. This can be done by querying a status table or a log where the start and end times of each transformation are recorded. But this requires to set up such table in place first.

Let me know what you think. :)

+2

Andrian Gasper
Practitioner
Answer
Forum|Forum|1 year ago
January 18, 2024

Hi :)
To my knowledge, there is no provision for that. If I am wrong, someone please correct me.

However, I would like to come with a suggestion on how to fix this.

My suggestion is to have 2 identical (except the name and id of course) transformations. Scheduled every 30 minutes but starting at different times, such that there is always at least one of them running every 15 minutes.

For example “transformation_1” will be running at 15:00, 15:30, 16:00 … etc. And “transformation_2” will be running at 15:15, 15:45, 16:15 … etc.

Note: This might lead to having both of them running a the same time (overlapping), so one needs to account for that. Or, before starting a transformation, add a step to check whether the other transformation is currently active. This can be done by querying a status table or a log where the start and end times of each transformation are recorded. But this requires to set up such table in place first.

Let me know what you think. :)

R