Hi Team,
Below is a requirement to increase user experience and performance while using transformations.
Current problem: As a data engineer, I do not have visibility of how many transformations are scheduled to run at a particular time period.
As a result, we just rely on manual check or intuition at what time we must schedule our transformations to not hit the concurrency limit or result in increased load on the cdf instance.
Proposed idea: A schedule heatmap between time of day against the count would help in this regard. The time should be divided into a group like 10-15 minutes to analyze how many transformations are scheduled within that time period. The count should also consider the repetitions based on the cron expression passed.
Note: This can be extended to functions and data workflows as well