Skip to main content
Gathering Interest

Transformation data quality

Related products:Data Staging (RAW)
  • March 1, 2023
  • 1 reply
  • 23 views

Gaetan  Helness
MVP ⭐️⭐️⭐️⭐️⭐️
Forum|alt.badge.img+2

Transformations are used quite extensively by Data Engineers. I feel more confident about my Python code as I can unit test. Then, if I make changes to my scripts, I have more guarantees that it will not cause changes in the data. There is no way to easily see the results of a code change in Transformations without having some other script/Transformation to investigate the changed data. This can result in many data bugs, especially if you change code that was written by someone else. It would be helpful to explore a functionality towards enabling the data quality of the transformations we write every day.

1 reply

Anita Hæhre
Expert ⭐️⭐️⭐️⭐️
Forum|alt.badge.img+3
  • Head of Community
  • March 3, 2023
NewGathering Interest