Is DevOps organizations always best for Industrial DataOps?

If you are curious on how to get the most out of your Cognite Data Fusion subscription, you have come to the right place. This is part of a series of posts where we share some of our experience from working with customers in their journey towards an Industrial DataOps organization. We want to share lessons learned, mistakes made, good practices observed, and observations of pitfalls and risks. This is not the absolute truth, but hopefully a way to spark good discussions around an inherently complex topic!

To quickly introduce ourselves, we are @Arjo Oosten, Digital Transformation Leader, wintersport addict and passionate about driving hands-on digital growth strategies and value based decision making, and @Karolina Luna, Solution Architect, cat lover, and passionate about the lifecycle perspective of everything (like solutions and data products). 

To learn more about Cognite Data Fusion, we recommend this post.

DevOps and Industrial DataOps

Both DevOps and DataOps have many different definitions. The Wikipedia definition of DevOps is “a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality“. One of many definitions of DataOps is “a collaborative data management practice, focused on improving the communication, integration and automation of data flows between data managers and consumers across an organization”

While DevOps, with good cause, has been in vogue for many years now, there may be several considerations to make when using DEvOps practices in digitalization and Industrial DataOps projects. The technical practices, such as CI/CD, are great tools that should definitely be used, but how might we think about the team setup? During the development and early operation phases, a DevOps mindset is critical to achieve efficient and, more importantly, relevant development. There are some areas where classical software product development, which is often continuous, differs from development of industrial DataOps and digitalization solutions. 

One key difference from conventional software development is that many projects are smaller in scale. For example, a project can be around integrating a new data source, adding a data science model, or creating a dashboard in Grafana for an end user. Even in the scenarios where these activities are combined, the development effort is relatively small. In a sense, these lightweight ways of accessing and handling data is exactly what digitalization and Industrial DataOps is about! There is often a long list of these small projects that each can bring real business value to the organization.

Adding up maintenance of such small components and solutions over time requires a lot of context switching and may be hard to combine with development tasks for unrelated solutions. People also move around and it can be unrealistic to expect to keep the same individuals from development to operations. So while a separate maintenance team is unlikely to be a good solution for the first few solutions, it can be worthwhile to monitor the development to maintenance ratio over time.
 

What is your experience with DevOps and Industrial DataOps organizations? How have you set up your teams?