Cross-functional Industrial DataOps

If you are curious on how to get the most out of your Cognite Data Fusion subscription, you have come to the right place. This is part of a series of posts where we share some of our experience from working with customers in their journey towards an Industrial DataOps organization. We want to share lessons learned, mistakes made, good practices observed, and observations of pitfalls and risks. This is not the absolute truth, but hopefully a way to spark good discussions around an inherently complex topic!

To quickly introduce ourselves, we are @Arjo Oosten, Digital Transformation Leader, winter sport addict and passionate about driving hands-on digital growth strategies and value based decision making, and @Karolina Luna, Solution Architect, cat lover, and passionate about the lifecycle perspective of everything (like solutions and data products). 

To learn more about Cognite Data Fusion, we recommend this post.

Different types of cross-functional

Industrial DataOps, like all other Ops models out there, is a team sport and part of making sure it is successful is ensuring good collaboration across the functions. There are two key dimensions I would like to highlight:

1. Collaboration across the lifecycle

2. Collaboration between different development teams.

These types of cross-functional dimensions needs to be taken into account both when developing data products, i.e. owned and governed sets of data built for consumption toward business value, and solutions, i.e. the end-to-end components needed to solve a use case, including end-user facing components such as dashboards.

Collaboration across the lifecycle

The first dimension is the lifecycle of the solution or data product, where you want to take in the potential ideas on one end, and come out with selected, highly functional, and value-generating products on the other end. 

The first part of the lifecycle is the delivery model where you identify, prioritize, and plan which solutions and data products to create. This model also includes the creation of those deliverables, from design to go-live. The second part is the operations model where the product is operated in production. This model handles change requests, upgrades, defects, questions, and access requests that need to be resolved for the product to continue delivering value to the users. A separate model for delivery and operations does not mean that it has to be handled by separate teams. Throughout the lifecycle, the same data and technology stack, and the same principles for governance apply, including for governance of data.

So why is it so important to get this perspective of cross functional collaboration to work well? One reason is to ensure that functionality is developed in a quick and cost-efficient manner. A method to reduce costly transitions of tasks between teams is to cover more of the lifecycle in the same team and utilize technology to reduce overhead, for example as represented by DevOps practice of handling development and operations in the same team. 

Another reason why it is important is to not lose important knowledge about the inner workings of the product that the developer team possesses, which easily happens, particularly when handing over to an operations team. This potential gap can reduce your ability and speed to resolve user issues, which ultimately results in a poor experience due to long times in handling issues. 

Poor contact between the developer team and the end users can also translate to a lack of understanding in the developer team of what the user experience is actually like, and thus to a low quality product. 

Collaboration between different development teams

The second cross-functional dimension is a collaboration across different disciplines/departments such as domain experts, data scientists, and data engineers, that must collaborate to be able to create the final deliverable. There is a balance to be struck between working with team members of the same profession, and working in a cross-functional team. In a team with members from the same professions there is a better opportunity to learn from team members and develop professionally, which can be particularly valuable for more junior employees. A cross-functional team offers better communication and speed in the project, but often requires more experience and poses a larger risk for diverging standards. Which team is the home team and how do you ensure that neither professional growth or cross-functional collaboration is neglected?

Operations is not just operations

We would also like to highlight that the operations part consists of different tasks and may also consist of different teams. 

• There is a maintenance part that requires a deep understanding of the solutions’ technical sides to make sure bugs fixes, new features, and upgrades are implemented timely. This will require the same technical skills as the original development team, and thus these responsibilities often land on that development team, but these tasks can also be handled by a separate sustaining team. 

• Then there are some operations tasks such as setting up, handling or improving monitoring and alerting for the solutions and data products. These are traditional IT tasks and often have a fairly clear home within existing IT teams 

• And then there are support tasks where some or the incoming questions will require good product and domain understanding. This isn’t IT support, but rather questions that usage experts or super users are best at answering. 

I think different parts of the production phase require quite distinct skill sets, but are often grouped together in the single word “Operations”.

What are your lessons learned from setting up cross-functional teams for Industrial DataOps?

Previous posts in the series

• Customer & Cognite operational responsibilities
• Roles, tasks and skill sets for development and operations