Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
If you are curious on how to get the most out of your Cognite Data Fusion subscription, you have come to the right place. This is part of a series of posts where we share some of our experience from working with customers in their journey towards an Industrial DataOps organization. We want to share lessons learned, mistakes made, good practices observed, and observations of pitfalls and risks. This is not the absolute truth, but hopefully a way to spark good discussions around an inherently complex topic!
To quickly introduce ourselves, we are
To learn more about Cognite Data Fusion, we recommend this post. Knowing which roles you need for your Industrial DataOps organization, as covered in previous posts, is only one part of setting it up. You also need to consider how to place the different roles and the different implications it has for governance and operations, as well as for the initial development.
General operating model
Three possible general types are centralized, decentralized, and hybrid organizations. These are taken from models for digitalization organizations. As seen in the following examples, these rarely exist in their pure form in reality, but organizations tend to lean more heavily towards one of them. Sometimes these are active and conscious choices, but it is important to note that they are heavily influenced by the organization’s history, culture, current organizational dynamics, and other factors. An organization may not use the same operating model throughout its existence, but can change and adapt the model as the organization evolves or grows larger.
The examples are from customers we have worked with so far, but have been modified, simplified, and anonymized to illustrate some key points. Also note that some roles, such as the data product owner, were not defined consistently, or at all, at the time some of these collaborations started. We have made an attempt to illustrate where we think this role belongs.
Centralized operating model - somewhere to start
We typically see centralized operating models when the organization is small or, in some cases, in early phases where the initiative is driven by a digital office with varying involvement from the business units.
It is more often than not the realistic choice, rather than the strategic one, for smaller companies where business units are small and deep technology resources are scarce. The size of the organization also makes the communication with the operations team easier and the orchestration more manageable. To reduce the risk of disconnect between operations and digital development, we see that the involvement of SMEs in the digital team and very strong communication between stakeholders in operations and IT has been critical. In this example, the roll-out of new solutions is managed by line managers and a lot of enthusiasm and motivation has been seen on the business side.
We commonly see outsourcing in these scenarios as well, which can mitigate the problem of retaining some types of technical talent while also increasing the need for centralization to manage requirements towards consultants.
Here, the SME in the digital organization currently takes the role most similar to a data product owner.
In terms of operations, this is a devOps team of sorts, where development and operations are handled by the same team.
Hybrid operating model - mixing it up, scaling it up
Among the hybrid operating models, there is a flavor that leans more towards the centralized side with a central data liberation team. Most organizations that are slightly larger naturally move towards a more hybrid operating model. This model counters the disconnect between the development team and the business units by placing more responsibility and ownership in these units. This also improves the speed of development from the business units’ perspective. On the other hand, the coordination challenges need to be considered.
Some other challenges to consider are how to handle support, maintenance, and operations. For example, some core data products may be managed and owned by the central digital team, while the platform admin/IT team are responsible for some standard integrations towards business and OT systems, and the ownership of a solution data product lies with the business unit. During development, members from these teams may work together in a virtual team to leverage agile methods and speed up delivery. In the operations phase, the agile methodology is likely replaced by good instrumentation, clear lines of responsibility, and good processes for quickly identifying the right team to resolve an issue or answer a question.
Another challenge can be how to align the central digital team with the platform administration/IT team illustrated in the image. Oftentimes, a separate digital team drives executive ownership and ensures that there is a focus on progression. The motivations for such a setup are numerous, and include a different cost control focus and perspective on risk and agility. While this makes sense for speeding up delivery, it can make it harder to establish good operations. Some potential alternatives are to build up a shadow organization within the digital team that has the responsibility to operate the deliverables or hand over a set of tools and solutions to the IT team, which entails transferring a lot of knowledge and may require rework if the artifacts produced do not follow the standards expected by the IT team.
We believe that the organization should strive to have data product owners in the business units as much as possible. The exception may be company wide source data products or core domain data products where a data product owner from a single business unit may not have the overview required.
There is also a flavor of hybrid operating models that leans more towards decentralization by having local data liberation teams in each business unit. While this can be an active choice, some organizations are also pushed in this direction by regulations on data export or GDPR. Therefore, it is not uncommon to see some version of this in multinational organizations, particularly in the energy sector.
In this example, the central digital team and IT team retains the platform ownership and governance, while the business units take full ownership of their deliveries. Some data products and integrations are still owned centrally, but more of the technical responsibility lies with the business units, for example integration with OT and IT systems and development of custom extractors for those when needed.
Operations in this model may be slightly easier as the business units take almost full ownership and are less dependent on technical resources from other teams. Creating data products and data models that work well together and are useful across the company can be more challenging though.
In this model, the data product ownership has a strong home in the business units, with the potential exception of more generic company-wide products.
Decentralized - Large and global
A fully decentralized organization is rare but can be seen with very large multinationals where sufficient decentralization is a must to avoid a significant coordination overhead and the slow-downs that entails. There are still some coordination efforts in this model and central teams for digitalization and IT act as centers of excellence that support the business units, but all the work and responsibilities ultimately lie with the business units. In these cases, the business units may be large enough that they themselves have their own digitalization teams. Additionally, operations and data product ownership is firmly planted in the business units.
There may be multiple levels of platform and application owners to handle local and global questions around data and practices. The challenges around enabling synergies from the different initiatives in the company will likely be significant, but also have the potential to be significantly impactful if they are successful.
Data liberation teams
A commonality in several of the larger organizations is the existence of one or more data liberation teams. For some reason, the first such team was called a “chapter” and this terminology has stuck to some of our material. This is a team that is responsible for the technical handling of the data pipelines, including integrating data, and ensuring quality and monitoring of the pipelines. The team members are not the experts on the data or the domain model itself, but work to create a good foundation for the data product owner and other subject matter experts to govern and develop the data models.
Assigning a data liberation team often has the benefit of making it easier to attract the right talent when they can work with peers in the same field. It also improves the development and implementation of good practices. The skillset is slightly different from what you find in the typical IT organization, but if you have a dedicated data organization, that may be where you find these skills. Depending on the size of your organization, you may want to consider how much having parallel teams makes sense and to what extent you can leverage the IT experience of service and processes related to that.
A very real issue is that a disconnect between the data liberation team and the business unit can easily appear. Although there likely isn’t a silver bullet for this, we believe that a strong data product manager from the business unit is an important component in reducing such a disconnect.
We would really like to learn more from how you have organized to make sure the communication works well between business units, IT and and digitalization offices. Or have you organized in an entirely different way?
Previous posts in the series