Why Data Mesh is all the rage now

I have always loved building machines, the more autonomous the better, and building the machine we do now feels like kicking off a game of Life (cellular automata) and see how it behaves – did we give it the right conditions to grow into something fantastic on its own, bigger than us?

Let me explain in short how I see the Data Mesh concept and why it has now become one of the hottest trends in the data space.

The Data Mesh idea is as simple as it is powerful. It is about distributing both the ownership and the technical development and operations of data domains out to the source locations in the company, rather than the previous paradigm of centralized data warehouses and data engineering teams.

 

With the mesh, there is no centralized bottleneck really. Instead, a growing number of teams serve their domain data sets in well-shaped packages, from people who really know the business domain and the systems behind the data. Without getting caught in a silo nightmare.

A team in a mesh would also present that data in the most useful and relevant technical ways and shapes, not just the old batch export of transactional data for DW teams to figure out. Data warehouses have been great for the last two decades, but they won’t be enough looking forward. 

Data is everywhere now and with a mesh, the use of it really, really scales much better with the fast pace of modern business and the online context.  

I think data mesh is a perfect fit to enable the teams to add data centric capabilities to their products here, and for anyone in analytics to really explore and find insights in the vast data resulting from the complex business realities. 

Considering this, this is where our team, Data Enablement Platform, comes in as an enabler of doing the data mesh. We do this with a self-service approach to both publishing and consuming data, as well as some tooling and resource provisioning around that. It is a dream concept for someone who has been in classic data engineering for a long time! 

What are my initial lessons learnt from applying the Data Mesh concept in practice? 

This is not easy! It is certainly a challenge when you are a 70+ years old company with so much established, things that actually work already, and in this huge scale. It challenges a lot of truths, but it also opens amazing potential to be faster and smarter in the long run.

But what happens if we start distributing bits and pieces from this massive data solution we have, to autonomous ownership here and there? And if we scale out and distribute our mesh so much that data does not even live in the same place as other data? How do we agree on data models, identifiers, and all other practices that concerns everyone? We do not have all answers yet, but we can’t re-do the same things over again either, and we learn something new every day.

Everything, in the end, is up in the air. But I think that’s the way it has to be.