Data Mesh

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

What is Data Mesh?

Data mesh is a new approach for designing modern data architectures by embracing organizational constructs as well as data-centric ones, data management, governance, etc. The idea is that data should be easily accessible and interconnected across the entire business.
Similar to the way that microservices are a set of principles for designing modern software architectures, data mesh is a much newer concept that’s so early in its evolution. While microservices have frameworks such as Spring Boot and Micronaut, along with defined practices, books, and all kinds of other infrastructure, there aren’t many training materials, tutorials, and guides for data mesh and how to use it in action.
While the concept is so new and may seem like an abstract concept at this moment, you’re doing the right thing by learning it now. In this data mesh tutorial, we’ll give you a complete overview of data mesh architecture, how it works, its benefits and use cases, examples, and how to get started.

Four Principles of Data Mesh
To understand how data mesh works, we need to understand its four principles: data marts, domain-driven design (DDD), microservices, and event stream processing. Data mesh has its own unique set of four principles, however. You need to specifically adhere to these principles in order to gain its benefits.
The four principles are all technology agnostic, so they don't confine you to Java or Apache Kafka® or relational databases. A good way to think about data mesh is to frame the problem that it solves: a well-implemented data mesh lets you scale a data architecture in both a technological sense and an organizational sense. So data mesh is flexible in the sense that it lets you add more compute when necessary, but also in the sense that as a business evolves and changes and grows—along with the things that people want out of the data—it can accommodate them.

Data Ownership by Domain
In much the same way that microservices each own a specific business function, in data mesh, data is broken down around a specific business domain. Access to that data is decentralized; there's not a central data bureau, a data team, an analytics team, etc. Rather, there's a place where that data lives, probably next to its functionality, i.e. the microservices that produce the data. Access to the data is granted at that point.
Data as a Product
In data mesh, data is considered a product by each team that publishes it. A team owns data just like a team would own the set of services that implement the slice of the business that they support. That team has to engage in product thinking about the data: They're wholly responsible for the data, including its quality, its representation, and its cohesiveness.
Data is Available Everywhere, Self Serve
All data is available everywhere in the company by self-serve in data mesh. Governance is still an issue, but in principle, data products are published and they are available everywhere. If you're producing a sales forecast for Japan, for example, you could find all of the data that you need to drive that report—ideally in a few minutes. You'd be able to quickly get all of the data you need from all of the places it lives into a database or reporting system that you control.
Data is Governed Wherever it is
As mentioned, governance is still a concern in data mesh, but that concern is ideally driven down to the place where the data is created and produced. No data architecture is ever perfect or perfectly static; things always evolve and grow. Governance allows you to ring fence this, allowing you to trust and more quickly navigate data in the mesh, and believe—subject to governance restraints—that you can use the data you find.

Data Mesh Architecture, Illustrated
Data architectures often lack rigor, and commonly evolve in an ad hoc way with minimal discipline and structure. A typical data architecture may look like this:

Fortunately, applying data mesh can turn such a point-to-point architecture into something more uniform and manageable—a kind of central nervous system for your data:

In fact, a nervous system is a great metaphor for data mesh because it is itself a mesh: Look inside a brain and you'll see a mesh of independent little products connected to one another. Similarly, with data mesh, an event in one part of your organization can immediately trigger an event in another part, since, in an analytics sense, they are all interconnected.
Benefits of Data Mesh
Data mesh solves a number of common problems. At the smaller scale, it addresses many of the issues seen with data pipelines, which often become brittle and problematic over time by creating their own webs and messy point-to-point kind of systems. It also addresses larger organizational issues, such as different departments in a company disagreeing on core facts of the business. In a data mesh, you’re less likely to have copies of facts. In both of these cases, a data mesh can bring much needed order to a system, resulting in a more mature, manageable, and evolvable data architecture.
Organizing Data Products with Nodes
As you've learned, data mesh has organizational and technological dimensions. You can think of a data mesh as a network for exchanging data about a business, about what's happening right now with nodes and with connections.
The nodes in the mesh are data products: a microservice, a database, an application, etc. The nodes produce high quality, locally curated data, within the mesh. Data is curated by the team that's building it for the benefit of the other nodes, and related nodes are grouped into domains (for example, “inventory” or “pipeline sensors”). The data flows from node to node, and thus from domain to domain, as needed.

Data Ownership by Domain
The first principle of data mesh is data ownership by domain, or domain-driven decentralization. This principle simply requires that data be owned by those who understand it best. It's a reaction to the years that most companies have spent doing things the other way: putting everything into a centralized data warehouse.

The Antipattern

To illustrate the data mesh pattern, it's most effective to begin with its antipattern: the data warehouse. A typical data warehouse implementation has many data sources spread across a company, with varying levels of quality. There will be many ETL jobs, which are possibly running in different systems and pulling data sets back to the central data warehouse. The data warehouse teams will often be required to clean up and fix a lot of the data. If you've ever done this kind of work, you know that 95% of the work is cleaning. Extracting and loading takes up the little bit of remaining time.
This centralized approach cuts across domains, or units of business, by optimizing for a common set of technological skills, rather than a set of business skills: The team running the data warehouse usually doesn't understand the data sets very well because their focus is running the data warehouse as expertly as possible. Even if they do have some familiarity with the data, they are never going to know it as well as the domain teams where the data originates.

Another challenge is that the source systems that are feeding data into the data warehouse don't always behave particularly well. They haven't been built with responsible data sharing in mind, since the builders are being pressed to build application features, not thinking about the best possible way to make the data shareable.
So instead of optimizing horizontally as with a data warehouse, you should optimize vertically.
The Decentralized Data Mesh
In a data mesh, ownership of an asset is given to the local team that's most familiar with it—the ones who are intimately familiar with its structure, its purpose, and its value. In this decentralized approach, many parties work together to ensure excellent data. The parties that own the data have the responsibility to be good stewards of that data, and they are explicitly identified and known to the rest of the organization.
A Practical Example of Data Mesh

Imagine Alice, at the top of this diagram, is working on an orders domain. Joe, at the bottom of the diagram, is working on inventory management, and he gets some bad inventory data from Alice.
The first thing Joe does is try to fix the data locally for himself. While he may be successful, the problem is that this only fixes the data for his service. Effectively, by fixing the data locally, he couples his inventory product to his local fixes. So there will be a problem if Alice decides to fix the product on her end after Joe has made these local changes.
A better solution is for Alice to fix the data on her end, which will eliminate Joe's problem. This is for the benefit of not just Joe, but anyone else who might want to consume Alice's data.
So for the data mesh to work, there are a couple of requirements:
There must be a process that allows Joe and Alice to communicate quickly.
There must be a sense of responsibility amongst members of the mesh: Alice needs to be both willing and able to make her changes quickly, which will ensure that her data in the mesh is always of good quality (there should be incentives in place to ensure that this happens).

Respect Language and Nomenclature (DDD Style)
Similar to domain-driven design practices, a standard language and nomenclature should be used for all of the data in your decentralized data mesh. This ensures that your streams of events will create a shared narrative that all business users can understand. (All of the various flows through the mesh should be comprehensible to non-technical people).

Data as a Product
As you begin to try and comprehend data mesh, the second principle, data as a product, is possibly the most important concept to grasp.

Product Thinking
In a data mesh, your domain's shared data is managed as a true product, and your objective is to provide this data in a clean and proper way so that other teams in your organization can use it seamlessly. In other words, you treat other teams as internal customers of your data. Your data isn't just a commodity that you insert, update, and delete in a database until your job is done, but is rather something you're publishing for other people—something you're proud of. Product thinking prevents data chauvinism, i.e. "this is our data and no one else's."

Somewhat counterintuitively, decentralization and granting local autonomy over domains can risk causing even more data silos, which is exactly what you want to avoid. Domains must consider themselves to be servants of the organization in terms of the data they publish.
A Microservice of the Data World
A simple way to think of a data product is as the data equivalent of a microservice: It's a node on the data mesh, situated within a particular domain of the business. The role of a data product is to consume high-quality data within the mesh, and like a microservice, it encapsulates everything it needs for its function. It has the actual data to share with the outside world; the code that creates, manipulates, and shares the data; and the infrastructure to store the data and run the code. It is similar to a microservice, but it's publishing data rather than delivering compute.

The Mesh, Apache Kafka, and Event Streaming
There are several technologies that can distribute data between products in a mesh. Apache Kafka is a natural fit, given the way that it enables data to be easily shared using streams (or topics). Kafka allows any product to consume from the high-quality stream of any other product:

In fact, the data mesh itself is actually quite similar to Confluent's concept of a central nervous system of data, whereby data is continuously flowing, all the while being analyzed and processed. (Keep in mind that the data mesh in the image above is a logical view, not a physical view.)
You may recall the spaghetti-like diagram in What is Data Mesh?, which illustrated a system with many point-to-point data connections. An architecture employing Kafka is much simpler, because event streaming is a pub/sub pattern:

Data producers are decoupled from consumers in a scalable, fault-tolerant, and persistent way. One produces a message to a topic, and one consumes—it's write once, read many. All that a data producer has to do to participate in the data mesh is to publish its public data as a stream, so that other data products can consume it. There are a few naming and schema issues to work out, but overall it's relatively simple.
And Kafka's event streams are real-time; you can propagate data throughout the mesh immediately. They are also persistent and replayable: they capture real-time historical data in one step. Finally, their immutability makes them an ideal source of record and ideal for governance purposes.

Moving Data In and Out of a Data Product
There are three main options for getting data into and out of a data product:
Event Streams. See the previous section for an explanation of why these work well. An additional advantage of event streams is that they continuously import and export data as soon as new business information is available.
Request/Response APIs. You can use these synchronously to request data snapshots on demand (similar to a file export).
Traditional ETL for Batch Import/Export of Nightly Snapshots. Traditional ETL will give you a fixed cadence of importing and exporting nightly snapshots, but compared to option B, it provides less control over the snapshot data you'll be working with.

Kafka Connect for Quick Integrations
You can quickly onboard data from existing systems through Kafka Connect and its library of connectors, allowing you to stream data from your systems into your data products in the mesh, in real time. There are about 200 connectors currently available, so you will rarely have to write anything yourself.

Processing a Data Product's Contents: ksqlDB and Kafka Streams
Generally, only the team responsible for a data set needs to be concerned with its internal processes. Everyone else only cares about the public interface. How that public data is created—implementation details, language data infrastructure, and so on—doesn't matter as long as it meets the criteria of being addressable, trustworthy, and of good quality.
The team responsible for processing the data, however, has two options wholly within the Kafka ecosystem: ksqlDB and Kafka Streams. To learn more about these, see the courses ksqlDB 101, Inside ksqlDB, and Kafka Streams 101.

Sharing Streams in Your Data Mesh: Schemas and Versioning
We’ve seen in this course that data prepared for sharing has far more value to an organization than data treated chauvinistically. To properly share your data, however, you must use schemas to put data contracts in place. This requires some know-how.
When you publish a stream, you should use schemas with appropriate backwards compatibility and forwards compatibility. This ensures, for example, that when you add a new data field to a stream, any existing consumers of that stream can continue to operate without having to make any changes.
However, if you do need to introduce a breaking change to the schema of the stream, use versioned streams. For example, if you're required, moving forward, to obfuscate certain data fields, and this will break existing consumers, publish a new version of your stream and deprecate the original stream once it isn't being used any more.

Note that in practice, there will be situations where a data product is forced to make incompatible changes to its schemas. You can address this using the dual schema upgrade window pattern.
Some Data Sources Will Be Difficult to Onboard
You may find that some data sources are difficult to onboard into a data mesh as first-class data products. This can be for many reasons, both organizational and technical. For example, batch-based sources often lose event-level resolution, rendering events non-reproducible. In such a situation, it is best to use event streaming along with change data capture (CDC), with or without the outbox pattern.
Finally, as mentioned when we discussed the first principle, to better understand data mesh it is helpful to learn more about Domain Driven Design.

Data Available Everywhere, Self-Service
The third principle of data mesh is building a self-service platform. This concerns a problem that you may have faced in your work in the past: how to access the data you need in a reasonable amount of time.
A Real Use Case: A Trade Surveillance System in a Large Financial Institution
A new trade surveillance system was being built in a large financial institution. This system needed data from thirteen different systems:
The company had a messaging system in place so that it could publish events. Many teams used it, publishing events that other teams could receive.
Although there was this real-time event stream, there wasn't a great way to get historical data. The messaging system used a queue, and data just disappeared when it was consumed.
Some of the systems did have a feature where you could submit a request for back population and have historical events republished for you, but most systems didn't have that feature automated.
The current event streams and the historical snapshot file had different schemas, different names for each field, etc. This made it very difficult to get the real-time and historical data to match up.
The end result was that just getting data into the database in a logical way took around nine months.
The third principle, data available everywhere, as a self-service, specifically tackles the problems that the bank faced.
A Central—Highly Available—List of Streams and Their Data
Unlike the other data mesh principles, the third principle is achieved through a central infrastructure. As a rule, decentralization is preferable in data mesh, but in this case it is quite beneficial to implement a centralized service that lists available event streams and the data that they contain.
This isn't centralizing data ownership, data product ownership, domain expertise, and so on; we know that those things are decentralized. This is just an infrastructure layer. Basically, the system needs to manage both real-time and historical data, and make that data available everywhere within the company.
The process of accessing this company data and getting it into your individual database should be autonomous, or at least automatable. Just like with microservices, the most efficient architectures allow you to get the data you need and execute it at your own cadence through rapid data provisioning.

Lambda vs. Kappa Architectures
Consuming data from the data mesh means potentially consuming both real-time and historical data. There are two ways to do this. The first, described in the bank scenario above, is called the Lambda Architecture. It requires you to build two separate systems, one for historical data and one for real-time data.
A better solution is known as the Kappa Architecture. It begins with an event streaming platform that stores the streams indefinitely. Destination data ports, which consume the data, can choose to start with the latest events from the platform, or to consume all historical events first. In Apache Kafka®, you achieve this by doing two things: First you set a topic for infinite retention so that it has all of the history, then you reset your consumers back to offset: 0 to start at the beginning of the topic. In some cases, you can also use compacted topics as an optimization, to reduce the amount of time required to load the initial snapshot. This eliminates historical data, so it’s only for cases where you truly don’t need that data.
Note that if you use something like Confluent Cloud, which has a more efficient mechanism for infinite topic retention, you can store very large data sets indefinitely, in a relatively cost-efficient manner.

Self-Service Platform in Practice
As of the summer of 2021, there is no tool that will automatically implement such a centralized platform for data mesh. This is to be expected, because data mesh is so new. Similar to microservices, the whole point of a data mesh is to be decentralized, so there really shouldn't be only one tool. We would expect many tools to emerge.
But you need a single tool you can use in order to discover the information in the mesh—similar to a service discovery system in a microservice architecture. It could be a UI, a wiki, an API, or something else.
Assume it's a UI. A data mesh UI might look something like this:

You could type search terms and discover schemas. You could introspect the data flowing through the mesh, looking at the various data types and values, and requesting that event streams be joined and translated.
In the image above, orders and customers are selected. One of them is a historical data set and the other is solely real-time events. A stream processing query, in this case delivered by ksqlDB, joins these together, along with a preprocessing step that filters out the events you're not interested in (you're only looking for orders for Platinum clients). The results are deposited in Elasticsearch for you to query.

Database Inside Out
The data mesh is related to another data implementation pattern: the database inside out. If you're not familiar with the term, search for it, as there are a lot of materials covering it.

In the database inside out pattern shown in the image above, Kafka provides the storage layer and ksqlDB provides the query processing (transformations, joins, the ability to create passive views which you can then send to other queries). So you have storage and query processing, and thus a database.
To be clear, this doesn't mean that there isn't anything called a database in your system. Those can still exist. This is just construing the system as one giant custom distributed database.

A Global Mesh
In a data mesh, events can be processed either as pure events or as a queryable view of current state. This point really becomes important as the mesh grows. Imagine that the data mesh becomes global, with different nodes that span continents, where delays are significant and connectivity is less than reliable in some cases—probably still pretty good, but not as fast as inside a data center.
In the image below, you have a destination data port on the left, which has two types of interfaces that make the data in the mesh globally accessible. The first interface, at the top, uses events. The data port decides what data it's interested in, and submits it to the mesh. Then the mesh pushes the events into the data port as they become available. Those events can then be persisted by the data port in whatever way suits them best (into a database, S3, etc.).

The second implementation is query-based, rather than event-based. Here, you tell the data mesh to materialize a view, because that's what you need. You don't want a sequence of events; you want something you can look up using a key. For example, you might materialize a view of orders that are for a particular customer, which may be queryable by order ID or by customer. You can now interact with the mesh using a request-response paradigm.

When a global data mesh is implemented with Apache Kafka, the mesh is really one logical cluster made from many physical ones spread around the globe. While the data product can send data to or receive data from the mesh, if it's playing the role of a destination port, it typically can't use the underlying Kafka implementation, since that sits under central control. Instead, it should use its own Kafka instance. This pattern is simple to implement on the cloud using systems such as Confluent Cloud, where using additional standard clusters introduces very small marginal costs.
So that's the third principle of data mesh: making data available everywhere in a self-serve manner. Hopefully you’ve seen enough implementation detail to start to make the idea concrete: These are basically messages, produced into Kafka topics and consumed from Kafka topics.

Data Governed Wherever It Is
As with the first principle, data ownership by domain, federated governance is largely an organizational concern. The objective is to ensure that independent and autonomous teams, which own all of the data products within the mesh, can actually work together. You want to make sure that the various constituents of the mesh actually "mesh," so that you can generate real network effects and create value.
Predominantly, this has to do with creating global standards and applying those across the full mesh. So although you started with decentralization in the first and second principles, you now want to strike the right balance between decentralization and centralization in order to tie everything together. This isn't easy, and in practice, the execution will likely differ from organization to organization.

Example Standard: Data Contracts, Schemas, and Schema Registries
One important task is to ensure that the same data or data attribute is represented identically across domains. Otherwise, you couldn't join or correlate data (for example, customer data) across various domains in the company (since everyone has a different view of the customer). Data contracts, schemas, and schema registries will help you to implement and enforce the standards for the mesh.

Example Standard: Error Detection and Recovery
Another issue is how data products should detect and recover from error. Here, you can use common strategies such as application logs, data profiling, and data lineage to unearth errors in the mesh. Streams are very useful in this context, because they capture live and historical data in a sequence of events, which lets you identify cause and effect relationships much more easily (particularly if you can join and correlate streams from different domains). Streams also let you recover and fix errors by replaying and reprocessing historical data.

Example Standard: Tracking Data Lineage
There are a lot of streams in a data mesh and they may span across systems, technology platforms, data centers, clouds, and so on. For the purpose of tracking data lineage, you ideally want to cover the full mesh. Event streaming is a key technology for implementing this in practice, because it allows you to track data in motion all the way from its origins, through to its intermediate steps, to its final destination.

Recommendations for Federated Governance
You need the right mindset: Be pragmatic, and don't expect the governance system to be perfect. It only needs to be "sufficient" (a word whose definition is often answered by legal counsel). There will always be parts of a data architecture that aren't covered or that were just recently changed.
Governance is more of an ongoing organizational process than a technology.
There are things that you will need to centralize, but beware of centralized data models, which can become too slow to change. Use processes and tooling such as GitHub to collaborate and change quickly.

Implementing a Data Mesh
This final module covers the typical journey you might go through when implementing a data mesh in your organization. It won't cover the technical nitty-gritty details (code, APIs, etc.), but will rather give architectural guidance.
You can view the four principles of the data mesh, covered in previous modules, as an evolution:

For example, you wouldn't start with Principle 3, data available everywhere, as a self service, rather, you'd start with Principle 1, data ownership by domain. You'd move from there to Principle 2, data as a product, and so on. Each principle generally features an increasing level of difficulty, but as you move through them, your capabilities will develop as well.
Data mesh has a concrete implementation, but it is more importantly a set of ideas. Reducing those ideas to practice by building out the mesh is a journey. So if you get started now, you're probably not going to have a data mesh next week. But there will come a point when you can reasonably say "I've built a data mesh." There may not be an obvious threshold, but you will recognize when you've reached that point.
First Steps
To begin a data mesh, you first need to get the necessary management commitment. Then start with some concrete initial use cases: Ideally, things that are contained, simple, and owned by high-capability, forward-looking teams. You also want them to be visible, i.e., you want results that you can show to the business.
While data mesh is a valuable concept, it's not everything. It works in conjunction with other important systems such as microservices and domain-driven design, as we've mentioned. So those other methods are most likely going to need to be a part of your work, alongside and sometimes even orthogonal to data mesh. Basically, you should apply the data mesh concepts as you see fit to gain the maximum benefit for your company.
Concrete Steps to Implement a Data Mesh in Practice
Centralize data in motion. Introduce a central event streaming platform; Kafka and Confluent Cloud are good solutions.
Nominate data owners. You should have firm owners for the key datasets in your organization, and you want everyone to know who owns which dataset.
Publish data on demand. You can store events in Kafka indefinitely, or they can be republished by data products on demand.
Handle schema changes. Owners are going to publish schema information to the mesh (perhaps in the form of a wiki, or data extracted from the Confluent Cloud Schema Registry and transformed into an HTML document), and you need a process to deal with schema change approval.
Secure event streams. You need a central authority to grant access to individual event streams. There are probably regulatory concerns here—perhaps even actual laws.
Connect from any database. There are source and sink connectors available for many supported database types, and you should make sure that your desired connectors exist so that you can easily provision new output ports and input sources.
Make a central user interface for discovery and registration of new event streams. This can be an application you create, or even a wiki. Ultimately, you're going to need to support searching for schemas for data of interest. You also need to support previewing event streams and requesting access to new event streams, and you need to support data lineage views.
This course is being recorded early in summer 2021. If you're working on a data mesh implementation around then, you are on the leading edge. Five years from now, we are likely to look back on this time as the early days of data mesh. We will probably consider its early adopters to be risk-taking visionaries, who paved the way for the frameworks and products built around data mesh that we'll be using in 2026. So if you're blazing this trail, you really are making a significant contribution to the community. We appreciate your efforts and look forward to hearing about the things you build.

Unstarred

Dunk (exclude from study)

Data Mesh

Review Questions

Todos

Sensemaker