Data Management at Scale

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

The Disruption of Data Management

Summary:
Data Warehouses
Data Warehouses are here to stay. The need to harmonize data from different sources within a particular context will always remain
Big silos like enterprise data warehouses will become extinct because they are unable to scale. Tightly coupled integration layers, loss of context, and increased data consumption will force companies to look for alternatives. 
Data Lakes are here to stay
The need to process vast amounts of data in a distributed fashion for analytics will remain
Raw, polluted data, which can change any time, will prevent experiments and use cases from ever making it into production. Raw data itself carries a lot of repeatable work with it. 
Scaled Architecture
simplifies and integrates data management across the entire organization in a distributed fashion. 
a consistent view of how to uniformly apply security, governance, master data management, metadata, and data 
an architecture that can work using a combination of multiple cloud providers and on-premises platforms but still gives you the control and agility you need. 
It abstracts complexity for teams by providing domain-agnostic and reusable building blocks but still provides flexibility by providing a combination of different data delivery styles using a mix of technologies.

The Scaled Architecture

Managing Vast Amounts of Data: Read-Only Architectures

Services and API Management: API Architecture

Event and Response: The Streaming Architecture

The Streaming Architecture borrows its name from streaming data or event stream processing . These are continuous real-time processing and analyzing of generated data from various sources.

Advantages of Streaming Data

Real-time streaming data provides a competitive advantage: quicker responsiveness leads to higher customer satisfaction. Faster results make insights more relevant. Instead of waiting minutes or hours, you can react immediately. For example
Fraud can be detected the moment it happens. 
You can see where your web visitors are coming from, interact with them, and improve their customer experience as they stay on your website.

Data delivered in a stream of events can be of enormous value to organizations, and its sources can vary hugely, including Internet of Things (IoT) devices, smartphone clicks, in-game player activity, social networks, and changes to traditional operational systems.

Differences in thinking: Listening instead of Talking. Listening instead of Scheduling (Batch)

Unlike offline batches and APIs, streaming data focuses on interacting with and reacting to events. It is about asynchronously connecting applications and systems. Streaming emphasizes high throughput and can be either stateless or stateful. It allows you to exchange information about changes that occur at specific points in time what we call events and to move the application state between locations, which enables distributed, up-to-date, materialized views.

Naturally, these streaming patterns require very good data management. For example, one event may trigger a slew of messages or calls to many different services. From a data management standpoint, you really want to track which answer belongs to which original message.

The big difference between the API Architecture and the Streaming Architecture is the Streaming Architecture’s inverted conversational nature. You listen instead of talk. This is also where its complexity lies because most users aren’t familiar with this way of thinking.

Modern Messaging Platforms Advantages

modern messaging platforms scale up the asynchronous model with their distributed (big data) and fault-tolerance capabilities. 
Distributing the compute and data over many nodes means that if some nodes fail, others will take over. 
This replication gives another advantage: scalability for parallel processing. Each time you double the number of machines in a distributed architecture, you double the amount of compute/processing. 
Longer-term persistent storage is also where streaming platforms differ from traditional queueing systems. Their distributed nature and immutable, append-only file systems make it possible to retain an immense number of messages for a long time.

Connecting The Dots: Relationship Between the Architectures

Business requirements always come first. Turning data into insights or actions requires understanding how information flows and using that value to identify business opportunities.

Use cases may start as one-off but ideally you’ll turn your key data into maintainable solutions that deliver constant value for the organization. Depending on your business requirements, you might use different techniques, such as business intelligence, real-time decision making, or machine learning.

Then there are nonfunctional requirements. It takes many different database technologies to accommodate a large variety of complex use cases. Depending on the use case, you might pick one or several of these. Additionally, there are variations in the types of transformations, their velocities, optimizations for parallelization, and consumption patterns, all of which affect your end result.

the differences between self-service data and managed data, the nonfunctional considerations for selecting data stores, and the building blocks and principles for business intelligence and advanced analytics.

two consumption patterns for a better understanding of the integration challenge.

The first consumption pattern is directly using read-only data stores (RDSs). The RDSs, as described in Chapter 3 , are positioned to serve out larger volumes of immutable data repeatedly to consumers. Because of their performance and access patterns, they can be ideal candidates for direct reporting, ad hoc analysis, analytics, and self-service activities. In this pattern, data is read but no new data is created.

The big benefit of this model is that it does not require creating new data models. You don’t extract, transform, and load data into a new database. Transformations happen on the fly, but these results don’t need a permanent new home. This pattern of direct consumption requires RDSs to deliver adequate performance, so intelligent-based and consuming applications must be able to process data directly.

The problem, however, is that the consumers’ needs can exceed what RDSs offer. In some cases there is a clear need for new data creation: for example, complex data transformations followed by analytical models that generate new business insights. To preserve these business insights for later analysis, you need to retain this information somewhere, for example, in a database.

One more situation would be when multiple RDSs need to be combined and harmonized. This typically requires orchestrating many tasks. Making users wait until all of these tasks are finished will negatively affect user experience. These implications bring us to the second pattern of data consumption: creating domain data stores.

Strengholt, Piethein. Data Management at Scale (p. 308). O'Reilly Media. Kindle Edition.

Sustainable Data Governance and Data Security

Turning Data into Value

Mastering Enterprise Data Assets

Democratizing Data with Metadata

Conclusion

Unstarred

Dunk (exclude from study)

Data Management at Scale

Review Questions

Todos

Sensemaker