Change Data Capture

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

The problem with most databases’ replication logs is that they have long been considered to be an
internal implementation detail of the database, not a public API.

For this reason it was difficult to take all the changes made in a database and
replicate them to a different storage technology such as a search index, cache, or data warehouse.

CDC is the process
of observing all data changes written to a database and extracting them in a form in which they can
be replicated to other systems.

CDC is especially interesting if changes are made available as a
stream, immediately as they are written.

For example, you can capture the changes in a database and continually apply the same changes to a
search index.

If the log of changes is applied in the same order, you can expect the data in the
search index to match the data in the database. The search index and any other derived data systems
are just consumers of the change stream,

We can call the log consumers derived data systems

the data stored in the search index and the data warehouse is just another view
onto the data in the system of record.

Change data capture is a mechanism for ensuring that all
changes made to the system of record are also reflected in the derived data systems so that the
derived systems have an accurate copy of the data.

Essentially, change data capture makes one database the leader (the one from which the changes are captured), and turns the others into followers.

A log-based message broker is well suited for transporting the change events from the source database to the derived systems, since it preserves the ordering of messages

Database triggers can be used to implement change data capture (see “Trigger-based replication” ) by
registering triggers that observe all changes to data tables and add corresponding entries to a
changelog table. However, they tend to be fragile and have significant performance overheads.
Parsing the replication log can be a more robust approach, although it also comes with challenges,
such as handling schema changes.

Like message brokers, change data capture is usually asynchronous: the system of record database
does not wait for the change to be applied to consumers before committing it. This design has the
operational advantage that adding a slow consumer does not affect the system of record too much, but it
has the downside that all the issues of replication lag apply

if you don’t have the entire log history, you need to start with a
consistent snapshot,

The snapshot of the database must correspond to a known position or offset in the change log, so that you know at which point to start applying changes after the snapshot has been processed. Some CDC tools integrate this snapshot facility, while others leave it as a manual operation.

Log Compaction and CDC

Unstarred

Dunk (exclude from study)

Change Data Capture

Review Questions

Todos

Sensemaker