Change Data Capture (CDC) Pattern

Overview

Change Data Capture (CDC) is an architectural and data integration pattern used to detect changes in a database and propagate them to other systems. Instead of repeatedly copying entire tables, CDC captures only the data that has changed, such as inserted, updated, or deleted records.

CDC is commonly used in data synchronization, event-driven architectures, data replication, analytics pipelines, search indexing, and migration between databases. It is not a traditional Gang of Four object-oriented design pattern; it is better classified as an architectural integration pattern.

How Does CDC Work?

The following diagram illustrates how Change Data Capture (CDC) captures database changes from transaction logs and propagates them to downstream systems through a message broker.

CDC Processing Flow

In a typical log-based CDC pipeline, a connector reads the database transaction log, converts relevant operations into change events, and publishes them for downstream applications to process.

The connector also tracks its reading position, commonly called an offset, log sequence number (LSN), or binlog position, depending on the database and connector. After a restart, it uses the saved position to resume reading. Correct checkpoint management is essential to avoid missing changes; duplicate delivery may still occur and should be handled safely.

Components and Responsibilities

CDC Implementation Approaches

1. Log-Based CDC

Reads the database's transaction log, such as a write-ahead log, redo log, or binary log. It can capture committed changes with relatively low impact on application queries. This is a widely used approach for continuous, near-real-time data integration.

2. Trigger-Based CDC

Uses database triggers that execute when rows are inserted, updated, or deleted. The triggers typically write change information into separate audit or change tables, which another process reads and forwards.

3. Query-Based CDC (Polling)

Periodically queries source tables using a timestamp, version column, or another change indicator. This approach is relatively straightforward, but it may not reliably detect hard deletes unless additional tracking is implemented. Its latency depends on the polling interval.

Simple Java CDC Event Processing Example

The following simplified example illustrates how an application might represent and process a CDC event after a connector has captured it. It is not a database log reader or a complete CDC connector implementation.

CDC Event

public class ChangeEvent {

    private String operation;
    private int recordId;
    private String newValue;

    public ChangeEvent(String operation, int recordId, String newValue) {
        this.operation = operation;
        this.recordId = recordId;
        this.newValue = newValue;
    }

    public String getOperation() {
        return operation;
    }

    public int getRecordId() {
        return recordId;
    }

    public String getNewValue() {
        return newValue;
    }
}

Change Event Consumer

public class ChangeEventConsumer {

    public void process(ChangeEvent event) {

        switch (event.getOperation()) {

            case "INSERT":
                System.out.println(
                    "Insert record: " + event.getRecordId());
                break;

            case "UPDATE":
                System.out.println(
                    "Update record: " + event.getRecordId()
                    + ", value: " + event.getNewValue());
                break;

            case "DELETE":
                System.out.println(
                    "Delete record: " + event.getRecordId());
                break;

            default:
                throw new IllegalArgumentException(
                    "Unsupported operation: " + event.getOperation());
        }
    }
}

Using the Consumer

public class CDCExample {

    public static void main(String[] args) {

        ChangeEventConsumer consumer = new ChangeEventConsumer();

        ChangeEvent event =
            new ChangeEvent("UPDATE", 101, "Updated Value");

        consumer.process(event);
    }
}

In a production system, the event would normally be supplied by a CDC connector or message broker. The consumer would also need error handling, schema management, retry policies, and idempotent processing.

Offsets, Checkpoints, and Reliability

A CDC connector must remember how far it has read in the source transaction log. This position is stored as an offset or checkpoint. For example, a database may expose a log sequence number (LSN) or binary log file and position.

A key design consideration is coordinating event delivery and checkpoint advancement. If a connector advances its checkpoint before changes are safely delivered, events could be lost. If it delivers events and fails before saving progress, some events may be delivered again. This is why duplicate-safe consumers and carefully designed recovery behavior are important.

Where CDC is Used