Overview

Transaction Log Tailing is a message publication approach used alongside the Transactional Outbox pattern. Instead of polling the OUTBOX table, it implements a Transaction Log Miner component that reads the database's commit logs to detect OUTBOX insertions and publish them to a message broker.

Problem

While the Polling Publisher approach works well with relational databases, it can be inefficient for non-relational databases because it requires scanning all records to find pending messages. A more efficient mechanism is needed — one that reacts to database changes as they are committed rather than scanning periodically.

Solution

The Transaction Log Tailing approach implements a component called a Transaction Log Miner. Through the commit logs present in relational databases, it identifies insertions in the OUTBOX table, converts them into messages, and publishes them to a message broker.

Non-relational Databases

Applying Transaction Log Tailing to non-relational databases is more efficient than the Polling approach, because it does not need to scan all records to find pending messages. It reacts to change events captured from the database's change data capture (CDC) mechanism instead.

Technologies

Several technologies support the Transaction Log Tailing approach:

  • Debezium — an open-source tool that publishes database changes to the Apache Kafka message broker
  • LinkedIn Databus — an open-source project that analyses Oracle transaction logs and publishes change events
  • DynamoDB Streams — a feature that maintains an ordered sequence of changes made to a DynamoDB table in the last 24 hours
  • Eventuate Tram — developed by Chris Richardson, it resolves some limitations found in Debezium for this context, providing APIs for messages using both Log Tailing and Polling approaches for MySQL, PostgreSQL and Apache Kafka

Trade-offs

Benefits

  • More efficient than polling — reacts to database commits rather than scanning periodically
  • Better suited for non-relational databases
  • Lower latency — messages are published closer to when the commit occurs

Drawbacks

  • More complex to implement and configure than Polling Publisher
  • Requires access to database transaction or commit logs, which may not be available in all environments
  • Dependent on database-specific CDC mechanisms

When to Use

Apply Transaction Log Tailing when:

  • Using a non-relational database — log tailing is more efficient than polling across all records
  • Low publishing latency is required
  • Database transaction logs or CDC streams are accessible

Consider Polling Publisher instead when:

  • Transaction logs are not accessible in your environment
  • You are using a simple relational database where polling is straightforward

References