Overview

The Polling Publisher is a message publication approach used alongside the Transactional Outbox pattern. It implements a message relay component that periodically reads messages from the OUTBOX table (or attribute) and publishes them to a message broker.

Problem

The Transactional Outbox pattern persists messages together with database updates to guarantee atomicity. However, persisting the message in the OUTBOX is only the first step. A separate mechanism is required to read those messages and actually publish them to a message broker so that other microservices can receive them.

Solution

The Polling Publisher implements a component called a message relay. The relay is responsible for: reading the data from the OUTBOX table, converting and publishing the messages to the message broker, and then atomically deleting the processed messages from the OUTBOX.

Relational Databases

In a relational database, the message relay polls the OUTBOX table at regular intervals. For each message found, it converts it, publishes it to the message broker, and deletes it from the OUTBOX in a single atomic transaction. This ensures messages are not lost and are not published more than once.

Non-relational Databases

Applying the Polling approach to non-relational databases is more challenging, because it depends on the read capability of the database to search for pending messages across all records. This scan-based approach can be inefficient at scale, making Transaction Log Tailing a preferable alternative for non-relational databases.

Technologies

Technologies that support the Transactional Outbox and Polling Publisher approach include Eventuate Tram (developed by Chris Richardson), which provides APIs for message publication using both Polling and Log Tailing approaches. It supports MySQL and PostgreSQL with Apache Kafka as the message broker.

Trade-offs

Benefits

  • Simple to implement with relational databases that support OUTBOX tables
  • Decouples the publication timing from the original transaction
  • Works with any message broker

Drawbacks

  • Polling introduces latency — messages are not published instantly, only at poll intervals
  • Challenging and inefficient for non-relational databases (prefer Transaction Log Tailing)
  • The polling frequency must be tuned to balance latency and database load

When to Use

Apply Polling Publisher when:

  • You are using a relational database with a Transactional Outbox table
  • Small publishing latency is acceptable
  • You do not have access to database transaction logs

Prefer Transaction Log Tailing when:

  • Using a non-relational database — polling across all records is inefficient
  • Very low publishing latency is required

References