[EN] Das Outbox Pattern: Nie wieder Datenverlust in Microservices

In our previous articles about SAGA and CQRS we have tacitly assumed one thing:

“The service stores the order in the database AND sends an event to the message broker.”

Sounds trivial, right? Here is the typical code you see in hundreds of tutorials:

@Transactional
public void createOrder(Order order) {
    // 1. Speichern in der DB
    orderRepository.save(order);
    
    // 2. Event senden (z.B. RabbitMQ)
    eventPublisher.publish(new OrderCreatedEvent(order.getId()));
}

But this code is a ticking time bomb in a distributed system. It suffers from the so-called Dual Write Problem.

The Problem: The Dual Write Dilemma

We are trying to keep two different systems (database and message broker) in sync without using a slow and blocking “distributed transaction” (like 2-phase commit / 2PC). This can go wrong in two ways:

1. Scenario A (DB Commit, Broker Fail): The order is saved, the database transaction commits successfully. But exactly at the moment when we want to send the event, the broker is not available or the network goes down.
   • Result: The order is there, but no one knows about it. The SAGA stops running, the CQRS dashboard does not update. The system is inconsistent.
2. Scenario B (Event Sent, DB Rollback): We flip it and send the event first. That works. But then the database throws an error (e.g. constraint violation) when saving and rolls back the transaction.
   • Result: The “order created” event haunts the system and triggers follow-up effects even though the order does not exist.

We need a way to make database operation and event dispatch atomic. Either both happen, or neither.

The solution: Transactional Outbox Pattern

The solution lies in using what we already have: the ACID transaction of our local database.

Instead of sending the event directly to RabbitMQ or Kafka, we store it in the same database transaction in a special table: the OUTBOX.

The process then looks like this:

1. Transaction START
2. INSERT INTO orders ... (The business data)
3. INSERT INTO outbox ... (The event as JSON payload)
4. Transaction COMMIT

Since both steps run in the same DB transaction, we have the guarantee: When the order is saved, the event is also saved. If the order fails (rollback), no event is written to the outbox.

How does the event get to the broker? (The Relay)

Now the event is safely in our outbox table. A separate process (the Message Relay) must now read it and send it to the broker.

There are two common approaches to this:

1. Polling Publisher (Simple & Robust)

A small background job (e.g. with @Scheduled in Spring Boot) queries the table regularly:

1. SELECT * FROM outbox WHERE processed = false
2. For each entry: Send to Broker.
3. If send successful: DELETE FROM outbox (or mark as processed).

This approach is easy to implement and is sufficient for many use cases.

2. Transaction Log Tailing (High Performance)

For systems with extremely high throughput, tools like Debezium are used. These tools directly read the transaction log of the database (e.g. MySQL Binlog). As soon as an entry is written to the outbox table, Debezium detects it and automatically pushes it to Kafka. This is more performant, but more complex to set up.

Implementation with Spring Boot (polling approach)

Let’s implement the simple polling approach.

Step 1: The Outbox Entity

We need a table to cache our events.

@Entity
@Table(name = "outbox")
public class OutboxEvent {
    @Id
    @GeneratedValue
    private UUID id;

    private String aggregateType; // z.B. "Order"
    private String aggregateId;   // z.B. "123"
    private String eventType;     // z.B. "OrderCreated"
    
    @Lob
    private String payload;       // Das Event als JSON
    
    private LocalDateTime createdAt = LocalDateTime.now();
    
    // Getter, Setter...
}

Step 2: The Service (Atomic Save)

Our business service now saves event and order together in one transaction.

@Service
public class OrderService {

    private final OrderRepository orderRepository;
    private final OutboxRepository outboxRepository;
    private final ObjectMapper objectMapper; // Für JSON Konvertierung

    @Transactional // WICHTIG: Eine Transaktion für beides!
    public void createOrder(Order order) {
        // 1. Business State speichern
        orderRepository.save(order);

        // 2. Event in die Outbox speichern (statt direkt zu senden)
        OrderCreatedEvent event = new OrderCreatedEvent(order.getId());
        
        OutboxEvent outboxEvent = new OutboxEvent();
        outboxEvent.setAggregateType("Order");
        outboxEvent.setAggregateId(order.getId().toString());
        outboxEvent.setEventType("OrderCreated");
        
        try {
            outboxEvent.setPayload(objectMapper.writeValueAsString(event));
        } catch (JsonProcessingException e) {
            throw new RuntimeException("Error serializing event", e);
        }
        
        outboxRepository.save(outboxEvent);
    }
}

Step 3: The Polling Publisher (The Postman)

A separate job collects the letters and deposits them.

@Component
public class OutboxRelay {

    private final OutboxRepository outboxRepository;
    private final RabbitTemplate rabbitTemplate;

    @Scheduled(fixedDelay = 2000) // Alle 2 Sekunden prüfen
    public void processOutbox() {
        // Hole alle ältesten Events zuerst (FIFO)
        List<OutboxEvent> events = outboxRepository.findAllByOrderByCreatedAtAsc();

        for (OutboxEvent event : events) {
            try {
                // Sende an Broker
                rabbitTemplate.convertAndSend("exchange", event.getEventType(), event.getPayload());
                
                // Lösche aus Outbox nach Erfolg
                outboxRepository.delete(event);
                
            } catch (Exception e) {
                // Fehlerbehandlung: Einfach stehen lassen, der nächste Lauf probiert es erneut.
                // Achtung: Hier sollte man Dead-Letter-Logik einbauen, um Endlosschleifen zu vermeiden.
                System.err.println("Failed to send event: " + event.getId());
            }
        }
    }
}

Note: In a real production system, locking would be installed here (e.g. with ShedLock) so that not all of them send the same entry in multiple instances.

Conclusion

The Outbox Pattern is the glue that makes microservice architectures reliable. It guarantees “At-Least-Once Delivery”: Every event is sent at least once. Conversely, this means that the receiver must be idempotent (in case the event is sent twice because the DB delete fails after sending), but we never lose data.

Without an Outbox Pattern, Event-Driven Architecture is a gamble. With Outbox Pattern it becomes solid engineering.