Kafka to Database: Why a Message Gets Dead-Lettered or Hits a Duplicate Key (Event Hubs)

October 12, 2026 · How It Actually Works: Real Systems as State Machines (part 19)

▶ Watch on YouTube & subscribe to The Stack Underflow

A message lands in your dead-letter topic, and you don’t know why. Or worse: an order that is already in the database shows up in the dead-letter topic too, with a duplicate key error. The video follows six order messages from Azure Event Hubs, through a Spring Boot app, into two database tables. Each one is stored, dead-lettered, or turned into a duplicate key, and this page gives the exact rule or default behind every outcome.

The one-line version: by default, bad data (a message that can’t be read or doesn’t validate) is not retried; in this app it goes straight to the dead-letter topic. A database error is retried, then dead-lettered. And because delivery is at least once, a crash between the database commit and the offset commit brings the same message back, where a plain insert hits the primary key.

Last verified against the Spring for Apache Kafka 4.1.1 reference, the Spring Framework 7.0.9 javadoc, the Apache Kafka 4.3 documentation and Microsoft Learn: 1 October 2026. Scope: Spring for Apache Kafka 4.1.1 with a record @KafkaListener, reading Azure Event Hubs through its Kafka endpoint, AckMode RECORD, database writes in one transaction with plain inserts. Not covered: Kafka transactions or exactly-once, batch listeners, the Event Hubs SDK (the follow-up video uses it). The database product isn’t named.

Partitions and offsets in two lines

• Messages with the same key land in the same partition, in the order they arrived, as long as the partition count doesn’t change. (The partition comes from a hash of the key; the video gives the partition numbers rather than computing them.)
• Each message has an offset: its position in that partition. The consumer saves its position after processing a message, so it can carry on from there after a restart.

The setup

order-service reads the orders event hub (4 partitions) through the namespace’s Kafka endpoint, as consumer group order-writer. An event hub is a Kafka topic here. Every message goes through the same steps:

1. Receive a message from one partition, at one offset.
2. Deserialize the JSON into an OrderEvent object. Unknown fields are ignored. An ErrorHandlingDeserializer wraps the JSON deserializer, so a message that can’t be read is handed to the error handler.
3. Validate the required fields. A Validator is set on the listener registrar (where the app configures its Kafka listeners), so @Valid runs. (This isn’t automatic in Spring for Apache Kafka: the app has to add it.)
4. Map the values to columns.
5. Write all rows in one database transaction, with plain inserts.
6. Retry a failed step: Spring Kafka’s DefaultErrorHandler retries 9 times out of the box (with no delay, so 10 attempts); this app sets 2 (3 attempts).
7. Dead letter: after the last retry (or at once, for errors Spring never retries), the DeadLetterPublishingRecoverer publishes the message to orders-dlt, on the same partition as the original.
8. Commit the offset after each record (AckMode RECORD; Spring’s default is BATCH).

The tables

So one message becomes one orders row plus one order_items row per entry in items. customer.email is in the message, but no column stores it.

Message 1 · A normal order: stored

Partition 1, offset 1041, key cust-17:

{
  "orderId": "ord-1001",
  "customer": { "id": "cust-17", "email": "ana@example.com" },
  "placedAt": "2026-10-01T09:30:00Z",
  "currency": "EUR",
  "total": 42.5,
  "items": [
    { "line": 1, "sku": "MUG-01", "qty": 2, "price": 12.5 },
    { "line": 2, "sku": "TEA-07", "qty": 1, "price": 17.5 }
  ]
}

The JSON becomes an OrderEvent. Every required field is present, including each item’s line, SKU, quantity and price. In one transaction the app inserts 3 rows: orders (ord-1001), and order_items lines 1 and 2. Commit. The email is not stored.

Only then does the consumer commit the offset: the group’s next message on partition 1 is 1042.

Why: with AckMode RECORD, Spring commits the offset “when the listener returns after processing the record”. The container does the committing itself, so it sets Kafka’s enable.auto.commit to false unless you override it.

Message 2 · Broken JSON: dead-lettered, no retries

Partition 2, offset 877, key cust-42. The body stops halfway:

{"orderId": "ord-1002", "customer": {"id": "cust-42"}, "items": [{"line": 1, "sku": "MUG-01"

It can’t become an OrderEvent. The ErrorHandlingDeserializer catches the failure, so the container sends the record to the error handler instead of the listener. A deserialization error is not retried: the same bytes would fail the same way every time. The DefaultErrorHandler goes straight to recovery, and the recoverer (the step that handles a record once retries are over; here it publishes to the dead-letter topic) publishes it to orders-dlt on the same partition, 2, and the offset is committed: next is 878.

Why:

• With an ErrorHandlingDeserializer, a failed record carries a DeserializationException in a header, the container’s error handler is called, and “the record is not passed to the listener”.
• DeserializationException is on the DefaultErrorHandler’s list of fatal exceptions (errors it never retries): retries are skipped and “the recoverer is invoked on the first failure”.
• The dead-letter record keeps the original value that failed to deserialize, so you can inspect the raw bytes in orders-dlt.

Message 3 · A field the app doesn’t know: stored, and the field is gone

Partition 1, offset 1042, key cust-17. The message has a giftWrap field that OrderEvent doesn’t have:

{
  "orderId": "ord-1003",
  "customer": { "id": "cust-17" },
  "placedAt": "2026-10-01T09:41:00Z",
  "currency": "EUR",
  "total": 9.5,
  "giftWrap": true,
  "items": [ { "line": 1, "sku": "CARD-02", "qty": 1, "price": 9.5 } ]
}

The JSON deserializer silently drops giftWrap. Validation passes, the app inserts 2 rows (orders and one order_items line), commits, and the next offset on partition 1 is 1043. The order is stored, nothing failed, and nobody is told the gift wrap is gone.

Why: Spring for Apache Kafka’s JSON components use an object mapper with DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES disabled by default. An unknown field fails the message only if the app turns that feature on (and then it is a DeserializationException: not retried, dead-lettered).

Message 4 · An order with no currency: dead-lettered, no retries

Partition 3, offset 310, key cust-08. Everything is there except currency:

{
  "orderId": "ord-1004",
  "customer": { "id": "cust-08" },
  "placedAt": "2026-10-01T10:02:00Z",
  "total": 30,
  "items": [ { "line": 1, "sku": "MUG-01", "qty": 1, "price": 30 } ]
}

The JSON is valid, so it becomes an OrderEvent. Then validation fails: currency is required. A validation error is not retried, because retrying the same message gives the same answer. Straight to orders-dlt, partition 3. Next offset: 311. The database never saw it.

Why: MethodArgumentResolutionException is on the fatal list, and the Spring Framework javadoc describes its subclass MethodArgumentNotValidException as the exception for a @Valid failure.

Message 5 · The app crashes after the database commit: duplicate key

Partition 1, offset 1043, key cust-17, order ord-1005, one item. This is the one that surprises people.

First delivery:

1. Read, validated, mapped. In one transaction the app inserts 2 rows. Commit: both rows are stored.
2. Then the app crashes: after the database commit, before the offset commit. The offset is never committed.

After the restart:

3. The consumer reads partition 1 from the last committed offset, which still points at this message, 1043. The same message is delivered again. Delivery is at least once.
4. It is read, validated and mapped exactly as before. The insert into orders hits a row with the same primary key, ord-1005: duplicate key. The transaction rolls back, so no rows from this attempt are kept.
5. A duplicate key won’t go away on a retry. But it isn’t on Spring’s list of errors it never retries, so the DefaultErrorHandler runs the listener again. Attempt 2: fails. Attempt 3: fails. The retries are used up.
6. The recoverer publishes it to orders-dlt, partition 1, and the offset is committed: next is 1044.

So the order ends up in two places: its rows are stored from the first delivery, and the message is in the dead-letter topic from the second.

Why:

• Apache Kafka: if the consumer “crashes after processing messages but before saving its position”, the next process receives messages that “will already have been processed”. That is at-least-once delivery, Kafka’s default.
• Event Hubs says the same for its Kafka endpoint: consumers “can receive events more than once”, so the consumer should support “the idempotent consumer pattern”.
• A duplicate key is a database error, and database errors are not on the fatal list, so it is retried like any other.

The fix is to make the write idempotent: with an upsert, a second delivery does no harm. That is the follow-up video.

Message 6 · The database is down: retried, then dead-lettered

Partition 3, offset 311, key cust-08, order ord-1006. Read, validated and mapped as normal. Then the app tries to open a transaction and the database can’t be reached. Nothing is written.

A database outage is worth retrying: the database may be back in a moment. The DefaultErrorHandler runs the listener again, up to 2 more times. In the video the database stays down: attempt 2 fails, attempt 3 fails. The recoverer publishes the message to orders-dlt, partition 3, and the offset is committed: next is 312.

Why: a database error isn’t on the fatal list, so it is retried. After the retries, “the recovered record’s offset is committed”, and the partition moves on. Recovery is what keeps one failing message from blocking the partition; the dead-letter topic is where the message is kept instead of only being logged.

All six messages

Two stored, three dead-lettered, and one duplicate: stored the first time, then dead-lettered when it came back. Cover the “Outcome” column and predicting each one again.

What Spring retries, and what it never retries

The DefaultErrorHandler treats these exceptions as fatal by default: no retries, the recoverer runs on the first failure.

• DeserializationException
• MessageConversionException
• ConversionException
• MethodArgumentResolutionException (and so MethodArgumentNotValidException, a @Valid failure)
• NoSuchMethodException
• ClassCastException

Everything else is retried, including a database outage and a duplicate key.

Two notes on the dead-letter topic. Because the default goes to the same partition, the dead-letter topic must have at least as many partitions as the original. And older Spring Kafka versions named it .DLT; the 4.1.1 reference says -dlt.

Pause & Prove

1. The pinned question

Message 5. The app commits its rows to the database, then crashes before it commits the offset. It writes with plain inserts and has 2 retries. Where does order ord-1005 end up?

In two places. Its rows stay stored from the first delivery; the redelivered copy hits the primary key, is retried twice, and goes to orders-dlt. Walkthrough: Message 5.

2. The Community poll

Spring for Apache Kafka’s DefaultErrorHandler, default settings. Which failed message does it retry?

• Broken JSON (can’t be deserialized). Not retried. DeserializationException is on the fatal list, so the recoverer runs on the first failure.
• An order missing a required field (fails @Valid). Not retried. The validation exception is a MethodArgumentResolutionException, also on the fatal list.
• An order that hits “database unavailable”. ✓ A database error isn’t on the list, so it is retried: 9 times by default, with no delay.
• None of them. The database outage is retried.

Retrying the same bytes gives the same answer; retrying after an outage might not.

Related Shorts

• Broken JSON in Kafka: why Spring doesn’t retry it
• The app saved the order, then the same Kafka message came back
• The order was stored, but the gift wrap field vanished

Before / after this video

• Words first: Primer 7, “Message words”. Already know them? Start here; the video opens with a short recap.
• Next: The same crash, with an upsert, where the second delivery does no harm.

Sources

Read on 1 October 2026:

• Spring for Apache Kafka 4.1.1, Handling exceptions: DefaultErrorHandler default FixedBackOff(0L, 9), the default recoverer logs, the fatal exception list, <originalTopic>-dlt on the same partition, the recovered record’s offset is committed, the original value restored in the dead-letter record
• Spring for Apache Kafka 4.1.1, Message listener containers: AckMode RECORD and BATCH, default BATCH, enable.auto.commit set to false
• Spring for Apache Kafka 4.1.1, Validation: adding a Validator to the registrar
• Spring for Apache Kafka 4.1.1, Serialization, deserialization: FAIL_ON_UNKNOWN_PROPERTIES disabled, ErrorHandlingDeserializer, the record not passed to the listener
• Spring Framework 7.0.9 javadoc, MethodArgumentNotValidException: a subclass of MethodArgumentResolutionException
• Apache Kafka 4.3, Message Delivery Semantics: a crash before saving the position means at-least-once
• Microsoft Learn, Kafka protocol support: at-least-once delivery, the idempotent consumer pattern
• Microsoft Learn, Features and terminology: a partition key keeps related events in one partition, in order
• Microsoft Learn, Partition load balancing: at-least-once delivery, idempotent downstream systems

The database behaviour (a plain insert hits the primary key, a transaction rolls back every row, NOT NULL rejects an empty value) is standard SQL, not taken from a vendor page.

Change notes

• 12 Oct 2026: first published.

Not affiliated with or endorsed by Microsoft, The Apache Software Foundation, Broadcom, VMware or the Spring team. Azure is a trademark of the Microsoft group of companies. Apache Kafka and Kafka are trademarks of The Apache Software Foundation. Spring is a trademark of Broadcom Inc. and/or its subsidiaries. Defaults change between versions. Found a mistake? Tell us in the video’s comments and we’ll correct this page.