Error Handling Patterns for Apache Kafka
Applications
Apache Kafka® applications run in a distributed manner across multiple containers or machines.
And in the world of distributed systems, what can go wrong often goes wrong. This blog post
covers different ways to handle errors and retries in your event streaming applications. The nature
of your process determines the patterns, and more importantly, your business requirements.
This blog provides a quick guide on some of those patterns and expands on a common and specific
use case where events need to be retried following their original order. This blog post illustrates a
scenario of an application that consumes events from one topic, transforms those events, and
produces an output to a target topic, covering different approaches as they gradually increase in
complexity.
Pattern 1: Stop on error
There are cases when all input events must be processed in order without exceptions. An example is
handling the change-data-capture stream from a database.
The following diagram illustrates how events in the source topic are processed or transformed and
published to the target topic. An error in the process causes the application to stop and manual
intervention is required. Notice that the events in the source topic cannot take any other path.
Pattern 2: Dead letter queue
This is a common scenario where events that cannot be processed by the application are routed to
an error topic while the main stream continues. It’s important to note that in this approach there are
no conditions that require or support a retry process. In other words, an event can be processed
successfully, or it is routed to an error topic.
The following diagram illustrates how events in the source topic can take one of two paths:
1. Under normal circumstances, the application processes each event in the source topic
and publishes to the target topic
2. Events that cannot be processed, for example, those that don’t have the expected format
or are missing required attributes, are routed to the error topic
Pattern 3: Add a retry topic and retry application
What happens if the conditions required to process an event are not available when the application
attempts to process the event? For example, consider an application that handles requests to
purchase items. The price of an item may be produced by a different application and could be
missing at the time that the request is received.
Adding a retry topic provides the ability to process most events right away while delaying the
processing of other events until the required conditions are met. In the example, you would route
events to the retry topic if the price of the item is not available at the time. This recoverable
condition should not be treated as an error but periodically retried until the conditions are met.
In your containerized environment you may dedicate one or two instances to handle the retry
process, as few events are expected to follow this path.
The following diagram illustrates how an event in the source topic can take one of three different
paths:
1. Under normal circumstances, the application processes each event in the source topic
and publishes the result to the target topic
2. Events that cannot be processed, for example, those that don’t have the expected format
or are missing required attributes, are routed to the error topic
3. Events for which dependent data is not available are routed to a retry topic where a retry
instance of your application periodically attempts to process the events
Related events may be processed out of order
There is a very important aspect to highlight with this pattern: Events are not guaranteed to be
processed in the same sequence received in the source topic. This is because the retry path is
“longer” and usually slower than the normal execution path, as there are fewer retry instances and
retries are delayed.
The following diagram illustrates how the first event received (Event 1) reaches the target topic
after an event that was received later (Event 2). You should use this pattern only if you are okay
with this behavior.
Pattern 4: Maintain order of redirected events
As shown in the previous pattern, adding a retry topic and associated flow provides a few benefits
by delaying the execution of some events until the required conditions are met. However, the
previous pattern also illustrated that the order at the target topic may change.
There are some conditions where changing the order of the events is not acceptable. For instance,
an application that updates the inventory of an item would produce different results if an increase in
the inventory is processed before a decrease or vice versa. If there is a trigger that initiates a
“purchase request” when an item’s inventory falls below 10 and you currently have 10 units,
processing a decrease of inventory before a previously received increase may trigger an unwanted
“purchase request”, as the inventory may fall below a desired limit. This could also cause an error
if the inventory goes below zero. This final pattern addresses that problem.
In this pattern, the main application needs to keep track of every event routed to the retry topic.
When a dependent condition is not met (for instance, the price of an item) the main application
stores a unique identifier for the event in a local in-memory structure. The unique event identifiers
are grouped by the item that they belong to. This helps the application determine if events related to
a particular item, for example, Item A, are currently being retried and therefore subsequent events
related to Item A should be sent to the retry path to preserve order.
When the first event that has missing dependencies is received, the main application performs the
following tasks:
1. Notes the unique ID for the message (grouped by the item that they belong to), adding it
to a local in-memory store.
2. Routes the event to the retry topic, adding a header with the unique ID of the message.
Adding the message ID as a header prevents alteration of the original message, and as
explained below, enables the retry application to publish the appropriate tombstone
event when the retry is complete.
3. Publishes the unique ID of the received message to the redirect topic.
When the next event is received, the application checks the local store to determine if there are
events for that item. If one or more events for the item are found, the application knows that some
events are being retried and will route the new event to the retry flow.
In the example, if the application receives an event for Item A, which currently has events that are
being retried, the application will not attempt to process the event but will instead route it through
the retry flow. This ensures that all the events for Item A are processed in the same order that they
were received. The application adds the unique identifier for that new event to the local store and
routes to the retry and redirect topics as before.
The retry application handles the events in the retry topic in the order that they are received. When
an event is successfully retried and published to the target topic, the retry application instance sends
confirmation in the form of a tombstone event to the redirect topic. One tombstone event is
published for each successfully retried event.
The main application listens to the redirect topic for tombstone events that signal successful retry.
The application removes the messages from the in-memory store. This also allows for subsequent
events for the same item to be processed through the main flow.
The final diagram illustrates the path taken by an event for an item that has all the required
dependencies and can follow the normal flow. The main application performs the following tasks:
1. Reads the event from the source topic
2. Checks the local store to verify that no entries exist in the retry path
3. Processes the event and publishes to the target topic
Recovery
In the case of failure, the in-memory store that the main application was managing will be gone.
However, this can easily be restored by reading the events in the redirect topic and initializing that
in-memory store.
Conclusion
Error handling and retries are important aspects of the development of all types of applications, and
Kafka applications are no exception. The approaches mentioned are not intended to cover all
possible aspects but provide guidance that can be adapted to meet your needs.
If you’d like to learn more, check out Confluent Developer to find the largest collection of
resources for getting started, including end-to-end Kafka tutorials, videos, demos, meetups,
podcasts, and more.
Learn More