Spring for Apache Kafka

The raw kafka-clients library is powerful but low-level: you wire up producers and consumers by hand, manage their lifecycle, write your own poll loops, and reinvent error handling on every project. Spring for Apache Kafka (the spring-kafka project) sits on top of that client and turns it into idiomatic Spring — a single KafkaTemplate for sending, a @KafkaListener annotation for receiving, and Spring Boot auto-configuration that reads everything from application.yml. The result is far less boilerplate and a programming model that production teams already understand, with first-class hooks for retries, error handling, transactions, and testing.

What spring-kafka adds over the raw client

The plain KafkaConsumer is not thread-safe and must be driven by a poll loop you own. The plain KafkaProducer returns a Java Future you have to manage. Spring Kafka wraps both behind higher-level abstractions and integrates them into the Spring application context and lifecycle.

Concern	Raw `kafka-clients`	spring-kafka
Producing	Manage `KafkaProducer`, handle `Future`	Inject `KafkaTemplate`, async/sync sends
Consuming	Hand-written `poll()` loop, manual threads	`@KafkaListener` on a method
Configuration	Build `Properties` maps in code	`application.yml` auto-configuration
Offsets	Manual `commitSync`/`commitAsync`	Ack modes managed by the container
Error handling	Try/catch around your loop	`DefaultErrorHandler`, retries, DLT
Serialization	Configure serializers manually	`JsonSerializer`/`JsonDeserializer` beans
Testing	Spin up a real broker	`@EmbeddedKafka` in-process broker

Auto-configuration with Spring Boot

Add the starter and Spring Boot configures a KafkaTemplate, a ProducerFactory, a ConsumerFactory, and the container infrastructure for @KafkaListener — all from properties. No @Bean methods are required for the common case.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: order-service
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.devcraftly.orders.events"

The dependency in a Maven build is a single starter:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Spring Boot’s dependency management already pins a compatible spring-kafka version, so you do not declare a version yourself. Mixing an out-of-band spring-kafka with the wrong kafka-clients is the most common cause of NoSuchMethodError at startup.

Producing with KafkaTemplate

KafkaTemplate<K, V> is the producer-side workhorse. It is thread-safe, injectable, and returns a CompletableFuture so you can react to success or failure without blocking. Model the event as an immutable Java record.

import java.time.Instant;

public record OrderPlaced(String orderId, String customerId, long amountCents, Instant placedAt) {}

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class OrderProducer {

    private final KafkaTemplate<String, OrderPlaced> template;

    public OrderProducer(KafkaTemplate<String, OrderPlaced> template) {
        this.template = template;
    }

    public void publish(OrderPlaced event) {
        template.send("orders", event.orderId(), event)
                .whenComplete((result, ex) -> {
                    if (ex != null) {
                        System.err.println("Send failed: " + ex.getMessage());
                    } else {
                        System.out.println("Sent to offset " + result.getRecordMetadata().offset());
                    }
                });
    }
}

Consuming with @KafkaListener and listener containers

On the consumer side, you annotate a method with @KafkaListener and Spring builds a listener container that owns the poll loop, the consumer thread(s), and offset commits. Your method just receives deserialized records.

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onOrder(OrderPlaced event) {
        System.out.println("Processing order " + event.orderId()
                + " for " + event.amountCents() + " cents");
    }
}

Output:

Processing order ord-1042 for 4999 cents
Processing order ord-1043 for 1299 cents

Behind the annotation, a KafkaMessageListenerContainer (or ConcurrentMessageListenerContainer for multiple threads) is created by the ConcurrentKafkaListenerContainerFactory bean. The container handles rebalances, commits offsets according to the configured ack mode, and routes failures to the error handler — all the loop logic you would otherwise write by hand.

Error handling, retries, and DLT

By default a listener container uses a DefaultErrorHandler that retries a failed record a few times with back-off and then logs it. You can configure exponential back-off and, when retries are exhausted, publish the poison record to a dead-letter topic using DeadLetterPublishingRecoverer — turning unrecoverable messages into something you can inspect and replay rather than an infinite retry loop or silent loss.

Testing support

Spring Kafka ships an embedded, in-process Kafka broker for integration tests via @EmbeddedKafka, so your tests exercise real producers, listeners, and serializers without Docker or an external cluster. This makes round-trip tests — send a record, assert the listener consumed it — fast and deterministic.

What the later pages cover

This section walks the full stack: project setup, producing with KafkaTemplate, consuming with @KafkaListener, tuning producer and consumer config, JSON serialization, robust error handling with retries and dead-letter topics, manual acknowledgment, concurrency and containers, transactions, and testing with embedded Kafka.

Best Practices

Let Spring Boot auto-configuration drive the common case — declare beans only when you need behaviour the properties cannot express.
Never pin a manual spring-kafka or kafka-clients version; rely on Boot’s managed dependency to keep them compatible.
Use constructor injection for KafkaTemplate and keep it a singleton — it is thread-safe and reused across requests.
React to the CompletableFuture returned by send() so producer failures are logged or retried rather than swallowed.
Configure a DefaultErrorHandler with back-off and a dead-letter topic instead of leaving listeners to retry forever.
Restrict spring.json.trusted.packages to your event packages to avoid deserialization-gadget risks.
Cover producers and listeners with @EmbeddedKafka integration tests so serialization and offset behaviour are verified before deploy.