Kafka with Spring Boot

Apache Kafka is a distributed event-streaming platform built around a durable, append-only log. Unlike a traditional queue, Kafka retains messages after they are read, so many independent consumers can read the same stream — and replay it. Spring Boot integrates Kafka through the spring-kafka library, which provides a KafkaTemplate for publishing and the @KafkaListener annotation for consuming, with auto-configuration driven entirely by properties. This page wires a producer and consumer together; for the broker fundamentals, see the dedicated Kafka docs and Messaging Intro.

Dependency

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Note: Kafka uses the spring-kafka artifact, not a spring-boot-starter-*. Its version is managed by the Spring Boot parent, so you omit the <version> tag.

Topics, partitions, and consumer groups

Three concepts define how Kafka scales and orders messages:

• A topic is a named stream of records (e.g. orders).
• Each topic is split into partitions — the unit of parallelism and ordering. Order is guaranteed within a partition, not across them. The record’s key decides its partition, so all events for one order land in the same partition and stay ordered.
• A consumer group is a set of instances sharing a group.id. Kafka assigns each partition to exactly one consumer in the group, so adding instances (up to the partition count) increases throughput. Different groups each receive the full stream — that is how Kafka does pub/sub.

topic "orders" (3 partitions)        group "billing" (2 instances)
  P0 ─────────────►  instance A  ◄── handles P0, P1
  P1 ─────────────►
  P2 ─────────────►  instance B  ◄── handles P2

group "analytics" (separate group) gets its own full copy of P0–P2

Term	Role
Topic	logical stream of records
Partition	ordered, parallelizable shard of a topic
Key	determines a record’s partition (preserves per-key order)
Offset	a record’s position within a partition
Consumer group	instances that split partitions for scale

Running Kafka locally

Modern Kafka runs in KRaft mode (no ZooKeeper). A minimal single-broker compose.yaml:

services:
  kafka:
    image: apache/kafka:3.8.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d

Producer and consumer configuration

Configure both sides through properties. We send and receive a record as JSON:

public record OrderEvent(Long orderId, String customer, double total) {}

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all                       # wait for all in-sync replicas
    consumer:
      group-id: order-processors
      auto-offset-reset: earliest     # read from the start on a new group
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.example.orders"

Tip: JsonSerializer/JsonDeserializer handle JSON serde for you. Set spring.json.trusted.packages (or *) so the deserializer is allowed to instantiate your payload classes; leaving it unset blocks deserialization for security.

Creating a topic

Declare topics as @Beans with TopicBuilder and Spring Kafka creates them on startup if they are missing:

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.kafka.config.TopicBuilder;

@Bean
public NewTopic ordersTopic() {
    return TopicBuilder.name("orders")
            .partitions(3)
            .replicas(1)
            .build();
}

Producing with KafkaTemplate

KafkaTemplate.send(topic, key, value) publishes a record. Passing the order id as the key keeps all events for one order in the same partition, preserving their order.

import lombok.RequiredArgsConstructor;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
@RequiredArgsConstructor
public class OrderProducer {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public void publish(OrderEvent event) {
        kafkaTemplate.send("orders", String.valueOf(event.orderId()), event)
            .whenComplete((result, ex) -> {
                if (ex == null) {
                    var meta = result.getRecordMetadata();
                    // partition + offset confirm the broker accepted it
                    System.out.printf("Sent to %s-%d@%d%n",
                            meta.topic(), meta.partition(), meta.offset());
                }
            });
    }
}

send returns a CompletableFuture, so publishing is non-blocking. Attach whenComplete to react to the broker’s acknowledgement.

Output (console):

Sent to orders-1@57

Consuming with @KafkaListener

Annotate a method with @KafkaListener, set the topics and groupId, and Spring delivers each record deserialized into the parameter type.

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Component
public class OrderListener {

    private static final Logger log = LoggerFactory.getLogger(OrderListener.class);

    @KafkaListener(topics = "orders", groupId = "order-processors")
    public void onOrder(OrderEvent event) {
        log.info("Processing order {} for {} (${})",
                event.orderId(), event.customer(), event.total());
        // ... reserve inventory, charge payment, etc.
    }
}

Output (console):

2026-06-13T11:30:08.441  INFO  OrderListener : Processing order 1042 for Asha ($89.95)

To scale out, run more instances with the same groupId — Kafka redistributes the topic’s partitions among them. To add a wholly independent consumer (e.g. analytics), use a different groupId and it receives its own copy of every record.

You can also access metadata such as partition and offset via headers:

import org.apache.kafka.clients.consumer.ConsumerRecord;

@KafkaListener(topics = "orders", groupId = "audit")
public void audit(ConsumerRecord<String, OrderEvent> record) {
    log.info("key={} partition={} offset={}",
            record.key(), record.partition(), record.offset());
}

Note: Kafka delivers at-least-once by default — after a rebalance or restart a record may be redelivered. Make listeners idempotent (e.g. track processed order ids) so reprocessing is harmless.

Offsets and error handling

Spring Kafka auto-commits offsets after a batch is processed successfully. When a listener throws, the default DefaultErrorHandler retries with backoff and, after exhausting attempts, can route the record to a dead-letter topic via a DeadLetterPublishingRecoverer. Because offsets advance only on success, a crash mid-batch simply re-reads the uncommitted records — no data is lost.

Warning: Do not over-provision partitions hoping to scale later — a consumer group cannot have more active consumers than partitions, but repartitioning a live topic reshuffles key-to-partition assignment and breaks ordering guarantees. Size partitions to peak parallelism from the start.

RabbitMQ vs Kafka, briefly

Question	Lean RabbitMQ	Lean Kafka
Need complex routing per message?	Yes (exchanges)	No
Need to replay/re-read history?	No	Yes (retained log)
Very high sustained throughput?	Moderate	Yes
Discrete tasks for workers?	Yes	Possible, less natural

See RabbitMQ with Spring Boot for the queue-and-exchange model.

Best Practices

• Use a meaningful key to preserve per-entity ordering within a partition.
• Configure JsonSerializer/JsonDeserializer and set spring.json.trusted.packages.
• Set acks: all for durability on writes you cannot lose.
• Make @KafkaListener methods idempotent; configure a dead-letter topic for poison records.
• Scale consumers within a group up to the partition count; use separate groups for independent readers.

Related Topics

• Messaging Intro
• RabbitMQ with Spring Boot
• Apache Kafka Fundamentals
• Microservices Intro
• Async Methods
• Records for DTOs