Your First Producer & Consumer

The fastest way to understand Kafka is to send a few messages and read them back. In this guide you will build a minimal Java producer that publishes records to a topic and a consumer that polls and prints them, using the official kafka-clients library with no framework in the way. Seeing the raw API first makes everything else — Spring for Apache Kafka, Kafka Streams, Connect — far easier to reason about, because they are all built on top of these same primitives.

Prerequisites

You need a running Kafka broker and a topic to write to. Assuming a local broker on localhost:9092 (KRaft mode, no ZooKeeper), create a topic with three partitions:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

You also need Java 17+ and Maven or Gradle. The only dependency required for plain producers and consumers is org.apache.kafka:kafka-clients. A logging backend such as slf4j-simple keeps the console quiet but informative.

Adding the dependency

For Maven, add the following to pom.xml:

<dependencies>
  <dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.7.0</version>
  </dependency>
  <dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-simple</artifactId>
    <version>2.0.13</version>
  </dependency>
</dependencies>

For Gradle, add this to build.gradle:

dependencies {
    implementation 'org.apache.kafka:kafka-clients:3.7.0'
    implementation 'org.slf4j:slf4j-simple:2.0.13'
}

Writing the producer

A producer needs three essential settings: the bootstrap.servers address of the cluster, and serializers that turn your keys and values into bytes on the wire. Here we send string keys and string values, so we use StringSerializer for both.

The producer is thread-safe and expensive to create, so in real applications you build one and share it. send() is asynchronous and returns a Future<RecordMetadata>; calling get() blocks until the broker acknowledges the record, which is the simplest way to surface errors while learning.

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class OrderProducer {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            for (int i = 1; i <= 5; i++) {
                String key = "order-" + i;
                String value = "{\"id\":" + i + ",\"item\":\"book\"}";
                ProducerRecord<String, String> record = new ProducerRecord<>("orders", key, value);

                RecordMetadata meta = producer.send(record).get();
                System.out.printf("Sent %s -> partition %d, offset %d%n",
                        key, meta.partition(), meta.offset());
            }
        }
    }
}

Setting acks=all tells the broker not to acknowledge until all in-sync replicas have the record, the safest durability setting. The try-with-resources block guarantees the producer is flushed and closed when the loop finishes.

Output:

Sent order-1 -> partition 2, offset 0
Sent order-2 -> partition 1, offset 0
Sent order-3 -> partition 0, offset 0
Sent order-4 -> partition 2, offset 1
Sent order-5 -> partition 0, offset 1

Records with different keys land on different partitions because Kafka hashes the key to pick a partition. Records that share a key always go to the same partition, which is how Kafka preserves ordering per key.

Writing the consumer

A consumer is configured with deserializers (the mirror image of the producer’s serializers) and a group.id. The group id ties consumers together: Kafka divides the topic’s partitions among all consumers in the same group so they share the load. auto.offset.reset=earliest means a brand-new group starts reading from the beginning of the topic rather than only seeing new messages.

The consumer is not thread-safe and follows a poll loop: subscribe to topics, then repeatedly call poll() to fetch batches of records.

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

public class OrderConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("Received key=%s value=%s (partition=%d, offset=%d)%n",
                            record.key(), record.value(), record.partition(), record.offset());
                }
            }
        }
    }
}

Output:

Received key=order-3 value={"id":3,"item":"book"} (partition=0, offset=0)
Received key=order-5 value={"id":5,"item":"book"} (partition=0, offset=1)
Received key=order-2 value={"id":2,"item":"book"} (partition=1, offset=0)
Received key=order-1 value={"id":1,"item":"book"} (partition=2, offset=0)
Received key=order-4 value={"id":4,"item":"book"} (partition=2, offset=1)

Notice the records arrive grouped by partition, not in the order you sent them. Kafka guarantees ordering within a partition, not across the topic. With enable.auto.commit left at its default of true, the consumer periodically commits its offsets, so a restart resumes where it left off.

Tip: The consumer poll loop is a long-running blocking process. Run the producer and consumer as two separate programs (two terminals). Start the consumer first, then run the producer, and watch the messages appear.

Producer and consumer config at a glance

Property	Side	Purpose
bootstrap.servers	both	Initial broker address(es) used to discover the cluster
key.serializer / value.serializer	producer	Convert keys/values to bytes
key.deserializer / value.deserializer	consumer	Convert bytes back to objects
acks	producer	Durability vs. latency trade-off (0, 1, all)
group.id	consumer	Logical group that shares partitions
auto.offset.reset	consumer	Where a new group starts (earliest / latest)

Best Practices

• Create one KafkaProducer per application and reuse it; it is thread-safe and pools connections for you.
• Always close producers and consumers (try-with-resources or a shutdown hook) so buffered records are flushed and group membership is released cleanly.
• Use acks=all for data you cannot afford to lose, and enable idempotence later to avoid duplicates on retries.
• Choose a meaningful message key when ordering matters — records with the same key keep their relative order on one partition.
• Keep the poll loop tight and non-blocking; long work between polls can trigger a consumer group rebalance.
• For real services, prefer manual offset commits over auto-commit so you only acknowledge messages after they are successfully processed.

Related Topics

• Introduction to Kafka
• Installing Kafka
• Kafka CLI tools
• Core concepts glossary
• Kafka architecture overview