KRaft Mode (No ZooKeeper)

For most of Kafka’s history the cluster relied on an external service, ZooKeeper, to store metadata and elect a controller. KRaft (Kafka Raft) removes that dependency entirely: Kafka now manages its own metadata using a built-in Raft consensus protocol running inside the brokers themselves. This is not a minor tuning knob — it changes how you provision, bootstrap, and operate a cluster, and as of Kafka 4.0 it is the only supported mode. Understanding KRaft is essential for anyone deploying or upgrading Kafka in production today.

What KRaft actually is

KRaft is an implementation of the Raft consensus algorithm, specialized for managing Kafka’s cluster metadata. Rather than writing metadata to ZooKeeper znodes, KRaft stores every metadata change — topic creation, partition reassignment, broker registration, leader changes, ISR updates — as records in an internal, replicated log.

A small set of nodes act as controllers and form a quorum. One controller is the leader (the active controller) and the others are followers. The leader appends new metadata records to the log; followers replicate them. A record is committed once a majority of the quorum has persisted it, which is exactly the durability guarantee Raft provides. Because the source of truth is now an event log rather than a tree of znodes, brokers can catch up on metadata by simply replaying the log from their last known offset.

The `__cluster_metadata` topic

The metadata log lives in a special internal topic called __cluster_metadata. It has a single partition (partition 0) and is replicated only across the controller quorum, not across regular brokers.

The active controller is the leader of __cluster_metadata partition 0 and is the only node that writes to it.
Follower controllers replicate the log so they can take over instantly if the leader fails.
Brokers are observers: they fetch the log to build an in-memory view of the cluster, then serve metadata requests to clients from that cache.

Each broker persists its latest applied metadata offset, so after a restart it fetches only the delta. This incremental, log-based propagation is why KRaft startup and failover are dramatically faster than the old “reload everything from ZooKeeper” model.

Combined vs. separate roles

A KRaft node’s behavior is set by process.roles. There are two common topologies:

Topology	`process.roles`	When to use
Separate (dedicated)	`controller` on some nodes, `broker` on others	Production at scale — isolates metadata workload from data traffic for predictable failover
Combined	`broker,controller` on the same node	Development, testing, and small clusters where running extra nodes is wasteful

In combined mode a single node serves both client traffic and the controller quorum — convenient for a laptop or a tiny three-node cluster. In separate mode you run, for example, 3 dedicated controllers and N brokers. Dedicated controllers stay lean and fast, which is the recommended layout for large clusters.

Tip: A controller quorum needs an odd number of nodes (typically 3 or 5) to tolerate failures cleanly. A 3-node quorum survives 1 controller failure; a 5-node quorum survives 2. More controllers does not mean more throughput — it means more fault tolerance at the cost of slightly higher commit latency.

Bootstrapping and configuration

A KRaft cluster is identified by a cluster ID, which you generate once and use to format each node’s metadata log before first start.

# 1. Generate a unique cluster ID (do this once)
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
echo "$KAFKA_CLUSTER_ID"

# 2. Format the storage directory on EVERY node using that ID
bin/kafka-storage.sh format \
  --cluster-id "$KAFKA_CLUSTER_ID" \
  --config config/kraft/server.properties

# 3. Start the broker/controller
bin/kafka-server-start.sh config/kraft/server.properties

Output:

Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.0-IV3.

A representative server.properties for a combined-mode node looks like this:

# This node acts as both a broker and a controller
process.roles=broker,controller

# Unique numeric id, distinct on every node
node.id=1

# All voting controllers: id@host:port (must match on every node)
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093

# Listeners: 9092 for clients, 9093 reserved for the controller quorum
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
advertised.listeners=PLAINTEXT://kafka1:9092

# Where the data and __cluster_metadata logs are stored
log.dirs=/var/lib/kafka/data

For a dedicated controller node, set process.roles=controller, expose only the CONTROLLER listener, and omit the client-facing PLAINTEXT listener. Dedicated brokers set process.roles=broker and must still list the same controller.quorum.voters so they know where to fetch metadata.

Benefits over ZooKeeper

Simpler operations — one system to deploy, secure, monitor, and upgrade instead of two. No separate ZooKeeper ensemble.
Faster failover — a follower controller already has the full metadata log in memory, so controller failover takes milliseconds rather than seconds.
Faster startup and recovery — brokers replay an incremental log instead of dumping the entire metadata tree from ZooKeeper.
More partitions — KRaft scales to millions of partitions per cluster, well beyond what the ZooKeeper-backed controller could manage.
A single security model — authentication and authorization are configured once, in Kafka, instead of being split across Kafka and ZooKeeper.

Migration from ZooKeeper

If you still run a ZooKeeper-based cluster, Kafka provides a documented, online migration path (available in the 3.x line; ZooKeeper support is removed in 4.0, so migrate before upgrading to 4.x).

Deploy a new KRaft controller quorum alongside the existing ZooKeeper cluster, enabling zookeeper.metadata.migration.enable=true.
The controllers enter migration mode and copy all existing metadata out of ZooKeeper into the __cluster_metadata log.
Restart each broker in dual-write mode so it registers with the KRaft controllers while ZooKeeper stays in sync as a fallback.
Once all brokers are migrated and metadata is verified, finalize the migration — the controllers leave migration mode and ZooKeeper is decommissioned.

Warning: Migration is one-directional once finalized. Run it first in a staging environment, take a metadata backup, and validate topic, ACL, and quota state before retiring ZooKeeper.

Best Practices

Run dedicated controllers (separate process.roles) for any production cluster of meaningful size; reserve combined mode for development.
Use an odd-sized quorum of 3 or 5 controllers — never an even number, which provides no extra failure tolerance.
Keep the CONTROLLER listener on its own port and, in production, secure it with TLS/SASL just like client traffic.
Ensure controller.quorum.voters is identical on every node and points at stable hostnames or DNS names, not ephemeral IPs.
Place controller log.dirs on fast, durable storage; metadata commit latency directly affects leader-election and failover speed.
Generate the cluster ID once and format every node with the same ID — mismatched IDs prevent nodes from joining the quorum.
Migrate off ZooKeeper before upgrading to Kafka 4.0, since that release drops ZooKeeper support entirely.