Custom Partitioner

By default, Kafka chooses a partition for each record using the key hash (or sticky batching when there is no key). That works well for most workloads, but sometimes business rules demand explicit control over routing — for example, isolating high-value traffic onto a dedicated partition, co-locating related records for ordering guarantees, or steering load away from a hot partition. A custom Partitioner lets you encode that logic at the producer, so every record lands on exactly the partition you decide.

How partitioning works

When you call producer.send(record), the producer must resolve a target partition before the record is appended to an accumulator batch. The resolution order is:

If the ProducerRecord was constructed with an explicit partition, that value wins.
Otherwise, the configured Partitioner is consulted.
If no partitioner is set, Kafka uses its built-in default (key-hash with sticky batching for null keys).

A custom partitioner plugs into step 2. It sees the topic, key, value, and cluster metadata, and returns an int partition number. Because it runs on the producer thread for every record, it must be fast and side-effect free.

The Partitioner interface

The interface lives in org.apache.kafka.clients.producer.Partitioner and extends Configurable and Closeable. You implement three methods:

Method	When it runs	Purpose
`configure(Map<String,?> configs)`	Once, at producer startup	Read custom config properties you pass through producer config
`partition(...)`	Per record	Return the target partition number
`close()`	Once, when the producer closes	Release any resources

Example: route VIP customers to a dedicated partition

Suppose orders are keyed by customer ID and we want all VIP customers’ orders to flow through partition 0 (which downstream consumers process with higher priority), while everyone else is spread across the remaining partitions by key hash.

package com.devcraftly.kafka;

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import org.apache.kafka.common.PartitionInfo;
import org.apache.kafka.common.utils.Utils;

import java.util.List;
import java.util.Map;
import java.util.Set;

public class VipCustomerPartitioner implements Partitioner {

    private static final int VIP_PARTITION = 0;
    private Set<String> vipCustomerIds = Set.of();

    @Override
    public void configure(Map<String, ?> configs) {
        Object raw = configs.get("vip.customer.ids");
        if (raw instanceof String s && !s.isBlank()) {
            this.vipCustomerIds = Set.of(s.split(","));
        }
    }

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {

        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
        int numPartitions = partitions.size();

        if (key == null) {
            // No key: spread across non-VIP partitions only.
            return 1 + (int) (Math.random() * (numPartitions - 1));
        }

        String customerId = key.toString();
        if (vipCustomerIds.contains(customerId)) {
            return VIP_PARTITION;
        }

        // Hash non-VIP keys across partitions 1..N-1, keeping 0 reserved.
        int hash = Utils.toPositive(Utils.murmur2(keyBytes));
        return 1 + (hash % (numPartitions - 1));
    }

    @Override
    public void close() {
        // No resources to release.
    }
}

Reserving partition 0 means partition 0 will never see non-VIP traffic, while partitions 1..N-1 carry everything else. Make sure the topic has enough partitions for this split to be worthwhile, and remember that any change to the partition count silently changes which key maps where.

Wiring it via partitioner.class

Register the partitioner with the partitioner.class property. Any extra keys you add to the producer config (such as vip.customer.ids above) are passed straight into configure().

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("partitioner.class", "com.devcraftly.kafka.VipCustomerPartitioner");
props.put("vip.customer.ids", "cust-7,cust-42,cust-99");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    producer.send(new ProducerRecord<>("orders", "cust-42", "{\"amount\":250}"));
    producer.send(new ProducerRecord<>("orders", "cust-3",  "{\"amount\":12}"));
    producer.flush();
}

Spring Boot configuration

In a Spring for Apache Kafka application, set the same properties under spring.kafka.producer:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      properties:
        partitioner.class: com.devcraftly.kafka.VipCustomerPartitioner
        vip.customer.ids: cust-7,cust-42,cust-99

Verifying the routing

You can confirm where records landed by reading offsets per partition with the console tooling.

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --partition 0 --from-beginning --property print.key=true

Output:

cust-42	{"amount":250}
cust-7	{"amount":980}

Only VIP keys appear on partition 0, confirming the partitioner routed cust-3 elsewhere.

Best Practices

Keep partition() deterministic and cheap — it runs on the hot send path for every record and must not block on I/O.
Always read the live partition count from cluster.partitionsForTopic(topic) instead of hard-coding it; topics can be expanded.
Be aware that ordering is only guaranteed within a single partition, so route records that must stay ordered (e.g., per-customer events) to the same partition consistently.
Avoid creating skew: forcing many keys onto one partition can produce a hot partition that bottlenecks consumers.
Pass tunable values (like VIP IDs) through producer config and read them in configure() rather than hard-coding them in the class.
Document the partition-count assumption — changing the number of partitions remaps every key and can break ordering and idempotent dedup expectations.