Fan-Out & Pub/Sub

Fan-out is the pattern where a single event produced once is delivered to many independent consumers, each free to react in its own way. In Kafka this falls out naturally from how consumer groups and offsets work: you do not need a separate queue per subscriber, you simply give each subscriber its own consumer group. This is the foundation of publish/subscribe in an event-driven system, and getting the consumer-group model right is the difference between every service seeing every event and services accidentally stealing each other’s messages.

The two delivery modes of a consumer group

Kafka’s delivery semantics are governed entirely by the group.id. A topic is partitioned, and each partition is assigned to exactly one consumer within a group. That single rule produces two very different behaviours depending on how you assign group IDs.

Load sharing (one group): When several consumer instances share the same group.id, Kafka splits the topic’s partitions among them. Each record is processed by exactly one member. This is competing consumers — you scale throughput by adding instances, up to the partition count.

Fan-out (many groups): When you create a distinct group.id per logical consumer, each group maintains its own committed offsets and independently reads the full stream. The same record is delivered once to every group. This is pub/sub — adding a subscriber does not reduce what anyone else receives.

The key insight is that Kafka retains records based on retention policy, not on whether they have been consumed. A read does not remove data, so any number of groups can replay the same partitions at their own pace.

                         topic: orders
                  ┌──────────────────────────┐
                  │  partition 0  1  2  3     │
                  └──────────────────────────┘
                        │          │
        ┌───────────────┘          └────────────────┐
        ▼                                            ▼
  group: billing                            group: analytics
  (own offsets)                             (own offsets)
   ┌────────────┐                            ┌──────────────┐
   │ instance A │  shares partitions         │ instance X   │
   │ instance B │  within the group          │ instance Y   │
   └────────────┘                            └──────────────┘
   each ORDER seen once per group, but BOTH groups see every order

When to use a new group versus scaling one

Decide based on whether the consumers represent the same job or different jobs.

Question	New consumer group	Scale existing group
Does this consumer have its own business purpose?	Yes	No
Should it receive every event regardless of other consumers?	Yes	No
Do you only need more throughput for the same logic?	No	Yes
Are offsets/lag tracked independently?	Yes	No (shared)
Limited by partition count?	No (per group)	Yes

A useful rule: one consumer group per concern. Billing, search indexing, and notifications are three concerns, so they get three groups. The three instances of the billing service that exist for high availability share one group.

Tip: Scaling a single group beyond the topic’s partition count gains you nothing — the extra consumers sit idle because there is no partition left to assign. If you need more parallelism within one concern, increase partitions, not just instances.

Spring Boot: many groups on one topic

With Spring for Apache Kafka each @KafkaListener declares its own groupId. Two listeners on the same topic with different group IDs both receive every record. This is all the wiring fan-out requires.

public record OrderPlaced(String orderId, String customerId, long amountCents) {}

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class OrderFanOutListeners {

    @KafkaListener(topics = "orders", groupId = "billing")
    public void charge(OrderPlaced event) {
        // independent offsets for the "billing" group
        System.out.println("billing: charging " + event.amountCents() + " for " + event.orderId());
    }

    @KafkaListener(topics = "orders", groupId = "analytics")
    public void index(OrderPlaced event) {
        // "analytics" group sees the SAME records, tracked separately
        System.out.println("analytics: recording " + event.orderId());
    }
}

The consumer factory must know how to deserialize the JSON payload into the record. Configure trusted packages so the JsonDeserializer can map incoming messages.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.value.default.type: com.example.OrderPlaced
        spring.json.trusted.packages: "com.example"

When the application starts and a single order is produced, both groups process it:

billing: charging 4999 for ord-1001
analytics: recording ord-1001

Verifying independent offsets

Each group has its own offset position, which you can confirm with the CLI. The same topic shows separate lag per group.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group analytics

Output:

GROUP      TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
billing    orders  0          1042            1042            0
analytics  orders  0          985             1042            57

Here analytics is 57 records behind while billing is caught up — proof that the groups read independently. Because offsets are per group, you can also reset one group to replay history without touching the others:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group analytics --topic orders --reset-offsets --to-earliest --execute

Warning: Never reuse a group.id across two services that do different work. They will share partition assignments and silently split the stream, so each service sees only a fraction of the events — a notoriously hard bug to spot in production.

Best Practices

Treat group.id as a contract: one group per logical concern, never shared between unrelated services.
Use a stable, descriptive group name (for example billing or search-indexer) so lag dashboards and offset resets are meaningful.
Keep partition count at or above the maximum instances you expect in any single group; add partitions before you outgrow them, since reducing them is impossible.
Monitor consumer lag per group — fan-out means one slow subscriber does not block others, but it can quietly fall far behind.
Set auto-offset-reset deliberately: earliest for new groups that must process history, latest for groups that only care about future events.
Prefer adding a new group over forking a topic when a fresh consumer needs the same data — Kafka’s retention lets every group read the same records cheaply.