High Watermark & Replica Sync

Kafka durability hinges on a single, deceptively simple rule: consumers can only read data that has been replicated to every in-sync replica. The mechanism that enforces this is the high watermark (HW). Understanding the relationship between the high watermark and the log-end offset is essential for reasoning about consistency, failover, and the exact moment a record becomes “safe” to deliver. Get this wrong and you risk surprises during broker failures or when tuning acks and min.insync.replicas.

Log-end offset vs high watermark

Every partition replica maintains two key offsets in its local log:

Log-end offset (LEO) — the offset of the next record to be appended. It always points one past the last record physically written to that replica’s log.
High watermark (HW) — the offset up to which all records are committed, meaning they have been successfully replicated to all members of the in-sync replica set (ISR).

The leader’s HW is computed as the minimum LEO across all in-sync replicas. A record is only “committed” once every ISR follower has fetched and persisted it, advancing the slowest follower’s LEO past that record. Consumers are never allowed to read past the high watermark — even though the leader physically holds those records in its log.

Offset	Scope	Meaning	Visible to consumers?
LEO	Per replica	Next offset to be written	No (above HW)
HW	Partition (from leader)	Min LEO across ISR; last committed offset	Up to HW only
Committed	Logical	Replicated to all ISR	Yes

Why consumers stop at the high watermark

Imagine the leader accepts records up to LEO=100 but the followers have only replicated up to offset 95. If a consumer were allowed to read offset 99 and the leader then crashed, a follower with only 95 records could be elected the new leader. Offsets 96-99 would vanish — the consumer would have read records that no longer exist in the cluster. This is exactly the “phantom read” Kafka must prevent.

By withholding everything above the HW, Kafka guarantees that any record a consumer sees has already survived on every in-sync replica. When a new leader is elected from the ISR, it is guaranteed to hold every committed record, so consumers never observe data loss or rewound offsets.

Gotcha: Records sitting between the HW and the leader’s LEO are real and durable on the leader, but they are not yet committed. A producer using acks=all is not acknowledged until the HW advances past its record. Lowering acks to 1 acknowledges at the leader’s LEO instead, trading durability for latency.

Visualizing leader and follower offsets

The diagram below shows a partition with one leader and two followers. The leader has written through offset 9 (LEO=10), follower B has caught up, but follower C lags. The HW is the minimum committed point.

                  committed (HW=8)        uncommitted
                 |<----------------->|  |<----------->|
 Leader  (LEO=10): [0][1][2][3][4][5][6][7][8][9]
 Follower B (LEO=9): [0][1][2][3][4][5][6][7][8]
 Follower C (LEO=8): [0][1][2][3][4][5][6][7]
                                      ^HW=8
                                      |
   Consumers may read offsets 0..7. Offsets 8,9 are hidden
   until follower C fetches them and the HW advances.

As follower C fetches offset 8, its LEO becomes 9, the leader recomputes HW = min(10, 9, 9) = 9, and offset 8 becomes visible to consumers. Followers learn the current HW from the leader’s fetch responses and use it to advance their own HW.

Inspecting offsets in practice

You can observe the LEO and HW directly with the standard tooling. The log-end offset is the latest offset, while the consumer-visible end is bounded by the HW.

# Show the high watermark (latest readable offset) per partition
kafka-get-offsets.sh --bootstrap-server localhost:9092 \
  --topic orders --time -1

Output:

orders:0:842
orders:1:917
orders:2:903

To see replication health and whether followers are keeping the HW close to the LEO, describe the topic and watch the ISR list:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

Output:

Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3

Partition 1 shows broker 1 missing from the ISR. The HW for that partition is now the minimum LEO of brokers 2 and 3 only — a slow or failed replica shrinks the ISR but does not stall commits, as long as min.insync.replicas is still satisfied.

Producer-side configuration that interacts with the HW

A producer’s durability contract is enforced relative to the HW via these settings. With acks=all, the leader waits for the HW to cover the record before acknowledging.

spring:
  kafka:
    producer:
      acks: all                 # ack only after ISR replication (HW advances)
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
    # Broker-side, set per topic or cluster:
    # min.insync.replicas: 2   # require at least 2 ISR for a commit

A minimal idempotent producer confirms the commit before returning control:

@Service
public class OrderPublisher {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderPublisher(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(OrderEvent event) {
        // get() blocks until the broker acks; with acks=all this means
        // the record is committed (HW has advanced past it).
        kafkaTemplate.send("orders", event.orderId(), event)
                .whenComplete((result, ex) -> {
                    if (ex == null) {
                        var meta = result.getRecordMetadata();
                        System.out.printf("Committed offset %d in partition %d%n",
                                meta.offset(), meta.partition());
                    }
                });
    }

    public record OrderEvent(String orderId, String status, long amountCents) {}
}

Best Practices

Set min.insync.replicas=2 (with replication factor 3) and acks=all so a record is only committed when at least two replicas hold it, keeping the HW meaningful.
Monitor the gap between LEO and HW: a persistently large gap means followers are lagging and durability acknowledgements are slow.
Watch ISR shrink/expand metrics — an undersized ISR raises the HW from fewer replicas and increases the blast radius of a single failure.
Never lower acks to 0 or 1 for data you cannot afford to lose; you would be acknowledging at the leader’s LEO, ahead of the HW.
Enable idempotent producers to avoid duplicate records when retries occur after a slow HW advance.
Treat any offset above the HW as provisional in your mental model — only committed (≤ HW) data is guaranteed across leader failover.