Offsets & Position Tracking

An offset is the single integer that pins a record to its place in a partition, and almost every consumer correctness decision — exactly-once processing, replay, lag, and recovery after a crash — comes down to understanding which offset means what. Producers never see offsets; the broker assigns them. Consumers, on the other hand, juggle several different offsets at once, and conflating them is one of the most common sources of duplicate or lost messages in production. This page untangles the offset types, explains why a consumer’s position is not the same as its committed offset, and shows where Kafka stores group offsets internally.

What an offset actually is

Each partition is an ordered, append-only log. When a record is appended, the broker assigns it a monotonically increasing 64-bit integer — its offset — that is unique and stable within that partition. Offsets are not globally unique across partitions, and they are not message IDs you choose; they are positions in the log. A consumer reads by saying “give me records starting at offset N from partition P.”

partition 0:
  offset:   42   43   44   45   46   47   48   49   50
          [ R ][ R ][ R ][ R ][ R ][ R ][ R ][ R ]
            ^                        ^         ^    ^
       log-start              committed     position
        offset (42)          offset (47)   (48)   log-end
                                                offset / LEO (50)
            <-------- retained, replayable ------->

The offset types you must distinguish

Term	Owned by	Meaning
Log-start offset	broker	Oldest offset still retained. Older records were deleted by retention or compaction.
Log-end offset (LEO)	broker	Offset that will be assigned to the next record — i.e. one past the last appended record.
High watermark (HW)	broker	Highest offset replicated to all in-sync replicas; the last offset consumers are allowed to read.
Current position	consumer	The offset the consumer will fetch next. Lives in memory on the consumer.
Committed offset	group (stored in Kafka)	The last position the consumer group durably saved, used to resume after a restart or rebalance.

The LEO and high watermark can differ: records past the HW exist on the leader but are not yet acknowledged by all replicas, so consumers cannot see them. See High watermark for the replication detail.

Position vs. committed offset

This is the distinction that trips people up. The position advances every time poll() hands you records — it is purely in-memory and moves whether or not you commit. The committed offset is what the group has persisted; it only changes when you (or auto-commit) call a commit. After a crash or a partition reassignment, the consumer resumes from the committed offset, not from wherever its position happened to be.

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
    process(record); // your real business logic
}
// position is now LEO of what we fetched, but nothing is durable yet:
long pos = consumer.position(new TopicPartition("orders", 0));
OffsetAndMetadata committed =
        consumer.committed(Set.of(new TopicPartition("orders", 0)))
                .get(new TopicPartition("orders", 0));
// pos > committed.offset()  until we explicitly commit
consumer.commitSync();

The committed offset is the offset of the next record to read, not the last one processed. If you process record 47 and want to resume after it, you commit 48. Off-by-one here causes either a reprocessed or a skipped message on every restart.

Auto-commit vs. manual commit

enable.auto.commit=true periodically commits the current position in the background every auto.commit.interval.ms. It is convenient but at-most-once-leaning: a crash after the position advanced but before your processing finished loses those records. For at-least-once semantics, disable it and commit after processing.

spring:
  kafka:
    consumer:
      enable-auto-commit: false
      auto-offset-reset: earliest
    listener:
      ack-mode: manual

In Spring for Apache Kafka the same logic is expressed with an Acknowledgment:

@Component
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onMessage(ConsumerRecord<String, String> record, Acknowledgment ack) {
        handle(record.value()); // do the work first
        ack.acknowledge();      // then commit -> at-least-once
    }
}

auto.offset.reset: where a brand-new group starts

When a group has no committed offset for a partition (a new group, or the old offset aged out of __consumer_offsets), the consumer must decide where to begin. auto.offset.reset controls this: earliest starts at the log-start offset and replays everything retained; latest (the default) starts at the LEO and ignores history; none throws so you handle it explicitly.

The __consumer_offsets topic

Committed offsets are not stored in ZooKeeper or on the consumer — they live in an internal, compacted Kafka topic called __consumer_offsets. Each commit is a record keyed by (group, topic, partition) whose value is the offset and metadata. Because the topic is log-compacted (see Log compaction), only the latest offset per key is retained, so it stays compact regardless of how many commits a long-running group makes.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-service

Output:

GROUP          TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
order-service  orders  0          48              50              2    consumer-1-...
order-service  orders  1          120             120             0    consumer-1-...

CURRENT-OFFSET is the committed offset, LOG-END-OFFSET is the broker’s LEO, and lag = LEO − committed — the most important health metric for any consumer. The topic has 50 partitions by default (offsets.topic.num.partitions); a group’s offsets all hash to one partition, whose leader broker acts as that group’s coordinator.

Best Practices

Commit the offset of the next record (lastProcessed + 1); never commit the offset you are still processing.
Disable auto-commit and commit after processing for at-least-once delivery; pair it with idempotent handlers since duplicates are still possible.
Monitor consumer lag (LEO − committed) per partition and alert on sustained growth — it is your earliest warning of a stuck or under-scaled consumer.
Set auto.offset.reset deliberately: earliest for replay/ETL pipelines, latest for live dashboards that don’t care about history.
Use commitSync for correctness-critical flows and commitAsync for throughput, but always do a final commitSync in your shutdown/rebalance hook.
Remember offsets can expire: offsets.retention.minutes deletes committed offsets for inactive groups, after which auto.offset.reset decides where they restart.