Log Retention & Cleanup

Kafka is a durable, append-only log, but it is not infinite storage. Every topic has a retention policy that decides how long records stay on disk before the broker reclaims the space. Getting retention right is one of the most consequential operational decisions you make: too short and consumers that fall behind lose data forever, too long and you pay for disks full of records nobody will ever read. This page explains time- and size-based retention, how segment files are actually deleted, and how to tune it per topic.

How retention works

Retention does not operate on individual records. Kafka stores each partition as a sequence of segment files, and retention is enforced by deleting whole segments once they become eligible. A segment becomes eligible when it is closed (rolled) and the entire segment falls outside the retention window. This is why retention is best understood as “approximately N hours” rather than an exact per-message expiry — a record physically survives until the segment that contains it is fully aged out and the broker’s background cleaner thread runs.

Two independent limits govern eligibility, and either one triggers deletion:

Setting	Scope	Meaning	Default
`retention.ms`	per partition	Maximum age of data before deletion. `-1` disables time-based deletion.	604800000 (7 days)
`retention.bytes`	per partition	Maximum size of a partition’s log. `-1` (default) disables size-based deletion.	-1
`segment.ms`	per partition	Force a new segment after this much time, even if not full.	604800000 (7 days)
`segment.bytes`	per partition	Roll to a new segment when the active one reaches this size.	1073741824 (1 GB)
`cleanup.policy`	per topic	`delete` (age/size based) or `compact` (key-based) or both.	delete

retention.bytes is per partition, not per topic. A topic with 12 partitions and retention.bytes=1073741824 can hold up to ~12 GB on disk. Multiply by the partition count when you size storage.

Segment rolling and deletion

The active (newest) segment is always being written and is never deleted. Records are only ever removed by aging out closed segments. The lifecycle looks like this:

[ seg .00 ]  [ seg .500 ]  [ seg .900 ]  <- active, being written
   closed        closed        open
   age 8d        age 3d        age 0
     |
     v  past retention.ms -> deleted as a whole file

A segment is rolled (closed and replaced) when it hits segment.bytes or segment.ms, whichever comes first. The log.retention.check.interval.ms broker setting (default 5 minutes) controls how often the cleaner scans for eligible segments. The practical consequence: a low retention.ms paired with a large segment.bytes may keep data far longer than expected, because data can’t be deleted until its segment is closed. If you need tight retention, lower segment.ms so segments roll more frequently.

The `delete` cleanup policy

cleanup.policy=delete is the default and is what “retention” normally refers to. Eligible segments are simply removed. It is the right choice for event streams, logs, metrics, and any data where old records have no lasting value. The contrasting policy, cleanup.policy=compact, instead retains the latest value per key indefinitely — covered on the log compaction page. You can even combine them with cleanup.policy=compact,delete to compact keys and still age out very old tombstones.

Setting retention per topic

Topic-level configs override the broker-wide log.retention.* defaults. Use kafka-configs.sh to alter them at runtime — no restart required.

Set a topic to keep data for 3 days:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=259200000

Cap a topic at 5 GB per partition and roll segments hourly:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name clickstream \
  --add-config retention.bytes=5368709120,segment.ms=3600000

Inspect the effective configuration of a topic:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --describe --entity-type topics --entity-name orders

Output:

Dynamic configs for topic orders are:
  retention.ms=259200000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=259200000}
  cleanup.policy=delete sensitive=false synonyms={DEFAULT_CONFIG:log.cleanup.policy=delete}

Remove a topic-level override to fall back to the broker default:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name orders \
  --delete-config retention.ms

You can also set retention at creation time:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic audit-events --partitions 6 --replication-factor 3 \
  --config retention.ms=2592000000 --config cleanup.policy=delete

Setting retention.ms=0 does not delete everything instantly — the active segment is never deleted, and closed segments are removed on the next check interval. To purge a topic, prefer kafka-delete-records.sh or recreate the topic.

Retention vs. compaction at a glance

Aspect	`delete` (retention)	`compact`
Keeps	Recent window by time/size	Latest value per key
Old data	Permanently removed	Latest per key retained forever
Best for	Event streams, logs, metrics	Changelogs, state, lookups
Triggered by	`retention.ms` / `retention.bytes`	`min.cleanable.dirty.ratio`, key changes

Best Practices

Set retention deliberately on every production topic rather than relying on the 7-day default — match it to how far your slowest consumer can realistically fall behind plus a safety margin.
Use retention.bytes as a guardrail against unbounded disk growth, and always remember it is per partition when sizing storage.
Lower segment.ms (for example to one hour) on topics that need tight, predictable retention, since data cannot be deleted until its segment is closed.
Monitor disk usage and consumer lag together; a topic near retention.bytes will silently drop data that lagging consumers have not yet read.
Prefer dynamic kafka-configs.sh overrides for per-topic tuning instead of editing broker server.properties, which requires restarts and applies cluster-wide.
Reach for cleanup.policy=compact (not shorter retention) when you need the latest state per key to persist indefinitely.