Event Streaming with Apache Kafka

Apache Kafka is a distributed event-streaming platform that lets services publish and subscribe to streams of records with high throughput, durability, and replay. Instead of services calling each other directly, producers append events to durable, append-only logs called topics, and consumers read from them at their own pace. This decouples teams and timelines: a producer does not need the consumer to be online, and new consumers can be added later to react to the same events. In Node.js, the most popular client is KafkaJS — a pure-JavaScript implementation with no native dependencies.

Why event streaming for microservices

Synchronous REST and gRPC calls couple the caller’s availability to the callee’s. If the inventory service is down, the order service stalls. Event streaming flips this around: the order service emits an order.created event and moves on. The inventory, billing, and notification services each consume that event independently and asynchronously.

Aspect	Request/response (REST, gRPC)	Event streaming (Kafka)
Coupling	Caller waits on callee	Producer and consumers decoupled
Delivery	One target	Many consumers, fan-out
Replay	Not possible	Re-read the log from any offset
Backpressure	Caller blocked	Consumers pace themselves
Throughput	Limited by sync hops	Very high, partitioned

Because Kafka retains events on disk, a brand-new analytics service can replay the entire history of order.created events on its first deploy — something a fire-and-forget message queue cannot do.

Core concepts

A topic is a named stream of events, split into one or more partitions. Partitions are the unit of parallelism and ordering: records within a single partition are strictly ordered, and Kafka guarantees order only within a partition, not across the whole topic. A record’s key determines its partition, so all events with the same key (for example, a given customerId) land in the same partition and stay ordered.

A consumer group is a set of consumers that cooperatively read a topic. Kafka assigns each partition to exactly one consumer in the group, so adding consumers (up to the partition count) scales throughput horizontally. Different groups each get their own independent copy of the stream — that is how fan-out to multiple services works.

Your maximum consumer parallelism per group equals the partition count. If a topic has 6 partitions, a 7th consumer in the same group sits idle. Provision partitions for your expected peak parallelism up front, because increasing them later breaks key-based ordering.

Installing and connecting

npm install kafkajs

Create a shared client module. The clientId identifies your app in broker logs, and brokers lists the bootstrap servers.

// kafka.js
import { Kafka, logLevel } from "kafkajs";

export const kafka = new Kafka({
  clientId: "order-service",
  brokers: (process.env.KAFKA_BROKERS ?? "localhost:9092").split(","),
  logLevel: logLevel.INFO,
  retry: { initialRetryTime: 300, retries: 8 },
});

Using CommonJS? Swap the import for const { Kafka, logLevel } = require("kafkajs"); — the rest of the API is identical.

Producing events

A producer connects once, then sends batches of messages to a topic. Setting a key controls partitioning and ordering. KafkaJS serializes only strings and Buffers, so encode objects as JSON.

// producer.js
import { kafka } from "./kafka.js";

const producer = kafka.producer();

export async function publishOrderCreated(order) {
  await producer.connect();
  await producer.send({
    topic: "order.created",
    messages: [
      {
        key: order.customerId, // same customer -> same partition -> ordered
        value: JSON.stringify(order),
        headers: { "content-type": "application/json" },
      },
    ],
  });
}

// Demo
await publishOrderCreated({ id: "o-1001", customerId: "c-42", total: 79.95 });
console.log("Published order.created for o-1001");
await producer.disconnect();

Output:

Published order.created for o-1001

Consuming with a consumer group

A consumer subscribes to one or more topics and processes records via eachMessage. The groupId ties this instance into a consumer group; run several copies with the same groupId and Kafka load-balances partitions across them automatically.

// consumer.js
import { kafka } from "./kafka.js";

const consumer = kafka.consumer({ groupId: "inventory-service" });

async function start() {
  await consumer.connect();
  await consumer.subscribe({ topic: "order.created", fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const order = JSON.parse(message.value.toString());
      console.log(
        `[p${partition} @${message.offset}] reserve stock for ${order.id}`
      );
      // reserveStock(order) ... real work here
    },
  });
}

start().catch((err) => {
  console.error("Consumer crashed", err);
  process.exit(1);
});

Output:

[p0 @0] reserve stock for o-1001
[p2 @0] reserve stock for o-1002
[p0 @1] reserve stock for o-1003

KafkaJS commits offsets automatically after eachMessage resolves. If your handler throws, the offset is not committed and the record is redelivered — giving you at-least-once delivery. Design handlers to be idempotent (for example, key writes on order.id) so a redelivery does not double-charge a customer.

Event-driven choreography

In a choreographed flow, each service reacts to events and emits new ones, with no central orchestrator. An order triggers a chain: inventory reserves stock and emits stock.reserved, billing charges the card and emits payment.captured, and shipping listens for both.

// billing-service.js
import { kafka } from "./kafka.js";

const consumer = kafka.consumer({ groupId: "billing-service" });
const producer = kafka.producer();

await producer.connect();
await consumer.connect();
await consumer.subscribe({ topic: "stock.reserved" });

await consumer.run({
  eachMessage: async ({ message }) => {
    const order = JSON.parse(message.value.toString());
    // charge(order) ...
    await producer.send({
      topic: "payment.captured",
      messages: [{ key: order.id, value: JSON.stringify(order) }],
    });
    console.log(`payment.captured emitted for ${order.id}`);
  },
});

Output:

payment.captured emitted for o-1001

Best practices

• Choose message keys deliberately — keying by entity id preserves per-entity ordering and spreads load evenly across partitions.
• Make consumers idempotent; at-least-once delivery means handlers will occasionally see the same event twice.
• Keep one long-lived producer and consumer per process; connecting per message is slow and exhausts broker connections.
• Provision enough partitions for your peak parallelism, since raising the count later disrupts key-based ordering.
• Handle errors so failing handlers do not commit offsets, and route poison messages to a dead-letter topic after a retry budget.
• Serialize with a schema (JSON Schema or Avro via a registry) so producers and consumers evolve without breaking each other.
• Disconnect producers and consumers on SIGTERM for a clean shutdown and prompt partition rebalancing.

Related Topics

• Microservices Introduction
• Event-Driven Architecture
• Message Queues
• gRPC in Node.js
• Saga Pattern