Event-Driven Architecture with Apache Kafka

Introduction

Event-driven architecture (EDA) decouples services by communicating through events rather than direct API calls. Apache Kafka is the de facto standard for building these systems at scale. This guide covers the core concepts and practical patterns I've used in production.

Why Event-Driven?

Traditional request-response architectures create tight coupling between services. When Service A calls Service B directly, both must be available simultaneously, and changes to B's API break A. With events:

• Loose coupling: Producers don't know who consumes their events
• Scalability: Consumers scale independently based on load
• Resilience: If a consumer is down, events queue up and replay when it recovers
• Audit trail: Every state change is recorded as an immutable event

Kafka Core Concepts

Topics & Partitions

A topic is a named stream of events. Topics are split into partitions for parallelism. Each partition is an ordered, immutable log.

# Create a topic with 3 partitions
kafka-topics --create --topic order-events \
  --partitions 3 --replication-factor 2 \
  --bootstrap-server localhost:9092

Producers

Producers publish events to topics. Use a partition key to ensure related events go to the same partition (maintaining order).

const { Kafka } = require('kafkajs');

const kafka = new Kafka({ brokers: ['localhost:9092'] });
const producer = kafka.producer();

await producer.connect();
await producer.send({
  topic: 'order-events',
  messages: [{
    key: orderId,  // ensures all events for this order go to same partition
    value: JSON.stringify({
      type: 'ORDER_PLACED',
      orderId,
      customerId,
      items,
      timestamp: Date.now()
    })
  }]
});

Consumers & Consumer Groups

Consumers read events from topics. A consumer group distributes partitions across group members — each partition is read by exactly one consumer in the group.

const consumer = kafka.consumer({ groupId: 'inventory-service' });

await consumer.connect();
await consumer.subscribe({ topic: 'order-events', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());
    
    if (event.type === 'ORDER_PLACED') {
      await reserveInventory(event.items);
    }
    if (event.type === 'ORDER_CANCELLED') {
      await releaseInventory(event.items);
    }
  }
});

Real-World Patterns

1. Event Sourcing

Store every state change as an event. Reconstruct current state by replaying events. Useful for financial systems, audit trails, and undo/redo functionality.

2. CQRS (Command Query Responsibility Segregation)

Separate write (command) and read (query) models. Commands produce events, events update read-optimized views. Kafka acts as the bridge between the two.

3. Saga Pattern

For distributed transactions spanning multiple services. Each service publishes success/failure events, and compensating actions roll back on failure.

Order Service → ORDER_PLACED
Payment Service → PAYMENT_COMPLETED (or PAYMENT_FAILED)
Inventory Service → INVENTORY_RESERVED (or INVENTORY_INSUFFICIENT)
Shipping Service → SHIPMENT_CREATED

If any step fails → compensating events undo previous steps

Production Considerations

• Idempotency: Consumers must handle duplicate events gracefully (at-least-once delivery)
• Schema evolution: Use Avro/Protobuf with a Schema Registry to evolve event schemas safely
• Dead letter queues: Route failed events to a DLQ for manual inspection
• Monitoring: Track consumer lag — if it grows, your consumer can't keep up
• Retention: Configure topic retention based on your replay needs (7 days default, infinite for event sourcing)

When to Use Kafka

• High-throughput event streaming (100K+ events/sec)
• Decoupling microservices that don't need synchronous responses
• Real-time data pipelines (logs, metrics, user activity)
• Event sourcing and audit trails

Don't use Kafka for simple request-response between two services — a REST call is simpler. Use it when you need durability, replay, and fan-out to multiple consumers.

Conclusion

Kafka transforms how services communicate — from fragile direct calls to resilient, scalable event streams. Start with a single topic for your highest-traffic integration, prove the pattern, then expand. The investment in infrastructure pays for itself in reliability and flexibility.