Kafka Lab — Hands-On Learning Playground

Architecture

Quick Start

            # Download and extract

            tar xzf kafka-lab.tar.gz && cd kafka-lab

            # Start 3-broker cluster + Kafka UI

            ./setup.sh

            # Open Kafka UI in browser

            open http://localhost:8080

            # Produce 100 messages

            cd kotlin-client

            ./gradlew run --args="produce --count 100"

            # Consume them

            ./gradlew run --args="consume --group my-group --from-beginning"

Prerequisites

Docker & Docker Compose — for the 3-broker cluster
JDK 17+ — for the Kotlin client
~2 GB RAM free — 3 Kafka brokers + Schema Registry + UI

What's Inside

kafka-lab/
├── docker-compose.yml — 3-broker KRaft cluster + Kafka UI + Schema Registry
├── setup.sh — Start cluster, create 4 topics
├── teardown.sh — Stop everything, remove volumes
├── observe.sh — CLI snapshot: topics, consumer groups, lag
├── chaos.sh — Kill a broker, watch failover, restart
├── README.md — Full guide with 7 experiments
└── kotlin-client/
    ├── build.gradle.kts
    └── src/main/kotlin/.../
        ├── ProducerExample.kt — Configurable producer (acks, keys, batching)
        ├── ConsumerExample.kt — Consumer with auto/manual commit
        ├── MultiConsumerDemo.kt — Partition rebalancing demo
        ├── ExactlyOnceDemo.kt — Transactional read-process-write
        └── Main.kt — CLI entry point

Kotlin Client Commands

            # Produce messages

            ./gradlew run --args="produce --count 100"

            ./gradlew run --args="produce --count 1000 --delay-ms 50 --acks all"

            ./gradlew run --args="produce --count 50 --key-mode random"

            # Consume messages

            ./gradlew run --args="consume --group my-group --from-beginning"

            ./gradlew run --args="consume --group slow --process-time-ms 500"

            ./gradlew run --args="consume --group manual --auto-commit false"

            # Multi-consumer (partition assignment demo)

            ./gradlew run --args="multi --consumers 3 --group test"

            # Exactly-once semantics demo

            ./gradlew run --args="exactly-once"

7 Guided Experiments

1. Partition Assignment & Rebalancing

Learn: How consumer groups distribute work

Start one consumer, then add a second in the same group. Watch partitions get redistributed. Kill one — watch them rebalance back. The core mechanism behind Kafka's horizontal scaling.

2. Ordering Guarantees

Learn: Why keys matter

Produce with --key-mode userid. Same userId always goes to the same partition → messages arrive in order. Switch to --key-mode random → no ordering guarantee. This is the #1 thing interviewers ask about.

3. At-Least-Once vs At-Most-Once

Learn: The fundamental delivery tradeoff

Consume with auto-commit (at-most-once) and kill mid-processing — messages are lost. Switch to manual commit (at-least-once) — messages get reprocessed (duplicates). Neither is "exactly-once" on its own.

4. Broker Failure & Recovery

Learn: Replication and fault tolerance

Run ./chaos.sh to kill a broker while producing with acks=all. The producer continues after a brief pause. ISR shrinks, then recovers when the broker comes back. Zero message loss.

5. Consumer Lag

Learn: Monitoring slow consumers

Produce fast, consume slow (--process-time-ms 500). Watch lag grow in Kafka UI. This is the most important operational metric — if lag grows unboundedly, you need more consumers or faster processing.

6. Exactly-Once Semantics

Learn: Transactional produce-consume

The exactly-once demo reads from orders, transforms, writes to processed-orders — atomically. It simulates a mid-transaction crash to show zero partial writes. Check both topics in Kafka UI.

7. Acks Modes & Throughput

Learn: Durability vs performance tradeoff

Produce 1000 messages with acks=0, acks=1, and acks=all. Compare throughput numbers. acks=0 is fastest but can lose data. acks=all waits for all replicas — slowest but safest.

Pre-Created Topics

Topic	Partitions	Replication	Notes
orders	3	3	Main topic for experiments
events	6	2	More partitions = more parallelism
compacted-users	3	2	Log compaction — keeps latest per key
processed-orders	3	3	Output of exactly-once demo

How to Observe

Kafka UI (http://localhost:8080)

The visual dashboard — open alongside your terminal:

Topics → orders → Partitions — leader broker, replica set, ISR for each partition
Topics → orders → Messages — browse actual messages with partition, offset, key, value
Consumer Groups — which consumer owns which partitions, current offset vs end offset (= lag)
Brokers — health, disk usage, partition count per broker

CLI

            # Quick cluster snapshot

            ./observe.sh

            # Watch producer output: which partition each message goes to

            # Watch consumer output: partition, offset, processing time

🧪 Kafka Lab

Download kafka-lab