Kafka Basics: Events, Partitions, and Consumer Groups

Quickstart: the fastest way to understand Kafka

If you only do one thing, make Kafka feel tangible: run it locally, create a multi-partition topic, then start two consumers in the same group and watch Kafka split the work. The goal isn’t a perfect setup—it’s a working mental model you can reuse.

Overview: what this post covers (and why it matters)

Kafka is often introduced as “a messaging system,” but that framing hides the two ideas that make it powerful: durability (events are persisted and replayable) and scalable consumption (consumer groups coordinate parallel processing).

You’ll also get a minimal local setup and a few operational habits (like checking consumer lag) that save hours when you move from “toy pipeline” to “production job.”

Core concepts: the mental model that prevents confusion

Here are the terms you’ll see everywhere, explained in the “what it is / what it does / why you should care” style. If you internalize these, Kafka stops feeling magical.

Events: what to include (and what to avoid)

A Kafka event is typically a key/value pair with metadata. Your “value” is often JSON/Avro/Protobuf, but the bigger question is: can someone process this later without guessing?

Partitions: ordering, scaling, and why keys matter

Partitions exist so Kafka can scale. But they also define what “in order” means. Kafka can preserve order for events that land in the same partition—so you typically use a key that groups related events (like order_id) to keep their sequence consistent.

Consumer groups: “one partition, one consumer” (within a group)

A consumer group is Kafka’s scaling primitive for reading: Kafka assigns partitions to consumers so that a partition is processed by at most one consumer in the group at a time. Add consumers and Kafka may rebalance assignments. Add partitions and you increase maximum parallelism.

Step-by-step: run Kafka locally and see the behavior

This is a practical walk-through you can copy/paste. It uses a single-node setup (good for learning) and focuses on the three things that trip people up: partitions, keys, and consumer groups.

Step 1 — Start Kafka locally (single broker, KRaft)

For a local demo, we keep it simple: one broker, plaintext listener, and a small amount of storage. This is not “production secure,” and that’s okay—the goal is to observe behavior.

version: "3.8"

services:
  kafka:
    image: bitnami/kafka:latest
    container_name: kafka
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      # KRaft (no ZooKeeper) single-node demo
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER

      # Listeners: one for inside Docker, one for your host machine
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092,CONTROLLER://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - ALLOW_PLAINTEXT_LISTENER=yes

      # Demo-friendly defaults
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=false
      - KAFKA_CFG_NUM_PARTITIONS=3
      - KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
      - KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=1

Step 2 — Create a topic and produce a few keyed events

We’ll create a topic named payments with 3 partitions. Then we’ll publish a few events with a key so related events stick to the same partition (your future self will thank you when debugging ordering).

# Create a 3-partition topic (replication=1 for a single-broker demo)
docker exec -it kafka /opt/bitnami/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic payments --partitions 3 --replication-factor 1

# Describe it (check partitions)
docker exec -it kafka /opt/bitnami/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe --topic payments

# Produce keyed events: KEY|VALUE
# (parse.key=true means split key/value using key.separator)
docker exec -it kafka /opt/bitnami/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic payments \
  --property parse.key=true \
  --property key.separator="|"

Step 3 — Consume with a consumer group (and watch parallelism)

Now the payoff: start consumers with the same group id. Kafka will assign partitions across them. With 3 partitions, at most 3 consumers can actively read in parallel in a single group.

import json
from confluent_kafka import Consumer, KafkaException

BOOTSTRAP = "localhost:29092"
TOPIC = "payments"
GROUP_ID = "demo-payments"

conf = {
    "bootstrap.servers": BOOTSTRAP,
    "group.id": GROUP_ID,
    "auto.offset.reset": "earliest",      # first run only; after commits it will resume
    "enable.auto.commit": True,           # start simple; see notes below for manual commits
}

c = Consumer(conf)
c.subscribe([TOPIC])

try:
    while True:
        msg = c.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            raise KafkaException(msg.error())

        key = msg.key().decode("utf-8") if msg.key() else None
        value = msg.value().decode("utf-8") if msg.value() else None

        payload = None
        if value:
            try:
                payload = json.loads(value)
            except json.JSONDecodeError:
                payload = value

        print(
            f"topic={msg.topic()} partition={msg.partition()} offset={msg.offset()} key={key} value={payload}"
        )

except KeyboardInterrupt:
    pass
finally:
    c.close()

Step 4 — Check consumer lag (the fastest health signal)

When a pipeline “feels slow,” consumer lag usually tells you whether you’re keeping up. Lag is the gap between the end of the log and the group’s current committed offset.

Step 5 — Choose partitions and consumer groups deliberately

This is where Kafka design becomes “engineering” instead of “copy a tutorial.” Use partitions for parallelism and ordering boundaries; use consumer groups for scaling a specific workload.

Common mistakes (and how to fix them)

Most Kafka confusion comes from a handful of predictable assumptions. Here are the ones that cause the most production pain, with fixes that are straightforward once you know what to look for.

FAQ

Is Kafka a queue or a log?

Kafka is a distributed log. It can behave like a queue when a single consumer group processes a topic, but the underlying model is append + retain + replay, with consumption tracked via offsets.

What’s the difference between a topic and a partition?

A topic is the named stream; partitions are shards inside it. Partitions are what Kafka uses to scale storage and throughput. Ordering guarantees apply within a partition, not across the entire topic.

How many partitions do I need?

Start with the parallelism you need and grow intentionally. A consumer group can process up to one partition per consumer at a time, so partitions set your max parallelism. Don’t over-partition early; too many partitions can increase overhead (more files, more coordination).

Why are my events “out of order”?

Because ordering is per partition. If related events use different keys (or no key), they may land in different partitions and be consumed in different orders. Fix it by choosing a stable key (like order_id) for sequences that must stay ordered.

Can two different consumer groups read the same topic?

Yes—and that’s a feature. Each consumer group maintains its own offsets, so multiple teams/systems can independently process the same event stream without interfering with each other.

What does committing an offset actually do?

It records the group’s progress. Committing an offset means “we have processed up to this point in this partition.” It does not delete messages. Deletion is controlled by retention and compaction settings.

Do I still need ZooKeeper?

Newer Kafka deployments use KRaft (no ZooKeeper) for metadata management. Some existing clusters still run with ZooKeeper, so you may see both in the wild, but the “modern default” is moving toward ZooKeeper-free setups.

Cheatsheet

Use this as a “quick recall” when building or reviewing a Kafka pipeline. It’s intentionally short and practical.

Wrap-up

Kafka is easier than it looks once you stop thinking “messages” and start thinking “append-only logs with coordinated readers.” Events live in topics, topics scale via partitions, and consumer groups let you scale processing without duplicating work. If you can answer “what’s my key?” and “how many partitions do I need?” you’re already ahead of most first deployments.

When you’re ready to go deeper, the related posts below are great follow-ups: query performance, explain plans, modeling, and modern analytics architecture decisions.

Related posts

• SQL That Scales: Joins, Windows, and Real-World Patterns
• Explain Plans Demystified: Find the Slow Part Fast
• Star Schema vs Snowflake: Modeling for Analytics
• Postgres Power Moves: Indexes, JSONB, and Partitioning
• Warehouse vs Lake vs Lakehouse: What to Pick in 2026
• dbt in Practice: The Clean Way to Build Transformations