Apache Kafka in Production: Architecture, Semantics, and Reliable Patterns

Apache Kafka is a distributed event streaming platform designed for high-throughput, durable, and fault-tolerant data pipelines. Teams use Kafka to decouple services, ingest logs and telemetry, stream analytics, and build event-driven microservices where producers and consumers evolve independently.

Core Concepts: Brokers, Topics, Partitions, and Offsets

A Kafka cluster is made of brokers. Data is written to topics, and each topic is split into partitions for scalability. Producers append records to partitions, and Kafka assigns each record a monotonically increasing offset. Consumers read by offset and track progress independently, which makes replay and recovery straightforward.

• Broker: Kafka server responsible for storing and serving partition data.
• Topic: Logical stream of events (for example, orders, payments, or click events).
• Partition: Ordered append-only log shard that enables parallelism.
• Offset: Immutable position of a record within a partition.

Producers, Consumers, and Consumer Groups

Producers choose the destination topic and partition (directly or by key hashing). Consumers subscribe to topics and read records. A consumer group provides horizontal scale and fault tolerance: each partition is assigned to only one consumer in a group at a time, so processing is parallel without duplicate work in steady state.

acks=all
retries=2147483647
enable.idempotence=true
max.in.flight.requests.per.connection=5
compression.type=zstd

The producer configuration above prioritizes durability: `acks=all` waits for all in-sync replicas, retries tolerate transient errors, and idempotence prevents duplicate appends caused by retry races.

Delivery Semantics and Exactly-Once

Kafka supports at-most-once, at-least-once, and exactly-once processing paths, depending on producer and consumer behavior. At-most-once can lose data if offsets commit before processing. At-least-once is common in production and may duplicate processing during retries. Exactly-once requires idempotent producers plus transactions and careful sink integration.

• At-most-once: commit first, then process (fast but lossy).
• At-least-once: process first, then commit (durable but duplicates possible).
• Exactly-once: transactional producer + read-process-write transaction boundaries.

Ordering and Partitioning Strategy

Ordering in Kafka is guaranteed only within a single partition. If order matters for an entity (for example, accountId), partition by that key so all related events land in the same partition. Avoid hot partitions by selecting keys with enough cardinality and by monitoring skew in record distribution and consumer lag.

topic: transactions
partition key: accountId
result: strict per-account ordering with parallelism across accounts

Retention, Compaction, and Storage Model

Kafka stores data in segment files as an append-only log. Retention can be time-based, size-based, or both. Log compaction keeps the latest value per key and is ideal for changelog/state topics. Compaction does not guarantee immediate removal of stale records, so consumer logic should still tolerate old values during scans.

• Use delete retention for immutable event history and replay windows.
• Use compacted topics for latest-state materialization.
• Tune segment and retention settings according to disk budget and recovery objectives.

Schema Evolution with Avro and Schema Registry

Schema-managed serialization avoids fragile JSON contracts at scale. With Avro + Schema Registry, producers register schemas and write schema IDs with records. Consumers resolve compatible reader/writer schemas at runtime. Backward compatibility is commonly enforced to let old consumers keep running while producers evolve.

{
  "type": "record",
  "name": "OrderCreated",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "totalAmount", "type": "double"},
    {"name": "couponCode", "type": ["null", "string"], "default": null}
  ]
}

The optional `couponCode` field demonstrates safe evolution by providing a default, preserving compatibility for existing consumers.

Reliability and Operational Best Practices

Production reliability depends on coordinated broker, producer, and topic configuration. A common baseline is replication factor 3, `min.insync.replicas=2`, and producer `acks=all`, which tolerates one broker failure without sacrificing durability.

• Set replication factor >= 3 for critical topics.
• Use `min.insync.replicas` to prevent acknowledged writes with weak durability.
• Enable idempotent producers and configure retries for transient failures.
• Monitor consumer lag, under-replicated partitions, ISR shrink events, and request latency.
• Plan partition count early; increasing partitions later can impact key-based ordering assumptions.

Common Pitfalls and Practical Recommendations

• Pitfall: using random partitioning for entities that need order. Recommendation: partition by stable business key.
• Pitfall: committing offsets before side effects are durable. Recommendation: commit after successful processing.
• Pitfall: unbounded topic growth. Recommendation: define retention and compaction policies per topic category.
• Pitfall: breaking schema changes. Recommendation: enforce compatibility in Schema Registry CI checks.
• Pitfall: assuming exactly-once across external DB writes automatically. Recommendation: use idempotent sinks or transaction/outbox patterns.

Kafka is most effective when treated as a foundational data platform, not just a queue. Clear event contracts, keying strategy, and reliability defaults will prevent the majority of production incidents and make downstream analytics and services significantly easier to scale.