AI & Software Solution
Back to All Articles
Data Engineering

Apache Kafka: A Complete Production Guide to Event Streaming

GE

Guvantic Engineering

Guvantic Technical Publication

16 min
Apache Kafka: A Complete Production Guide to Event Streaming
Executive Architectural Overview (AEO Direct Answer)

Master Apache Kafka from core concepts to production deployment. Learn about topics, partitions, consumer groups, and how to build reliable event-driven architectures with practical code examples.

Kafka powers real-time data pipelines at companies like Netflix, Uber, and LinkedIn. This guide provides everything you need to build production-grade event streaming systems.

The Problem: Data Integration Complexity

Modern applications generate data everywhere: user clicks, payments, logs, IoT sensors. Moving this data between services is traditionally done with point-to-point integrations that become unmaintainable at scale.

Architectural Topology Specification
Before Kafka: The Integration Mess — Shows a spiderweb of point-to-point connections between databases, services, and analytics systems.

Kafka solves this by:

1.Decoupling producers from consumers — No direct dependencies
2.Persisting data — Events aren't lost if consumers are offline
3.Allowing multiple consumers — Same data feeds many systems
4.Providing high throughput — Millions of events per second

Core Architecture: Topics, Partitions, and Offsets

Architectural Topology Specification
Kafka Core Architecture — Shows:
Producer → Topic (with 3 partitions) → Consumer Group 1, Consumer Group 2
Each partition shows offsets 0, 1, 2, 3, 4
Consumer Group 1: Consumer 1 (Partition 0), Consumer 2 (Partition 1), Consumer 3 (Partition 2)
Consumer Group 2: Single consumer reading all partitions

Key Concepts Explained:

Topics: Named streams of records (e.g., "payment-events", "user-clicks")

Partitions: Each topic is split into partitions — ordered, immutable sequences of records. Partitions enable parallelism.

Offsets: Unique position of each record within a partition. Consumers track their position.

Consumer Groups: Multiple consumers that share the workload. Each partition is assigned to exactly one consumer per group.

Producers: Sending Events

Producer Code Example (Java)

javaProduction snippet
import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class EventProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "kafka-1:9092,kafka-2:9092,kafka-3:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all"); // Wait for all replicas
        props.put("retries", 3);
        props.put("linger.ms", 10); // Batch for 10ms
        props.put("batch.size", 16384);
        props.put("enable.idempotence", true);

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        try {
            // Send with key for consistent partition assignment
            ProducerRecord<String, String> record = new ProducerRecord<>(
                "orders",
                "user-123", // Key determines partition
                "{\"orderId\":\"ord-456\",\"amount\":99.99}"
            );
            
            producer.send(record, (metadata, exception) -> {
                if (exception != null) {
                    exception.printStackTrace();
                } else {
                    System.out.printf("Sent to partition %d offset %d%n",
                        metadata.partition(), metadata.offset());
                }
            });
            
            producer.flush();
        } finally {
            producer.close();
        }
    }
}

Producer Code Example (Node.js)

javascriptProduction snippet
const { Kafka } = require("kafkajs");

const kafka = new Kafka({
  clientId: "order-service",
  brokers: ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"],
});

const producer = kafka.producer({
  allowAutoTopicCreation: false,
  idempotent: true,
});

async function produceEvent() {
  await producer.connect();
  
  const result = await producer.send({
    topic: "orders",
    messages: [
      {
        key: "user-123", // Consistent partition
        value: JSON.stringify({ orderId: "ord-456", amount: 99.99 }),
        headers: { source: "web-checkout" },
      },
    ],
  });
  
  console.log("Sent to partition", result[0].partition);
  await producer.disconnect();
}

Consumers: Reading Events

Consumer Code Example (Java)

javaProduction snippet
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Properties;
import java.util.Arrays;

public class OrderConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "kafka-1:9092,kafka-2:9092,kafka-3:9092");
        props.put("group.id", "order-processor-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("enable.auto.commit", "false");
        props.put("auto.offset.reset", "earliest");
        props.put("max.poll.records", "500");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("orders"));

        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
                records.forEach(record -> {
                    System.out.printf("Partition: %d, Offset: %d, Key: %s, Value: %s%n",
                        record.partition(), record.offset(), record.key(), record.value());
                    
                    // Process the order
                    processOrder(record.value());
                });
                
                // Commit offsets after processing
                consumer.commitSync();
            }
        } finally {
            consumer.close();
        }
    }
    
    private static void processOrder(String orderJson) {
        // Business logic here
    }
}

Consumer Code Example (Node.js with consumer groups)

javascriptProduction snippet
const { Kafka } = require("kafkajs");
const { v4: uuidv4 } = require("uuid");

const kafka = new Kafka({
  clientId: "order-processor",
  brokers: ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"],
});

const consumer = kafka.consumer({
  groupId: "order-processor-group",
  sessionTimeout: 30000,
  rebalanceTimeout: 60000,
  allowAutoTopicCreation: false,
});

async function startConsumer() {
  await consumer.connect();
  await consumer.subscribe({
    topic: "orders",
    fromBeginning: false,
  });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      try {
        const order = JSON.parse(message.value.toString());
        console.log(`Processing order ${order.orderId} from partition ${partition}`);
        
        // Process with idempotency check
        const processed = await processOrder(order, message.offset);
        
        // Manual commit with offset
        await consumer.commitOffsets([{
          topic,
          partition,
          offset: (BigInt(message.offset) + 1n).toString(),
        }]);
      } catch (error) {
        console.error("Processing error:", error);
        // Send to dead letter topic
        await sendToDLQ(message);
      }
    },
    autoCommit: false, // Manual commit for exact once
  });
}

Kafka vs Message Queues

Architectural Topology Specification
Kafka vs RabbitMQ Decision Matrix — Shows:
Kafka: High throughput, event streaming, replay, complex routing
RabbitMQ: Task distribution, request-reply, low latency, complex routing
Use cases with recommendations

| Feature | Kafka | RabbitMQ |

|---------|-------|----------|

| Message Retention | Configurable (days/weeks) | Deleted after ack |

| Replay | Yes | No |

| Throughput | Millions/sec | Tens of thousands/sec |

| Ordering | Per partition | Per queue (with limitations) |

| Multiple Consumers | Consumer groups | Multiple queues |

| Complex Routing | Topics only | Exchanges and bindings |

| Use Case | Event streaming, log aggregation | Task distribution, RPC |

Production Deployment Best Practices

Architectural Topology Specification
Production Kafka Cluster Architecture — Shows:
3+ ZooKeeper nodes (or KRaft)
3+ Broker nodes with replication
Producers with retry and idempotency
Consumers with monitoring for lag
Schema Registry for schema evolution
Prometheus/Grafana for monitoring
Alerts for consumer lag, broker health

Configuration Checklist:

Broker Configuration:

propertiesProduction snippet
# Minimal production broker config
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka-1:9092
log.dirs=/data/kafka
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

# Performance tuning
log.segment.bytes=1073741824
log.retention.hours=168
compression.type=snappy

Monitoring Consumer Lag

bashProduction snippet
# Check consumer lag
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --group order-processor-group --describe

# Output:
# GROUP              TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# order-processor    orders  0          150              180             30
# order-processor    orders  1          200              200             0
# order-processor    orders  2          75               90              15

Alert on Lag:

yamlProduction snippet
# Prometheus alert rule
groups:
  - name: kafka_alerts
    rules:
    - alert: KafkaConsumerLag
      expr: kafka_consumer_lag > 1000
      for: 5m
      annotations:
        summary: "Consumer lag exceeds 1000 messages"

Getting Started with Guvantic

We help companies build production-ready event-driven architectures. Our services include:

Kafka Cluster Setup: AWS MSK, Confluent Cloud, or self-managed
Schema Management: Avro/Protobuf with Schema Registry
Monitoring: Grafana dashboards and alerting
Migration Support: Transition from legacy integrations

Start with a managed service like AWS MSK or Confluent Cloud to avoid operational overhead.

Keywords:#Apache Kafka#event streaming#message broker#data pipeline#microservices#Kafka production
GE

Guvantic Engineering

Engineering & System Architecture Division

Engineering teams at Guvantic deliver mission-critical software, high-concurrency microservices, and international organic positioning strategies for fast-growing companies in USA, UK, and India.

Related Engineering Insights