Kafka Engineer

Purpose

Provides Apache Kafka and event streaming expertise specializing in scalable event-driven architectures and real-time data pipelines. Builds fault-tolerant streaming platforms with exactly-once processing, Kafka Connect, and Schema Registry management.

When to Use

Designing event-driven microservices architectures
Setting up Kafka Connect pipelines (CDC, S3 Sink)
Writing stream processing apps (Kafka Streams / ksqlDB)
Debugging consumer lag, rebalancing storms, or broker performance
Designing schemas (Avro/Protobuf) with Schema Registry
Configuring ACLs and mTLS security

2. Decision Framework

Architecture Selection

What is the use case?
│
├─ **Data Integration (ETL)**
│  ├─ DB to DB/Data Lake? → **Kafka Connect** (Zero code)
│  └─ Complex transformations? → **Kafka Streams**
│
├─ **Real-Time Analytics**
│  ├─ SQL-like queries? → **ksqlDB** (Quick aggregation)
│  └─ Complex stateful logic? → **Kafka Streams / Flink**
│
└─ **Microservices Comm**
   ├─ Event Notification? → **Standard Producer/Consumer**
   └─ Event Sourcing? → **State Stores (RocksDB)**

Config Tuning (The "Big 3")

Throughput:
```
batch.size
```
,
```
linger.ms
```
,
```
compression.type=lz4
```
.
Latency:
```
linger.ms=0
```
,
```
acks=1
```
.

Durability:

acks=all

min.insync.replicas=2

replication.factor=3

Red Flags → Escalate to
sre-engineer
:

"Unclean leader election" enabled (Data loss risk)
Zookeeper dependency in new clusters (Use KRaft mode)
Disk usage > 80% on brokers
Consumer lag constantly increasing (Capacity mismatch)

3. Core Workflows

Workflow 1: Kafka Connect (CDC)

Goal: Stream changes from PostgreSQL to S3.

Steps:

Source Config (
postgres-source.json
)

json

{
  "name": "postgres-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db-host",
    "database.dbname": "mydb",
    "database.user": "kafka",
    "plugin.name": "pgoutput"
  }
}

Sink Config (
s3-sink.json
)

json

{
  "name": "s3-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "s3.bucket.name": "my-datalake",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "flush.size": "1000"
  }
}

Deploy

curl -X POST -d @postgres-source.json http://connect:8083/connectors

Workflow 3: Schema Registry Integration

Goal: Enforce schema compatibility.

Steps:

Define Schema (
user.avsc
)

json

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
  ]
}

Producer (Java)

Use
```
KafkaAvroSerializer
```
.
Registry URL:
```
http://schema-registry:8081
```
.

5. Anti-Patterns & Gotchas

❌ Anti-Pattern 1: Large Messages

What it looks like:

Sending 10MB images payload in Kafka message.

Why it fails:

Kafka is optimized for small messages (< 1MB). Large messages block the broker threads.

Correct approach:

Store image in S3.
Send Reference URL in Kafka message.

❌ Anti-Pattern 2: Too Many Partitions

What it looks like:

Creating 10,000 partitions on a small cluster.

Why it fails:

Slow leader election (Zookeeper overhead).
High file handle usage.

Correct approach:

Limit partitions per broker (~4000). Use fewer topics or larger clusters.

❌ Anti-Pattern 3: Blocking Consumer

What it looks like:

Consumer doing heavy HTTP call (30s) for each message.

Why it fails:

Rebalance storm (Consumer leaves group due to timeout).

Correct approach:

Async Processing: Move work to a thread pool.
Pause/Resume:
```
consumer.pause()
```
if buffer is full.

7. Quality Checklist

Configuration:

Replication: Factor 3 for production.
Min.ISR: 2 (Prevents data loss).
Retention: Configured correctly (Time vs Size).

Observability:

Lag: Consumer Lag monitored (Burrow/Prometheus).
Under-replicated: Alert on under-replicated partitions (>0).
JMX: Metrics exported.

Examples

Example 1: Real-Time Fraud Detection Pipeline

Scenario: A financial services company needs real-time fraud detection using Kafka streaming.

Architecture Implementation:

Event Ingestion: Kafka Connect CDC from PostgreSQL transaction database
Stream Processing: Kafka Streams application for real-time pattern detection
Alert System: Producer to alert topic triggering notifications
Storage: S3 sink for historical analysis and compliance

Pipeline Configuration:

Component	Configuration	Purpose
Topics	3 (transactions, alerts, enriched)	Data organization
Partitions	12 (3 brokers × 4)	Parallelism
Replication	3	High availability
Compression	LZ4	Throughput optimization

Key Logic:

Detects velocity patterns (5+ transactions in 1 minute)
Identifies geographic anomalies (impossible travel)
Flags high-risk merchant categories

Results:

99.7% of fraud detected in under 100ms
False positive rate reduced from 5% to 0.3%
Compliance audit passed with zero findings

Example 2: E-Commerce Order Processing System

Scenario: Build a resilient order processing system with Kafka for high reliability.

System Design:

Order Events: Topic for order lifecycle events
Inventory Service: Consumes orders, updates stock
Payment Service: Processes payments, publishes results
Notification Service: Sends confirmations via email/SMS

Resilience Patterns:

Dead Letter Queue for failed processing
Idempotent producers for exactly-once semantics
Consumer groups with manual offset management
Retries with exponential backoff

Configuration:

yaml

# Producer Configuration
acks: all
retries: 3
enable.idempotence: true

# Consumer Configuration
auto.offset.reset: earliest
enable.auto.commit: false
max.poll.records: 500

Results:

99.99% message delivery reliability
Zero duplicate orders in 6 months
Peak processing: 10,000 orders/second

Example 3: IoT Telemetry Platform

Scenario: Process millions of IoT device telemetry messages with Kafka.

Platform Architecture:

Device Gateway: MQTT to Kafka proxy
Data Enrichment: Stream processing adds device metadata
Time-Series Storage: S3 sink partitioned by device_id/date
Real-Time Alerts: Threshold-based alerting for anomalies

Scalability Configuration:

50 partitions for parallel processing
Compression enabled for cost optimization
Retention: 7 days hot, 1 year cold in S3
Schema Registry for data contracts

Performance Metrics:

Metric	Value
Throughput	500,000 messages/sec
Latency (P99)	50ms
Consumer lag	< 1 second
Storage efficiency	60% reduction with compression

Best Practices

Topic Design

Naming Conventions: Use clear, hierarchical topic names (domain.entity.event)
Partition Strategy: Plan for future growth (3x expected throughput)
Retention Policies: Match retention to business requirements
Cleanup Policies: Use delete for time-based, compact for state
Schema Management: Enforce schemas via Schema Registry

Producer Optimization

Batching: Increase batch.size and linger.ms for throughput
Compression: Use LZ4 for balance of speed and size
Acks Configuration: Use all for reliability, 1 for latency
Retry Strategy: Implement retries with backoff
Idempotence: Enable for exactly-once semantics in critical paths

Consumer Best Practices

Offset Management: Use manual commit for critical processing
Batch Processing: Increase max.poll.records for efficiency
Rebalance Handling: Implement graceful shutdown
Error Handling: Dead letter queues for poison messages
Monitoring: Track consumer lag and processing time

Security Configuration

Encryption: TLS for all client-broker communication
Authentication: SASL/SCRAM or mTLS for production
Authorization: ACLs with least privilege principle
Quotas: Implement client quotas to prevent abuse
Audit Logging: Log all access and configuration changes

Performance Tuning

Broker Configuration: Optimize for workload type (throughput vs latency)
JVM Tuning: Heap size and garbage collector selection
OS Tuning: File descriptor limits, network settings
Monitoring: Metrics for throughput, latency, and errors
Capacity Planning: Regular review and scaling assessment

Security:

Encryption: TLS enabled for Client-Broker and Inter-broker.
Auth: SASL/SCRAM or mTLS enabled.
ACLs: Principle of least privilege (Topic read/write).

kafka-engineer

NPX Install

Tags

SKILL.md Content

Kafka Engineer

Purpose

When to Use

2. Decision Framework

Architecture Selection

Config Tuning (The "Big 3")

3. Core Workflows

Workflow 1: Kafka Connect (CDC)

Workflow 3: Schema Registry Integration

5. Anti-Patterns & Gotchas

❌ Anti-Pattern 1: Large Messages

❌ Anti-Pattern 2: Too Many Partitions

❌ Anti-Pattern 3: Blocking Consumer

7. Quality Checklist

Examples

Example 1: Real-Time Fraud Detection Pipeline

Example 2: E-Commerce Order Processing System

Example 3: IoT Telemetry Platform

Best Practices

Topic Design

Producer Optimization

Consumer Best Practices

Security Configuration

Performance Tuning