Home

Setting up an Apache Kafka cluster in KRaft mode

What is Kafka?

Apache Kafka is a distributed event streaming platform — equal parts message queue, event store, and stream-processing system. Written in Java/Scala by the Apache Foundation, open source, and pretty much the default choice for high-throughput event pipelines.

KRaft mode

For most of Kafka's life, the cluster's metadata (topics, brokers, ACLs, partition assignments, etc.) was stored in Apache ZooKeeper — a separate service you had to run alongside your brokers. Since Kafka 2.8, an alternative called KRaft (Kafka Raft) lets the cluster manage its own metadata using a built-in Raft consensus protocol. No ZooKeeper, fewer moving parts, simpler ops. As of Kafka 3.5+, KRaft is production-ready and the recommended path for new deployments. ZooKeeper-based mode is being removed in 4.x.

This post sets up a 3-node Kafka 3.5 cluster in KRaft mode on Ubuntu.

What you'll need

Install Java

On every node:

bash
sudo apt-get install openjdk-11-jdk

Install Kafka

Example IPs (replace with your own):

bash
kafka1 = 192.168.159.135kafka2 = 192.168.159.136kafka3 = 192.168.159.137

Create a dedicated kafka user (every node):

bash
sudo useradd kafka

Create the data and install directories. Each node gets its own log directory:

bash
# node 1sudo mkdir -p /data/kafka/kafka1.logs# node 2sudo mkdir -p /data/kafka/kafka2.logs# node 3sudo mkdir -p /data/kafka/kafka3.logs

Install path (every node):

bash
sudo mkdir -p /opt/kafkasudo chmod 775 -R /data/kafka /opt/kafkasudo chown -R kafka:kafka /data/kafka /opt/kafka

Download Kafka — anything ≥ 2.8 supports KRaft, but use a 3.5+ release for production-grade KRaft. I'm using 3.5.0:

bash
cd /opt/kafkasudo wget https://archive.apache.org/dist/kafka/3.5.0/kafka_2.13-3.5.0.tgzsudo tar -xzf kafka_2.13-3.5.0.tgz

(All releases: kafka.apache.org/downloads.)

Configure the cluster

Each node has its own server.properties under config/kraft/. The fields you need to set per-node: node.id, the listener IPs, and the log directory. The controller.quorum.voters line is identical on every node.

Node 1 (/opt/kafka/kafka_2.13-3.5.0/config/kraft/server.properties):

properties
process.roles=broker,controllernode.id=1controller.quorum.voters=[email protected]:9093,[email protected]:9093,[email protected]:9093listeners=PLAINTEXT://192.168.159.135:9092,CONTROLLER://192.168.159.135:9093advertised.listeners=PLAINTEXT://192.168.159.135:9092log.dirs=/data/kafka/kafka1.logs

Node 2:

properties
process.roles=broker,controllernode.id=2controller.quorum.voters=[email protected]:9093,[email protected]:9093,[email protected]:9093listeners=PLAINTEXT://192.168.159.136:9092,CONTROLLER://192.168.159.136:9093advertised.listeners=PLAINTEXT://192.168.159.136:9092log.dirs=/data/kafka/kafka2.logs

Node 3:

properties
process.roles=broker,controllernode.id=3controller.quorum.voters=[email protected]:9093,[email protected]:9093,[email protected]:9093listeners=PLAINTEXT://192.168.159.137:9092,CONTROLLER://192.168.159.137:9093advertised.listeners=PLAINTEXT://192.168.159.137:9092log.dirs=/data/kafka/kafka3.logs

process.roles=broker,controller makes each node both a broker and a controller — fine for a small cluster. For larger setups you'd run the controllers separately.

Generate a cluster ID and format storage

On any node, generate a UUID for the cluster:

bash
KAFKA_CLUSTER_ID="$(/opt/kafka/kafka_2.13-3.5.0/bin/kafka-storage.sh random-uuid)"echo $KAFKA_CLUSTER_ID# 8f2tB2hHSnSZkLkef2T2kQ

Export the same ID on the other two nodes:

bash
export KAFKA_CLUSTER_ID="8f2tB2hHSnSZkLkef2T2kQ"

Format storage on every node with the shared ID:

bash
sudo /opt/kafka/kafka_2.13-3.5.0/bin/kafka-storage.sh format \  -t $KAFKA_CLUSTER_ID \  -c /opt/kafka/kafka_2.13-3.5.0/config/kraft/server.properties

systemd service

Create /etc/systemd/system/kafka.service on every node:

ini
[Unit]Description=kafka ServiceAfter=network-online.targetRequires=network-online.target[Service]Type=simpleRestart=on-failureUser=kafkaGroup=kafkaExecStart=/opt/kafka/kafka_2.13-3.5.0/bin/kafka-server-start.sh /opt/kafka/kafka_2.13-3.5.0/config/kraft/server.propertiesExecStop=/opt/kafka/kafka_2.13-3.5.0/bin/kafka-server-stop.sh /opt/kafka/kafka_2.13-3.5.0/config/kraft/server.propertiesWorkingDirectory=/opt/kafka/kafka_2.13-3.5.0[Install]WantedBy=multi-user.target

Reload systemd and start Kafka:

bash
sudo systemctl daemon-reloadsudo systemctl start kafkasudo systemctl enable kafka

Once all three nodes are up, log lines like these mean the cluster has formed and you're done:

Check the status on every node:

bash
systemctl status kafka

Smoke-testing the cluster

A few useful commands. The --bootstrap-server flag accepts a comma-separated list of any cluster members.

Check the cluster version:

bash
sudo /opt/kafka/kafka_2.13-3.5.0/bin/kafka-topics.sh \  --bootstrap-server 192.168.159.135:9092,192.168.159.136:9092,192.168.159.137:9092 \  --version

Create a topic:

bash
sudo /opt/kafka/kafka_2.13-3.5.0/bin/kafka-topics.sh \  --bootstrap-server 192.168.159.135:9092,192.168.159.136:9092,192.168.159.137:9092 \  --create --topic test

Create a topic with custom partitions and replication factor:

bash
sudo /opt/kafka/kafka_2.13-3.5.0/bin/kafka-topics.sh \  --bootstrap-server 192.168.159.135:9092,192.168.159.136:9092,192.168.159.137:9092 \  --create --topic testv1 --partitions 10 --replication-factor 3

List topics:

bash
sudo /opt/kafka/kafka_2.13-3.5.0/bin/kafka-topics.sh \  --bootstrap-server 192.168.159.135:9092,192.168.159.136:9092,192.168.159.137:9092 \  --list

Delete a topic:

bash
sudo /opt/kafka/kafka_2.13-3.5.0/bin/kafka-topics.sh \  --bootstrap-server 192.168.159.135:9092,192.168.159.136:9092,192.168.159.137:9092 \  --delete --topic testv1

For more, see the Kafka operations docs and Confluent's topic operations guide.

Watch the file ownership and permissions of Kafka's data directories — running Kafka as one user but the data dir owned by another is a classic foot-gun.

References