Kafka first step – running a simple server

Kafka is a very fast queue server.

The main functions are as follows.

• Support for peer to peer, publish – subscribe messaging
• Built in cluster support
• Consistent performance as data grows

This post shows the first step to run a simple server. (Refer to the full documention)

The following steps are tested on Kafka 2.12

1. Install Kafka

• Download from kafka.apache.org
• Unzip kafka_version.tgz
• For test, I copied kafka_2.12-2.1.0 directory into /home/tkstone/kafka_2.12-2.1.0 (I point it as KAKFA_HOME)

2. Start Zookeeper

Kafka uses Zookeeper as a metadata repository. (also as a Kafka cluster manager)

You can get Zookeeper from zookeeper.apache.org or use the one which comes with Kafka.

Run Zookeeper which is contained within Kafka as follows.

#{KAKFA_HOME}/bin/zookeeper-server-start.sh

3. Build a Kafka config

There are several config samples inside ${KAFKA_HOME}/conf. Copy server.properties into test.properties. And there are some properties to customize.

• broker.id : This value must be unique among cluster
• listeners : listen ip and port.
• log.dirs : The directories which saves log. Kafka logs are not an informative files, but persistent data files
• zookeeper.connect : zookeeper connect string

4. Start a standalone server

export JAVA_HOME=/usr/local/jdk1.8.0_151
KAFKA_HOME=/home/tkstone/kafka
${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/test.properties > ${KAFKA_HOME}/testserver/server.log 2>&1 &

5. Make a topic

${KAFKA_HOME}/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic topic_name

Now you are ready to test Kafka.

To check the topics which are generated, run the following command.

${KAFKA_HOME}/bin/kafka-topics.sh –list –zookeeper localhost:2181

After starting Kafka, you can verify that some zookeeper znodes are made.

[zk: localhost:2181(CONNECTED) 59] ls /
[log_dir_event_notification, isr_change_notification, zookeeper, admin, consumers, cluster, config, latest_producer_id_block, brokers, controller_epoch]

Also, you can verify generated topics.

[zk: localhost:2181(CONNECTED) 61] ls /config/topics
[test-topic2]