Kafka tip – keeping the order of delivery

Kafka guarantees the order of delivery within a partition. That is, the order is not kept among partitions in a topic. There are some tips to keep the order of delivery in a topic. 1. One partition for a Topic This is what Kafka basically supports. <Pros> Simple solution <Cons> A partition is applied to …

Some useful commands to manage Kafka

This post shows some useful commands to manage Kafka. (based on version 2.12) Detailed information is on Kafka official documentation Notice that some commands use zookeeper ip/port and others use kafka ip/port Topic management commands List all topics ${KAFKA_HOME}/bin/kafka-topics.sh --list --zookeeper zookeeper_ip:port Create a topic ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper zookeeper_ip:port --replication-factor num --partitions num --topic topic_name …

At which point does Kafka consumer start to read

At which point does  Kafka consumer start to read? It depends on the following scenarios. Basic concept Kafka can have several topics and each topic can have several partitions Each partition keeps offset for each message Offset starts from 0 and increases by 1 when a new message is published Kafka server keeps the next …

Managing Kafka log

Kafka log is not an informative file but repository for incoming queue message. Some queue software deletes queue message when it is acked by consumer. But Kafka keeps log regardless of consumer's ack. But it is anyhow a file, so it has storage limitation. Now I'm showing how to manage Kafka log. (The following contents …

How Kafka consumer works?

Kafka clients are composed of producer and consumer. Understanding consumer is very important for overall architecture. A Topic is composed of several partitions (the number is defined when creating the Topic). And Kafka consumer is composed of group and client. (There could be multi groups and multi clients in a group) Each group and client …

How Kafka Topic failover works?

After Kafka cluster has been configured, we need to create a Topic which enables failover and data replication. The conclusion in advance is that if a Topic's replication factor is more than 2, Kafka supports automatic leader failover Data rebalance is supported only in manual operation Test environment Kafka 2.12 3 Kakfa brokers (Id : …

Building Kafka cluster

Building Kafka cluster is crucial for the production system. Kafka cluster gives the following advantages. Support for failover in case of a node down Queue replication Support for consumer scale out (standalone Kafka also supports consumer scale out) Step 1 - Build Zookeeper ensemble Kafka depends on Zookeeper for it's configuration management. Therefore, Zookeeper needs …

Kafka first step – running a simple server

Kafka is a very fast queue server. The main functions are as follows. Support for peer to peer, publish - subscribe messaging Built in cluster support Consistent performance as data grows This post shows the first step to run a simple server. (Refer to the full documention) The following steps are tested on Kafka 2.12 …