At which point does Kafka consumer start to read

At which point does  Kafka consumer start to read? It depends on the following scenarios. Basic concept Kafka can have several topics and each topic can have several partitions Each partition keeps offset for each message Offset starts from 0 and increases by 1 when a new message is published Kafka server keeps the next …

Managing Kafka log

Kafka log is not an informative file but repository for incoming queue message. Some queue software deletes queue message when it is acked by consumer. But Kafka keeps log regardless of consumer's ack. But it is anyhow a file, so it has storage limitation. Now I'm showing how to manage Kafka log. (The following contents …

Building a robust Zookeeper client : connection management

To support high available Zookeeper service, we need to configure Zookeeper ensemble. But this is not enough. we need to understand how client manages connection to Zookeeper server. For test, I set up a Zookeeper ensemble. (port : 2181, 2182, 2183) After then, write a test client with 2 important parameters - 1) connect string …

Mockito thenReturn() vs thenAnswer()

Mockito is a very powerful test library for Java. In most cases, setting some mock action with thenReturn() is enough. But in some rare cases, we need to use thenAnswer(). thenReturn() example First, let's review the basic use of thenReturn() First, I want to test getResultWithParam(). Above test case succeeds. Wrong test case Next, I …

How Kafka consumer works?

Kafka clients are composed of producer and consumer. Understanding consumer is very important for overall architecture. A Topic is composed of several partitions (the number is defined when creating the Topic). And Kafka consumer is composed of group and client. (There could be multi groups and multi clients in a group) Each group and client …

How Kafka Topic failover works?

After Kafka cluster has been configured, we need to create a Topic which enables failover and data replication. The conclusion in advance is that if a Topic's replication factor is more than 2, Kafka supports automatic leader failover Data rebalance is supported only in manual operation Test environment Kafka 2.12 3 Kakfa brokers (Id : …

Building Kafka cluster

Building Kafka cluster is crucial for the production system. Kafka cluster gives the following advantages. Support for failover in case of a node down Queue replication Support for consumer scale out (standalone Kafka also supports consumer scale out) Step 1 - Build Zookeeper ensemble Kafka depends on Zookeeper for it's configuration management. Therefore, Zookeeper needs …

Kafka first step – running a simple server

Kafka is a very fast queue server. The main functions are as follows. Support for peer to peer, publish - subscribe messaging Built in cluster support Consistent performance as data grows This post shows the first step to run a simple server. (Refer to the full documention) The following steps are tested on Kafka 2.12 …

Redis replication gap tuning

Redis manages replication offset between master and slaves. If you type in "INFO REPLICATION" at redis-cli, the following output comes out. For the above result, offset number can vary depending on each environment. But slave0's offset should be equal or similar to master_repl_offset. The gap between master_repl_offset and slave's offset is the amount which is …

Programming Zookeeper – Watcher

The common pattern of Zookeeper is 1) to check existence of a znode or 2) to read a znode data. Let's suppose that you are building a program which does some action whenever a znode data is changed. The way to implement it is 1) to read the znode data periodically or 2) to get …