At which point does Kafka consumer start to read? It depends on the following scenarios. Basic concept Kafka can have several topics and each topic can have several partitions Each partition keeps offset for each message Offset starts from 0 and increases by 1 when a new message is published Kafka server keeps the next … Continue reading At which point does Kafka consumer start to read
Managing Kafka log
Kafka log is not an informative file but repository for incoming queue message. Some queue software deletes queue message when it is acked by consumer. But Kafka keeps log regardless of consumer's ack. But it is anyhow a file, so it has storage limitation. Now I'm showing how to manage Kafka log. (The following contents … Continue reading Managing Kafka log
Building a robust Zookeeper client : connection management
To support high available Zookeeper service, we need to configure Zookeeper ensemble. But this is not enough. we need to understand how client manages connection to Zookeeper server. For test, I set up a Zookeeper ensemble. (port : 2181, 2182, 2183) After then, write a test client with 2 important parameters - 1) connect string … Continue reading Building a robust Zookeeper client : connection management
Mockito thenReturn() vs thenAnswer()
Mockito is a very powerful test library for Java. In most cases, setting some mock action with thenReturn() is enough. But in some rare cases, we need to use thenAnswer(). thenReturn() example First, let's review the basic use of thenReturn() First, I want to test getResultWithParam(). Above test case succeeds. Wrong test case Next, I … Continue reading Mockito thenReturn() vs thenAnswer()
How Kafka consumer works?
Kafka clients are composed of producer and consumer. Understanding consumer is very important for overall architecture. A Topic is composed of several partitions (the number is defined when creating the Topic). And Kafka consumer is composed of group and client. (There could be multi groups and multi clients in a group) Each group and client … Continue reading How Kafka consumer works?
How Kafka Topic failover works?
After Kafka cluster has been configured, we need to create a Topic which enables failover and data replication. The conclusion in advance is that if a Topic's replication factor is more than 2, Kafka supports automatic leader failover Data rebalance is supported only in manual operation Test environment Kafka 2.12 3 Kakfa brokers (Id : … Continue reading How Kafka Topic failover works?
Building Kafka cluster
Building Kafka cluster is crucial for the production system. Kafka cluster gives the following advantages. Support for failover in case of a node down Queue replication Support for consumer scale out (standalone Kafka also supports consumer scale out) Step 1 - Build Zookeeper ensemble Kafka depends on Zookeeper for it's configuration management. Therefore, Zookeeper needs … Continue reading Building Kafka cluster
Kafka first step – running a simple server
Kafka is a very fast queue server. The main functions are as follows. Support for peer to peer, publish - subscribe messaging Built in cluster support Consistent performance as data grows This post shows the first step to run a simple server. (Refer to the full documention) The following steps are tested on Kafka 2.12 … Continue reading Kafka first step – running a simple server
Redis replication gap tuning
Redis manages replication offset between master and slaves. If you type in "INFO REPLICATION" at redis-cli, the following output comes out. For the above result, offset number can vary depending on each environment. But slave0's offset should be equal or similar to master_repl_offset. The gap between master_repl_offset and slave's offset is the amount which is … Continue reading Redis replication gap tuning
Programming Zookeeper – Watcher
The common pattern of Zookeeper is 1) to check existence of a znode or 2) to read a znode data. Let's suppose that you are building a program which does some action whenever a znode data is changed. The way to implement it is 1) to read the znode data periodically or 2) to get … Continue reading Programming Zookeeper – Watcher