At which point does Kafka consumer start to read

At which point does  Kafka consumer start to read? It depends on the following scenarios. Basic concept Kafka can have several topics and each topic can have several partitions Each partition keeps offset for each message Offset starts from 0 and increases by 1 when a new message is published Kafka server keeps the next … Continue reading At which point does Kafka consumer start to read

Managing Kafka log

Kafka log is not an informative file but repository for incoming queue message. Some queue software deletes queue message when it is acked by consumer. But Kafka keeps log regardless of consumer's ack. But it is anyhow a file, so it has storage limitation. Now I'm showing how to manage Kafka log. (The following contents … Continue reading Managing Kafka log

Building a robust Zookeeper client : connection management

To support high available Zookeeper service, we need to configure Zookeeper ensemble. But this is not enough. we need to understand how client manages connection to Zookeeper server. For test, I set up a Zookeeper ensemble. (port : 2181, 2182, 2183) After then, write a test client with 2 important parameters - 1) connect string … Continue reading Building a robust Zookeeper client : connection management