How Kafka consumer works?

Kafka clients are composed of producer and consumer. Understanding consumer is very important for overall architecture. A Topic is composed of several partitions (the number is defined when creating the Topic). And Kafka consumer is composed of group and client. (There could be multi groups and multi clients in a group) Each group and client …

Building Kafka cluster

Building Kafka cluster is crucial for the production system. Kafka cluster gives the following advantages. Support for failover in case of a node down Queue replication Support for consumer scale out (standalone Kafka also supports consumer scale out) Step 1 - Build Zookeeper ensemble Kafka depends on Zookeeper for it's configuration management. Therefore, Zookeeper needs …

Building Zookeeper cluster

Zookeeper is a cluster management solution. Therefore, Zookeeper needs to be set up as a cluster. If not, it can be the Single Point of Failure. Zookeeper cluster is called "ensemble". How to set up ensemble (based on v3.4.10) Basic config for a single Zookeeper is as follows. To change it into ensemble, add some …

Tips on customizing session manager

Tomcat internal cluster session manager can be used on a test system, but it has performance issue on a production system. (The problem of Tomcat session cluster) From my benchmark test, total TPS saturated from 3 instances. (It depends on H/W spec and business logic) Therefore I needed to make a custom session manager which …

About Tomcat BackupManager

Tomcat BackupManager is another cluster session manager. It is also sibling class with DeltaManager. Some important features of BackupManager It is activated by declaring <Manager className="org.apache.catalina.ha.session.BackupManager"> inside <Cluster> tag When an instance create a sesion, that instance becomes the session's primary node. Another node becomes backup node and all the other node become proxy node …

The problem of Tomcat session cluster

Some people said that Tomcat session cluster is not fit for high traffic web services. So I did performance test. (both DeltaManager and BackupManager) An ideal cluster is whose total TPS shows linear graph. But Tomcat session cluster showed log graph. (it's coefficient is dependent on usage pattern) I analyzed why it happened. The following …

About Tomcat session cluster

Servlet session on Tomcat is managed by "Tomcat manager". There are 3 kinds of session manager. Default manager (StandardManager) Delta manager Backup manager Default manager (StandardManager) is non clustered session manager. That is, sessions are not shared between Tomcat instances. Without specific setting, default manager is used when we use servlet session api. Delta manager …