Internal working of Sentinel

This post is based on Redis 4.0.6.

Main source of Sentinel is sentinel.c. When a Redis process is run in sentinel mode, sentinelTimer() is called. Inside it, all Sentinel’s logic flows.

The way to recognize the other Sentinel

  • At initial Sentinel config, there is no info for the other Sentinels. There are only options for Redis to monitor
  • When a Sentinel starts, it subscribe to Redis which is monitored (“SUBSCRIBE” command). The channel name is “__sentinel__:hello“. (The name can change as version changes) This subscription is used to receive other Sentinel’s hello message
  • The Sentinel publish hello message to the channel periodically (“PUBLISH” command). The following is a sample message
  • When the other Sentinels receive the message, they register new sentinel by parsing the message

The way to monitor Redis

  • Sentinel sends “INFO” command to Redis which are monitored (The ip and port is from “SENTINEL MONITOR ~”). By parsing “INFO” result, it can detect slaves
  • Afterwards, it sends “INFO” command to all masters and slaves periodically
  • It also sends “PING” command to all masters, slaves and sentinels. If the command fails, Sentinel detects that a process is down

The way to fail over Redis

  • When Sentinel detects  that a master is down, a new master is chosen from slaves and promotes it into a new master
  • It sends “SLAVEOF” to all the other slaves
  • Apply changed info to each process’s config file with “CONFIG REWITE” command

What happens when failed master restarts

  • When a master is down, a new master is chosen by Sentinel
  • But when the failed master starts again, it still recognizes itself as “master” (based on it’s config file)
  • Sentinel sends “INFO” command to the old master and compares role values (one is from Sentinel and the other from the old master)
  • Because two roles are different, Sentinel sends “SLAVEOF” to the old master. Now, the old master becomes a slave

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.