This post is based on Redis 4.0.6.
Main source of Sentinel is sentinel.c. When a Redis process is run in sentinel mode, sentinelTimer() is called. Inside it, all Sentinel’s logic flows.
The way to recognize the other Sentinel
- At initial Sentinel config, there is no info for the other Sentinels. There are only options for Redis to monitor
- When a Sentinel starts, it subscribe to Redis which is monitored (“SUBSCRIBE” command). The channel name is “__sentinel__:hello“. (The name can change as version changes) This subscription is used to receive other Sentinel’s hello message
- The Sentinel publish hello message to the channel periodically (“PUBLISH” command). The following is a sample message
“127.0.0.1,5000,59eadb47a4acc86546820bbf7c6879e80f6ae3a5,6,M1,127.0.0.1,6479,1” - When the other Sentinels receive the message, they register new sentinel by parsing the message
The way to monitor Redis
- Sentinel sends “INFO” command to Redis which are monitored (The ip and port is from “SENTINEL MONITOR ~”). By parsing “INFO” result, it can detect slaves
- Afterwards, it sends “INFO” command to all masters and slaves periodically
- It also sends “PING” command to all masters, slaves and sentinels. If the command fails, Sentinel detects that a process is down
The way to fail over Redis
- When Sentinel detects that a master is down, a new master is chosen from slaves and promotes it into a new master
- It sends “SLAVEOF” to all the other slaves
- Apply changed info to each process’s config file with “CONFIG REWITE” command
What happens when failed master restarts
- When a master is down, a new master is chosen by Sentinel
- But when the failed master starts again, it still recognizes itself as “master” (based on it’s config file)
- Sentinel sends “INFO” command to the old master and compares role values (one is from Sentinel and the other from the old master)
- Because two roles are different, Sentinel sends “SLAVEOF” to the old master. Now, the old master becomes a slave
TK's blog 