昨天学习了怎么进入维护模式,今天来研究一下手工故障转移,也就是常说的switchover和failover。
先来看看我们的集群
[postgres@133e0e204e206 ~]$ patronictl -c /etc/patroni.yml list
+ Cluster: patnori-test (6962171552537974697) --+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------+---------------+---------+---------+----+-----------+
| postgres1 | 133.0.204.206 | Replica | running | 16 | 0 |
| postgres2 | 133.0.204.207 | Replica | running | 16 | 0 |
| postgres3 | 133.0.204.208 | Leader | running | 16 | |
+-----------+---------------+---------+---------+----+-----------+
直接使用patronictl运行switchover
[postgres@133e0e204e206 ~]$ patronictl -c /etc/patroni.yml switchover
Master [postgres3]:
Candidate ['postgres1', 'postgres2'] []: postgres1
When should the switchover take place (e.g. 2021-06-08T16:32 ) [now]:
Current cluster topology
+ Cluster: patnori-test (6962171552537974697) --+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------+---------------+---------+---------+----+-----------+
| postgres1 | 133.0.204.206 | Replica | running | 16 | 0 |
| postgres2 | 133.0.204.207 | Replica | running | 16 | 0 |
| postgres3 | 133.0.204.208 | Leader | running | 16 | |
+-----------+---------------+---------+---------+----+-----------+
Are you sure you want to switchover cluster patnori-test, demoting current master postgres3? [y/N]: y
2021-06-08 15:32:41.86549 Successfully switched over to "postgres1"
+ Cluster: patnori-test (6962171552537974697) --+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------+---------------+---------+---------+----+-----------+
| postgres1 | 133.0.204.206 | Leader | running | 16 | |
| postgres2 | 133.0.204.207 | Replica | running | 16 | 0 |
| postgres3 | 133.0.204.208 | Replica | stopped | | unknown |
+-----------+---------------+---------+---------+----+-----------+
这里有几个需要输入的,先是要让你确认master,现在我们在postgres3上,然后让你选择候选人,也就是切换后的leader,我们选择postgres1。然后它会让你再确认一次,输入y之后切换正式开始。
[postgres@133e0e204e206 ~]$ patronictl -c /etc/patroni.yml list
+ Cluster: patnori-test (6962171552537974697) --+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------+---------------+---------+---------+----+-----------+
| postgres1 | 133.0.204.206 | Leader | running | 17 | |
| postgres2 | 133.0.204.207 | Replica | running | 17 | 0 |
| postgres3 | 133.0.204.208 | Replica | running | 17 | 0 |
+-----------+---------------+---------+---------+----+-----------+
注意看,切换完成TL,也就是时间线从16变成了17。
Jun 8 15:32:40 133e0e204e208 patroni: 2021-06-08 15:32:40,848 INFO: received switchover request with leader=postgres3 candidate=postgres1 scheduled_at=None
Jun 8 15:32:40 133e0e204e208 patroni: 2021-06-08 15:32:40,855 INFO: Got response from postgres1 http://133.0.204.206:8008/patroni: {"state": "running", "postmaster_start_time": "2021-06-07 16:44:00.382 CST
", "role": "replica", "server_version": 130002, "cluster_unlocked": false, "xlog": {"received_location": 3909288144, "replayed_location": 3909288144, "replayed_timestamp": null, "paused": false}, "timeline"
: 16, "database_system_identifier": "6962171552537974697", "patroni": {"version": "2.0.2", "scope": "patnori-test"}}
Jun 8 15:32:40 133e0e204e208 patroni: 2021-06-08 15:32:40,861 INFO: Lock owner: postgres3; I am postgres3
Jun 8 15:32:40 133e0e204e208 patroni: 2021-06-08 15:32:40,867 INFO: Got response from postgres1 http://133.0.204.206:8008/patroni: {"state": "running", "postmaster_start_time": "2021-06-07 16:44:00.382 CST
", "role": "replica", "server_version": 130002, "cluster_unlocked": false, "xlog": {"received_location": 3909288144, "replayed_location": 3909288144, "replayed_timestamp": null, "paused": false}, "timeline"
: 16, "database_system_identifier": "6962171552537974697", "patroni": {"version": "2.0.2", "scope": "patnori-test"}}
Jun 8 15:32:40 133e0e204e208 patroni: 2021-06-08 15:32:40,871 INFO: manual failover: demoting myself
Jun 8 15:32:41 133e0e204e208 systemd-logind: Removed session 2373.
Jun 8 15:32:41 133e0e204e208 patroni: 2021-06-08 15:32:41,201 INFO: Leader key released
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43,208 INFO: Local timeline=16 lsn=0/E9030180
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43,212 INFO: master_timeline=17
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43,216 INFO: master: history=13#0110/E9014430#011no recovery target specified
Jun 8 15:32:43 133e0e204e208 patroni: 14#0110/E9014B30#011no recovery target specified
Jun 8 15:32:43 133e0e204e208 patroni: 15#0110/E902D570#011no recovery target specified
Jun 8 15:32:43 133e0e204e208 patroni: 16#0110/E90301F8#011no recovery target specified
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43,217 INFO: closed patroni connection to the postgresql cluster
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43.406 CST [17816] LOG: redirecting log output to logging collector process
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43.406 CST [17816] HINT: Future log output will appear in directory "log".
Jun 8 15:32:43 133e0e204e208 patroni: 2021-06-08 15:32:43,406 INFO: postmaster pid=17816
Jun 8 15:32:43 133e0e204e208 patroni: localhost:5432 - rejecting connections
Jun 8 15:32:43 133e0e204e208 patroni: localhost:5432 - accepting connections
Jun 8 15:32:50 133e0e204e208 patroni: 2021-06-08 15:32:50,863 INFO: Lock owner: postgres1; I am postgres3
Jun 8 15:32:50 133e0e204e208 patroni: 2021-06-08 15:32:50,863 INFO: does not have lock
Jun 8 15:32:50 133e0e204e208 patroni: 2021-06-08 15:32:50,863 INFO: establishing a new patroni connection to the postgres cluster
Jun 8 15:32:50 133e0e204e208 patroni: 2021-06-08 15:32:50,918 INFO: no action. i am a secondary and i am following a leader
Jun 8 15:32:52 133e0e204e208 patroni: 2021-06-08 15:32:52,403 INFO: Lock owner: postgres1; I am postgres3
从日志上看,它接收到了switchover的请求,然后使用REST API访问要切换到候选人,这一步主要是要确认xlog的位置。确认完成后它就会将自己降级。并释放Leader key。而节点1会获取leader key,成为leader,并promoted自己。
当然还有一些选项,可以帮助我们直接用命令进行切换,而不是使用这种交互模式。
[postgres@133e0e204e206 ~]$ patronictl switchover --help
Usage: patronictl switchover [OPTIONS] [CLUSTER_NAME]
Switchover to a replica
Options:
--master TEXT The name of the current master
--candidate TEXT The name of the candidate
--scheduled TEXT Timestamp of a scheduled switchover in unambiguous format
(e.g. ISO 8601)
--force Do not ask for confirmation at any point
--help Show this message and exit.
scheduled 可以帮我们制定一个时间进行切换,比如你想2小时后切换。
force 避免反复的确认。
[postgres@133e0e204e206 ~]$ patronictl -c /etc/patroni.yml switchover --master postgres1 --candidate postgres2 --scheduled now --force
Current cluster topology
+ Cluster: patnori-test (6962171552537974697) --+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------+---------------+---------+---------+----+-----------+
| postgres1 | 133.0.204.206 | Leader | running | 17 | |
| postgres2 | 133.0.204.207 | Replica | running | 17 | 0 |
| postgres3 | 133.0.204.208 | Replica | running | 17 | 0 |
+-----------+---------------+---------+---------+----+-----------+
2021-06-08 16:21:31.79130 Successfully switched over to "postgres2"
+ Cluster: patnori-test (6962171552537974697) --+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------+---------------+---------+---------+----+-----------+
| postgres1 | 133.0.204.206 | Replica | stopped | | unknown |
| postgres2 | 133.0.204.207 | Leader | running | 17 | |
| postgres3 | 133.0.204.208 | Replica | running | 17 | 0 |
+-----------+---------------+---------+---------+----+-----------+
下面是我用一条命令实现从postgres1切换到postgres2。
和switchover一样,fileover有相同的参数,只是缺少了scheduled。
[postgres@133e0e204e206 ~]$ patronictl failover --help
Usage: patronictl failover [OPTIONS] [CLUSTER_NAME]
Failover to a replica
Options:
--master TEXT The name of the current master
--candidate TEXT The name of the candidate
--force Do not ask for confirmation at any point
--help Show this message and exit.
当然故障转移是自动的,但是也会有异常的情况,你可以手工进行failover。
最后有几个问题:
1. 一主两从的环境,所有的从库都落后于主库,此时leader节点出现问题,这种情况下该怎办?
这种情况下,没有任何副本有资格成为新的主库,因为它们都落后于主库,此时我们只能触发手动故障转移,也就是failover。patronictl failover <your-cluster-name>
,这样会丢数据,它将一个从库变成新的主库。
2.maximum_lag_on_failover参数的含义
这个参数的具体含义。我们的Patroni每隔10秒,leader节点会将它的wal_position写入到etcd。在leader竞争的情况下,每个副本都会将其wal_position与etcd中领导者最后注册的wal_position进行比较。如果差值大于maximum_lag_on_failover,它将不会提升。