OceanBase 集群故障测试-模拟节点故障
【摘要】 1，系统信息sys租户副本信息：为2分布在zone1,zone.sys租户副本数量为2：MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;*************************** 1. row *************************** gmt_crea...
1，系统信息
sys租户副本信息：为2分布在zone1,zone.
sys租户副本数量为2：
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
*************************** 1. row ***************************
gmt_create: 2021-12-07 16:42:12.523525
gmt_modified: 2021-12-07 16:42:12.523525
tenant_id: 1
tenant_name: sys
replica_num: -1
zone_list: zone1;zone2
primary_zone: zone1;zone2
locked: 0
collation_type: 0
info: system tenant
read_only: 0
rewrite_merge_version: 0
locality: FULL{1}@zone1, FULL{1}@zone2
logonly_replica_num: 0
previous_locality:
storage_format_version: 0
storage_format_work_version: 0
default_tablegroup_id: -1
compatibility_mode: 0
drop_tenant_time: -1
status: TENANT_STATUS_NORMAL
in_recyclebin: 0
1 row in set (0.001 sec)
ERROR: No query specified
2，observer信息
集群信息（1-1-1）
zone1 192.168.5.200 2881 2882
zone1 192.168.5.200 3881 3882
zone1 192.168.5.200 4881 4882
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-07 16:17:55.557006 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863788395674 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.002 sec)
3，故障模拟：2个节点kill
对外不能提供服务，只有zone1,zone2正常才能正常提供服务。
3.1 kill zone2,zone2
[admin@db01 ~]$ ps -ef|grep observer |grep -v grep
admin 727 1 99 02:52 ? 20:43:00 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO
admin 1559 1 99 02:55 ? 19:04:39 /home/admin/oceanbase/bin/observer -r 192.168.5.200:3882:3881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
admin 2141 1 80 02:55 ? 13:37:51 /home/admin/oceanbase/bin/observer -r 192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/observer03/store -i ens192 -l INFO
[admin@db01 ~]$ kill -9 1559 2141
[admin@db01 ~]$ ps -ef|grep observer |grep -v grep
admin 727 1 99 02:52 ? 20:43:22 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO
[admin@db01 ~]$
3.2，登陆数据库，发现系统已经不能正常提供服务
3.3，启动zone2
[admin@db01 ~]$ cd /data/observer02/ && /home/admin/oceanbase/bin/observer -r "192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
/home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
rs list: 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881
optstr: __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10
zone: zone2
mysql port: 3881
rpc port: 3882
cluster id: 1
data_dir: /data/observer02/store
devname: ens192
log level: INFO
[admin@db01 observer02]$
[admin@db01 observer02]$ ps -ef|grep obs|grep -v grep
admin 727 1 99 02:52 ? 20:47:35 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO
admin 11948 1 99 19:56 ? 00:00:16 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
3.4,查询系统表，发现正常提供服务
[admin@db01 ~]$ sh c.sh
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 3221567989
Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [oceanbase]> select * from __all_server;
ERROR 4012 (HY000): Timeout
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 08:52:52.289833 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
3 rows in set (3.090 sec)
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
3 rows in set (0.001 sec)
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.000 sec)
MySQL [oceanbase]>[admin@db01 ~]$ sh c.sh
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 3221567989
Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [oceanbase]> select * from __all_server;
ERROR 4012 (HY000): Timeout
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 08:52:52.289833 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
3 rows in set (3.090 sec)
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+
3 rows in set (0.001 sec)
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.000 sec)
MySQL [oceanbase]>
#状态截图：
4，故障模拟：kill zone2
对外不能提供服务，只有zone1,zone2正常才能正常提供服务
4.1检查租户副本信息
sys租户副本分布在：locality: FULL{1}@zone1, FULL{1}@zone2
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
*************************** 1. row ***************************
gmt_create: 2021-12-07 16:42:12.523525
gmt_modified: 2021-12-07 16:42:12.523525
tenant_id: 1
tenant_name: sys
replica_num: -1
zone_list: zone1;zone2
primary_zone: zone1;zone2
locked: 0
collation_type: 0
info: system tenant
read_only: 0
rewrite_merge_version: 0
locality: FULL{1}@zone1, FULL{1}@zone2
logonly_replica_num: 0
previous_locality:
storage_format_version: 0
storage_format_work_version: 0
default_tablegroup_id: -1
compatibility_mode: 0
drop_tenant_time: -1
status: TENANT_STATUS_NORMAL
in_recyclebin: 0
1 row in set (0.001 sec)
ERROR: No query specified
4.2，模拟zone2故障
4.3，故障现在，不能正常对外提供服务
4.4 启动zone2
[admin@db01 observer03]$
[admin@db01 observer03]$ cd /data/observer02/ && /home/admin/oceanbase/bin/observer -r "192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
/home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
rs list: 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881
optstr: __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10
zone: zone2
mysql port: 3881
rpc port: 3882
cluster id: 1
data_dir: /data/observer02/store
devname: ens192
log level: INFO
[admin@db01 observer02]$ ps -ef|grep 3882
admin 15281 1 99 20:25 ? 00:07:57 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO
admin 16343 14059 0 20:32 pts/3 00:00:00 grep --color=auto 3882
4.4查看服务状态
zone2 的observer2启动之后，可以正常查询。
MySQL [oceanbase]> select * from __all_server;
ERROR 4012 (HY000): Timeout
MySQL [oceanbase]> select * from __all_server;
ERROR 4012 (25000): Statement is timeout
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
ERROR 4012 (25000): Statement is timeout
ERROR: No query specified
MySQL [oceanbase]> select * from __all_server;
ERROR 4012 (HY000): Timeout
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:07.207814 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 09:26:07.208028 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926371893289 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.001 sec)
5，故障模拟：kill zone3
对外正常提供服务。
5.1检查server状态
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:15.207936 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926774220189 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 09:26:07.208028 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926371893289 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.001 sec)
5.2 模拟zone3 故障
[admin@db01 observer02]$
[admin@db01 observer02]$ ps -ef|grep '\-P 4882'|grep -v grep
admin 14080 1 90 20:17 ? 00:19:29 /home/admin/oceanbase/bin/observer -r 192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/observer03/store -i ens192 -l INFO
[admin@db01 observer02]$ kill -9 14080
[admin@db01 observer02]$ ps -ef|grep '\-P 4882'|grep -v grep
5.3 查询系统表
数据库状态正常，对外正常提供服务
MySQL [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 |
| 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:15.207936 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926774220189 | 0 | 1 | 0 |
| 2021-12-07 15:56:11.908905 | 2021-12-08 09:39:00.647693 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 |
+----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.001 sec)
【版权声明】本文为华为云社区用户原创内容，转载时必须标注文章的来源（华为云社区）、文章链接、文章作者等基本信息， 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱：
cloudbbs@huaweicloud.com
- 点赞
- 收藏
- 关注作者
评论（0）