keepAlived

举报
小源博客 发表于 2023/04/11 11:41:01 2023/04/11
【摘要】 1 系统可用性A = MTBF / (MTBF + MTTR)99.95%: (60*24*30)*(1-0.9995)=21.6分钟 # 一般按一个月停机时间统计指标：99.9%,99.99%,99.999%,99.9999%2 实现高可用解决方案：建立冗余机制active/passive 主/备active/active 双主3 VRRP全名：Virtual Router Re...

1 系统可用性

  • A = MTBF / (MTBF + MTTR)

99.95%: (60*24*30)*(1-0.9995)=21.6分钟      # 一般按一个月停机时间统计

  • 指标：99.9%,99.99%,99.999%,99.9999%


2 实现高可用

  • 解决方案：建立冗余机制

  • active/passive 主/备

  • active/active 双主


3 VRRP

  • 全名：Virtual Router Redundancy Protocol

  • 虚拟路由器冗余协议，解决静态网关单点风险

  • 物理层：路由器、三层交换机

  • 软件层：keepalived


4 keepalived介绍

官网与下载地址


5 keepalived环境准备

  • 各节点时间必须同步：ntp、chrony

  • 关闭防火墙及SELinux

实验准备

  • 准备五台机器

# 两台keepalived端
k1.xier.org
k2.xier.org
​
# 一台客户端
client.xier.org
​
# 两台web服务器
web1.xier.org
web2.xier.org


keepalived安装与使用

  • keepalived所有节点(yum安装)

  • 底层是vrrp协议：提供一个虚拟IP
yum install -y keepalived
​
vim /etc/hosts
10.0.0.100 www.xier.org    # 10.0.0.100 作为vip
​
cp /etc/keepalived/keepalived.conf{,.bak}
​
vim /etc/keepalived/keepalived.conf
第一板块为全局配置
第二板块为vrrp配置
第三模块为keepalived与lvs结合的配置
​
# 第一板块
global_defs {
  notification_email {
          # keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区分写多个
    2443599212@qq.com
  }
  notification_email_from ws3024321237@163.com     # 发邮件的地址
  smtp_server 127.0.0.1     # 邮件服务器地址
  smtp_connect_timeout 30   # 邮件服务器连接超时时间
  router_id ka1.xier.org      # 每个keepalived主机唯一标识，建议使用当前主机名，但多节点重名不影响
  vrrp_skip_check_adv_addr  # 不写默认是全检查，对所有通告报文都检查，会比较消化性能。启用此配置后，如果收到的通告报文和上一个报文是同一个路由器，则跳过检查
  vrrp_strict               # 严格遵守vrrp协议，启动此项后以下状况将无法启动服务：1.无VIP地址，2.配置看单播邻居，3.在vrrp版本2中有IPv6地址，开启此项并且没有配置vrrp_iptables时会开启iptables防火墙规则，默认导致VIP无法访问，不建议加此项配置（生产建议不要加）
  vrrp_garp_interval 0      # gratuitous ARP messages 报文发送延迟，0表示不延迟
  vrrp_gna_interval 0       # unsolicited NA messages (不请自来)消息发送延迟
  vrrp_mcast_group4 224.0.0.18   # 指定组播IP地址范围：224.0.0.0到239.255.255.255，默认值：224.0.0.18。如果有多个keepalived集群，则使用不同的组播范围，避免出现互相干扰
  vrrp_iptables         # 此项和vrrp_strict同时开启时，则不会添加防火墙规则(iptables -vnL查看)，如果无配置vrrp_strict项，则无需启用此配置
}
​
# 第二板块
vrrp_instance web {
   state MASTER|BACKUP        # 当前节点在此虚拟机路由器上的初始状态，默认为MASTER或者BACKUP
   interface eth0             # 绑定为当前虚拟路由器使用的物理接口，如：eth0,bond0,br0,可以和VIP不在一个网卡
   virtual_router_id 51       # 每个虚拟路由器唯一标识，范围：0-255，每个虚拟路由器此值必须唯一，否则服务无法启动，同属一个虚拟路由器的多个keepalived节点必须相同，务必确认在同一网络中此值必须唯一
   priority 100               # 当前物理节点在此虚拟路由器的优先级，范围：1-254，每个keepalived主机节点此值不同
   advert_int 1               # vrrp通告的时间间隔，检查优先级，1秒检查一次，默认1s
   authentication {           # 认证机制
       auth_type PASS         # AH为IPSEC认证(不推荐)，PASS为简单密码(建议使用)
       auth_pass 1111         # 预共享密钥，仅前8为有效，同一个虚拟路由器的多个keepalived节点必须一样
   }
   virtual_ipaddress {        # 虚拟IP，生产环境可能指定上百个IP地址
       192.168.200.16         # 指定VIP，不指定网卡，默认为eth0，注意：不知道/prefix,默认为/32
       192.168.200.17/24 dev eth1         # 指定VIP网卡，建议和interface指令指定的网卡不在同一个网卡 
       192.168.200.18/24 dev eth2 label eth2:1   # 指定VIP的网卡label
   }
}
​
# 第三板块
# virtual_server 192.168.200.100 443   略
....

  • 注意：centos7上有bug，可能有下面情况出现

systemctl restart keepalived     # 新配置可能无法生效
systemctl stop keepalived; systemctl start keepalived

  • client查看vip通告

tcpdump -i eth0 -nn host 224.0.0.18

keepalived编译安装

=================================================================所有keepalived节点=========================================================================
yum install gcc curl openssl_devel libnl3-devel net-snmp-devel
​
wget https://keepalived.org/software/keepalived-2.2.2.tar.gz
​
tar xvf keepalived-2.2.2.tar.gz -C /usr/local/src
​
cd /usr/local/src/keepalived-2.2.2
​
# 选项 --disable-fwmark 可用于禁用iptables规则，可防止VIP无法访问，无此选项默认会启用 iptables 规则
./configure --prefix=/apps/keepalived
​
make && make install
​
/apps/keepalived/sbin/keepalived -v
​
mkdir /etc/keepalived
​
cp /apps/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
​
=================================================================ka1=========================================================================
vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {

  2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
​
vrrp_instance web {
 state MASTER
 interface eth0
 virtual_router_id 66
 priority 100
 advert_int 1
 authentication {           # 认证机制
     auth_type PASS         # AH为IPSEC认证(不推荐)，PASS为简单密码(建议使用)
     auth_pass 1111         # 预共享密钥，仅前8为有效，同一个虚拟路由器的多个keepalived节点必须一样
 }
 virtual_ipaddress {
     10.0.0.18/24 dev eth0 label eth0:1
 }
}
​
systemctl start keepalived
​
scp -r /etc/keepalived  10.0.0.18:/etc/
​
​
=================================================================ka2=========================================================================
vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
 
  2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
​
vrrp_instance web {
 state BACKUP
 interface eth0
 virtual_router_id 66
 priority 70
 advert_int 1
 authentication {           # 认证机制
     auth_type PASS         # AH为IPSEC认证(不推荐)，PASS为简单密码(建议使用)
     auth_pass 1111         # 预共享密钥，仅前8为有效，同一个虚拟路由器的多个keepalived节点必须一样
 }
 virtual_ipaddress {
     10.0.0.18/24 dev eth0 label eth0:1
 }
}





抢占模式和非抢占模式

  • 默认为抢占模式，即当高优先级的主机恢复在线后，会抢占低优先级的master角色，造成网络波动，建议设置为非抢占模式nopreempt，即高优先级主机恢复后，并不会抢占优先级主机的master角色

  • 此外原主机宕机迁移至新主机后续也发生宕机时，会将VIP迁移回原主机，因此生产建议不要使用非抢占模式


非抢占模式

  • 注意：要关闭VIP抢占，必须将各keepalived服务器state配置为BACKUP

  • 效果就是当主VIP宕机时VIP飘移过后，重启主VIP也不会将VIP夺回来

  • 注意：需要各keepalived服务器state为BACKUP

# k1主机配置
vrrp_instance web {
   state BACKUP           # 都为BACKUP
   interface eth0
   virtual_router_id 51
   priority 100           # 优先级高
   advert_int 1
   nopreempt              # 添加此行，都为nopreempt
   authentication {
       auth_type PASS
       auth_pass 1111
   }
   virtual_ipaddress {
       10.0.0.100/24 dev eth0 label eth0:1
   }
}
​
​
# k2主机配置
vrrp_instance web {
   state BACKUP
   interface eth0
   virtual_router_id 51
   priority 80
   advert_int 1
   nopreempt              # 添加此行，都为nopreempt
   authentication {
       auth_type PASS
       auth_pass 1111
   }
   virtual_ipaddress {
       10.0.0.100/24 dev eth0 label eth0:1
   }
}


抢占式延迟模式

  • 需要先关闭vrrp严格模式：不要启用vrrp_strict

  • 抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回VIP
preempt_delay         # 指定抢占延迟时间为#s,默认延迟300s

  • 注意：需要各keepalived服务器state为BACKUP

  • 范例
# k1主机配置
vrrp_instance web {
  state BACKUP           # 都为BACKUP
  interface eth0
  virtual_router_id 51
  priority 100           # 优先级高
  advert_int 1
  preempt_delay 60      # 抢占延迟模式，默认延迟300s
  authentication {
      auth_type PASS
      auth_pass 1111
  }
  virtual_ipaddress {
      10.0.0.100/24 dev eth0 label eth0:1
  }
}
​
​
# k2主机配置
vrrp_instance web {
  state BACKUP
  interface eth0
  virtual_router_id 51
  priority 80
  advert_int 1
  preempt_delay 60    # 抢占延迟模式，默认延迟300s
  authentication {
      auth_type PASS
      auth_pass 1111
  }
  virtual_ipaddress {
      10.0.0.100/24 dev eth0 label eth0:1
  }
}




VIP单播配置

  • 默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量

  • 作用：可以减少一定的广播流量，因为不会跨交换机而导致广播到处蔓延

  • 注意：启动单播，不能启用 vrrp_strict,且vrrp_mcast_group4多播设置会失效

# 在所有节点vrrp_instance语句中设置对方主机的IP，建议设置为专用于对应心跳线网络的地址，而非使用业务网络
unicast_src_ip <IPADDR>    # 指定发送单播的源IP
unicast_peer {
   <IPADDR>               # 指定接受单播的对方目标主机IP
   .......
}

  • 范例(注意在第二模板位置)

# k1主机配置
global_defs {
 notification_email {
  
   2443599212@qq.com
 }
 notification_email_from 
 smtp_server 127.0.0.1
 smtp_connect_timeout 30
 router_id ka1.xier.org
 vrrp_skip_check_adv_addr
 vrrp_garp_interval 0
 vrrp_gna_interval 0
}
​
vrrp_instance web {
  state BACKUP
  interface eth0
  virtual_router_id 66
  priority 100
  advert_int 1
  unicast_src_ip 10.0.0.8    # 指定发送单播的源IP
  unicast_peer {
   10.0.0.18               # 指定接受单播的对方目标主机IP
  }
  virtual_ipaddress {
      10.0.0.18/24 dev eth0 label eth0:1
  }
}
​
​
​
# k2主机配置
global_defs {
 notification_email {
 
   2443599212@qq.com
 }
 notification_email_from 
 smtp_server 127.0.0.1
 smtp_connect_timeout 30
 router_id ka2.xier.org
 vrrp_skip_check_adv_addr
 vrrp_garp_interval 0
 vrrp_gna_interval 0
}
​
vrrp_instance web {
  state BACKUP
  interface eth0
  virtual_router_id 66
  priority 80
  advert_int 1
  unicast_src_ip 10.0.0.18    # 指定发送单播的源IP
  unicast_peer {
   10.0.0.8               # 指定接受单播的对方目标主机IP
  }
  virtual_ipaddress {
      10.0.0.18/24 dev eth0 label eth0:1
  }
}

  • 验证

tcpdump -i eth0 -nn host 10.0.0.8 and 10.0.0.18




启用keepalived日志功能

  • 打开配置文件(总共有0-7个日志等级)
cp /apps/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
​
vim /apps/keepalived/etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 6"
​
# 编辑日志配置文件
vim /etc/rsyslog.conf
local6.*    /var/log/keepalived.log
​
systemctl restart keepalived rsyslog
​
tail -f /var/log/keepalived.log




keepalived双主双从配置

  • 配置文件

  • k1主机配置

vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {

  2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
​
vrrp_instance web1 {
 state MASTER
 interface eth0
 virtual_router_id 66
 priority 100
 advert_int 1
 virtual_ipaddress {
     10.0.0.100...150/24 dev eth0 label eth0:0..150
 }
}
​
vrrp_instance web2 {
 state BACKUP
 interface eth0
 virtual_router_id 88
 priority 80
 advert_int 1
 virtual_ipaddress {
     10.0.0.151...200/24 dev eth0 label eth0:151...200
 }
}

  • k2主机配置

vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
  
  2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
​
vrrp_instance web1 {
 state BACKUP
 interface eth0
 virtual_router_id 66
 priority 80
 advert_int 1
 virtual_ipaddress {
     10.0.0.100...150/24 dev eth0 label eth0:0..150
 }
}
​
vrrp_instance web2 {
 state MASTER
 interface eth0
 virtual_router_id 88
 priority 100
 advert_int 1
 virtual_ipaddress {
     10.0.0.151...200/24 dev eth0 label eth0:151...200
 }
}

  • 重启服务查看状态（所有节点）

systemct restart keepalived
​
hostname -I | tr ' ' '\n'




子配置文件配置

  • 配置如下

  • k1
vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {

  2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.1.1.1
}
​
include /etc/keepalived/conf.d/*.conf
​
​
mkdir /etc/keepalived/conf.d/
​
vim /etc/keepalived/conf.d/web1.conf
vrrp_instance web1 {
 state MASTER
 interface eth0
 virtual_router_id 66
 priority 100
 advert_int 1
 virtual_ipaddress {
     10.0.0.100...150/24 dev eth0 label eth0:0..150
 }
}
​
vim /etc/keepalived/conf.d/web2.conf
vrrp_instance web2 {
 state BACKUP
 interface eth0
 virtual_router_id 88
 priority 80
 advert_int 1
 virtual_ipaddress {
     10.0.0.151...200/24 dev eth0 label eth0:151...200
 }
}
  • k2
vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
  
  2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.1.1.1
}
​
include /etc/keepalived/conf.d/*.conf
​
​
mkdir /etc/keepalived/conf.d/
​
vim /etc/keepalived/conf.d/web1.conf
vrrp_instance web1 {
 state BACKUP
 interface eth0
 virtual_router_id 66
 priority 80
 advert_int 1
 virtual_ipaddress {
     10.0.0.100...150/24 dev eth0 label eth0:0..150
 }
}
​
vim /etc/keepalived/conf.d/web2.conf
vrrp_instance web2 {
 state MASTER
 interface eth0
 virtual_router_id 88
 priority 100
 advert_int 1
 virtual_ipaddress {
     10.0.0.151...200/24 dev eth0 label eth0:151...200
 }
}
  • 重启服务再查看是否生效
systemctl restart keepalived
​
hostname -I | tr ' ' '\n'




keepalived通知脚本配置

  • 当keepalived的状态变化时，可以自动触发脚本的执行，比如：发邮件通知用户
  • 默认以用户keepalived_script身份执行脚本，如果此用户不存在，以root执行脚本
  • 可以用下面指令指定脚本执行用户的身份
global_defs {
   ''''''
   srcipt_user <user>
   ''''''
}

通知脚本类型

  • 当前节点成为主节点时触发的脚本

notify_master <STRING>|<QUOTED-STRING>

  • 当前节点转为备节点时触发的脚本

notify_backup <STRING>|<QUOTED-STRING>

  • 当前节点转为 "失败(keepalived故障)" 状态时触发的脚本

notify_fault <STRING>|<QUOTED-STRING>

  • 通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知

notify <STRING>|<QUOTED-STRING>

  • 当停止VRRP时触发的脚本

notify_stop <STRING>|<QUOTED-STRING>


脚本调用方法(邮件报警)

  • 在vrrp_instance VI_1 语句块的末尾加下面行
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"

  • 范例

  • k1和k2

vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
 
 2443599212@qq.com
}
notification_email_from 
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.xier.org
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.1.1.1
}
​
include /etc/keepalived/conf.d/*.conf
​
​
mkdir /etc/keepalived/conf.d/
​
vim /etc/keepalived/conf.d/web1.conf
vrrp_instance web1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
virtual_ipaddress {
    10.0.0.100/24 dev eth0 label eth0:100
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
​
​
vim /etc/keepalived/notify.sh
#!/bin/bash
contact='2443599212@qq.com'
notify() {
  mailsubject="$(hostname) to be $1, vip floating"    # 发送标题
  mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"   # 发送内容
  echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
  notify master
  ;;
backup)
  notify backup
  ;;
fault)
  notify fault
  ;;
*)
  echo "Usage: $(basename $0) {master|backup|fault}"
  exit 1
  ;;
esac
​
​
chmod +x /etc/keepalived/notify.sh
​
​
# 配置邮箱
dnf install -y mailx
​
发件人：
SMTP服务器(QQ邮箱中-左上角设置-账户-开启SMTP服务器,smtp.qq.com ---> SMTP服务器端口:465 ---> SMTP HELO:qq.com ---> SMTP电邮:360120854@qq.com ---> 安全链接:SSL/TLS ---> 勾选：SSL验证对端、SSL验证主机 ---> 认证:用户名和密码 ---> 用户名称:360120854@qq.com --->授权密码:ljroytmuhlkjbgje
​
vim /etc/mail.rc
''''
set from=360120854@qq.com
set smtp=smtp.qq.com
set smtp-auth-user=360120854@qq.com
set smtp-auth-password=ljroytmuhlkjbgje
​
systemctl restart keepalived

  • 测试邮件是否能发送

/etc/keepalived/notify.sh master




keepalived+lvs高可用

  • k1和k2安装ipvs工具

yum install -y ipvsadm

  • IPVS相关配置

  • 虚拟服务器配置结果

virtual_server IP port {
 '''
 real_server {
     '''
 }
 real_server {
     '''
 }
 '''
}

  • virtual server (虚拟服务器) 的定义格式

virtual_server IP port      # 定义虚拟主机IP地址及其端口
virtual_server fwmark int   # ipvs的防火墙打标，实现基于防火墙的负载均衡集群
virtual_server group string   # 使用虚拟服务器组

  • 虚拟服务器组

  • 将多个虚拟服务器定义成一个组，统一对外服务，如：http和https定义成一个虚拟服务器组
  • 参考文档：
/apps/keepalived/etc/keepalived/samples/

  • 虚拟服务器配置

  • rs是后端服务器、vs是lvs服务器
virtual_server IP port {    # VIP和PORT
 delay_loop <INT>       # 检查后端服务器的时间间隔
 lb_algo rr|wrr|lc|wlc|lblc|sh|dh   # 定义调度算法
 lb_king NAT|DR|TUN     # 集群的类型，注意要大写
 persistence_timeout <INT>  # 持续连接时长
 protocol TCP|UDP|SCTP      # 指定服务协议，一般为TCP
 sorry_server <IPADDR> <PORT>   # 所有RS故障时，备用服务器地址。
 real_server <IPADDR> <PORT> {  # RS的IP和PORT
     weight <INT>       # RS权重
     notify_up <STRING>|<QUOTED-STRING> # RS上线通知脚本
     HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC|CHECK {...}    # 定义当前主机健康状态检测方法
 }
}
​
# 注意：括号必须分行写，两个括号写在行，如：}} 会出错

应用层监测

  • 应用层检测：HTTP_GET|SSL_GET`
HTTP_GET|SSL_GET {
  url {
      path <URL_PATH>       # 定义要监控的URL
      status_code <INT>    # 判断上述检测机制为健康状态的响应码，一般为200
  }
  connect_timeout <INTEGER> # 客户端请求的超时时长，相当于haproxy的timeout server
  nb_get_retry <INT>        # 重试次数
  delay_before_retry <INT>      # 重试之前的延迟时长
  connect_ip <IP ADDRESS>    # 向当前RS哪个IP地址发起健康状态检测请求
  connect_port <PORT>         # 向当前RS的那个port发起健康状态检测请求
  bindto <IP ADDRESS>        # 向当前RS发出健康状态检测请求时使用的源地址
  bind_port <PORT>           # 向当前RS发出健康状态检测请求时使用的源端口
}

  • 范例

vim /etc/keepalived/conf.d/lvs_web.conf
virtual_server 10.0.0.100 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 10.0.0.7 80 {
        weight 1
        HTTP_GET {
            url {
                path /monitor.html
                status_code 200
​
            }
        connect_timeout 1
        nb_get_retry 3
        delay_before_retry 1
        }
    }
    real_server 10.0.0.17 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}


传输层(TCP)监测

  • 基于TCP监测

  • 传输层检测：TCP_CHECK

TCP_CHECK {
   connect_ip <IP ADDRESS>    # 向当前RS哪个IP地址发起健康状态检测请求
   connect_port <PORT>        # 向当前RS的那个port发起健康状态检测请求
   bindto <IP ADDRESS>        # 向当前RS发出健康状态检测请求时使用的源地址
   bind_port <PORT>           # 向当前RS发出健康状态检测请求时使用的源端口
   connect_timeout <INTEGER>    # 客户端请求的超时时长，相当于haproxy的timeout server
}

  • 范例

virtual_server 10.0.0.10 80 {
   delay_loop 6
   lb_algo wrr
   lb_kind DR
   # persistence_timeout 120    # 会话保持时间
   protocol TCP
   sorry_server 127.0.0.1 80
   real_server 10.0.0.7 80 {
       weight 1
       TCP_CHECK {
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 80
       }
   }
   real_server 10.0.0.17 80 {
       weight 1
       TCP_CHECK {
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 80
       }
   }
}


实战案例

实现单主的LVS-DR模式

  • lvs客户端准备好脚本

  • rs为lvs的客户端，vs为lvs的服务端
vim /root/lvs-dr-rs.sh
#!/bin/bash
# LVS DR默认初始化脚本
LVS_VIP=10.0.0.100/32
DEV=lo:1
# source /etc/rc.d/init.d/functions
case "$1" in
start)
/sbin/ifconfig lo:0 $LVS_VIP netmask 255.255.255.255 # broadcast $LVS_VIP
# /sbin/route add -host $LVS_VIP dev lo:0
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore 
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore 
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce 
sysctl -p > /dev/null 2>&1
echo "RealServer RS Start OK"
;;
stop)
/sbin/ifconfig lo:0 down
# /sbin/route del $LVS_VIP > /dev/null 2>&1
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore 
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore 
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce 
sysctl -p > /dev/null 2>&1
echo "RealServer RS Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
​
# 启动脚本
bash lvs-dr-rs.sh start

  • k1与k2配置keepalived的lvs

cd /etc/keepalived/conf.d/
​
vim lvs_web1.conf
virtual_server 10.0.0.100 80 {
   delay_loop 3
   lb_algo wrr
   lb_kind DR
   protocol TCP
   sorry_server 127.0.0.1 80
   real_server 10.0.0.7 80 {
       weight 1
       HTTP_GET {       # 应用层检测
           url {
               path /monitor.html
               status_code 200
           }
           connect_timeout 1
           nb_get_retry 3
           delay_before_retry 3
       }
   }
   real_server 10.0.0.17 80 {
       weight 1
       TCP_CHECK {      # 另一台主机使用TCP检测
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 80
       }
   }
}
​
# k1
scp lvs_web1.conf 10.0.0.18:/etc/keepalived/cond.d/
​
systemctl restart keepalived

  • web1

yum install -y httpd
​
echo 10.0.0.7-monitor.html > /var/www/html/monitor.html
​
echo 10.0.0.7 > /var/www/html/index.html
​
systemctl enable --now httpd

  • web2

yum install -y httpd
​
echo 10.0.0.17-monitor.html > /var/www/html/monitor.html
​
echo 10.0.0.17 > /var/www/html/index.html
​
systemctl enable --now httpd

  • 访问测试k1/k2

查看lvs规则是否生成
ipvsadm -Ln
​
curl 10.0.0.100

  • 测试keepalived的健康检测问题

web1停止httpd服务，查看是否在web1上调度
​
# web1
systemctl stop httpd
​
# client
curl 10.0.0.100
​
发现不会在宕掉的节点上调度了
  • 得到结果，lvs高可用与健康检查都解决了
  • lvs监听的地址与端口是在内核中的，比http应用层的优先级要高


实现双主的LVS-DR模式

  • 规划

  • 10.0.0.100做apache的vip

  • 10.0.0.200做mysql的vip

  • web1与web2操作

yum install -y mariadb-server; systemctl enable --now mariadb
​
mysql -e "grant all on *.* to test@'10.0.0.%' identified by '123456'"
  • client
yum install -y mariadb
​
mysql -utest -h10.0.0.7 -p123456 -e "show variables like '%hostname%'"
​
mysql -utest -h10.0.0.17 -p123456 -e "show variables like '%hostname%'"
  • ka1
cd /etc/keepalived/conf.d/
​
vim mysql_vip.conf
vrrp_instance mysql {
  state BACKUP
  interface eth0
   virtual_router_id 88
   priority 80
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 123456
   }
   virtual_ipaddress {
       10.0.0.200/24 dev eth0 label eth0:200
   }
}
  • ka2
cd /etc/keepalived/conf.d/
​
vim mysql_vip.conf
vrrp_instance mysql {
  state MASTER
  interface eth0
   virtual_router_id 88
   priority 100
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 123456
   }
   virtual_ipaddress {
       10.0.0.200/24 dev eth0 label eth0:200
   }
}

  • ka1配置keepalived的lvs规则

cd /etc/keepalived/conf.d/
​
vim lvs_mysql.conf
virtual_server 10.0.0.200 3306 {
  delay_loop 3
  lb_algo wrr
   lb_kind DR
   protocol TCP
   sorry_server 127.0.0.1 33006
   real_server 10.0.0.7 3306 {
       weight 1
       TCP_CHECK {      # 另一台主机使用TCP检测
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 3306
       }
   }
   real_server 10.0.0.17 3306 {
       weight 1
       TCP_CHECK {      # 另一台主机使用TCP检测
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 3306
       }
   }
}

  • ka2配置keepalived的lvs规则

cd /etc/keepalived/conf.d/
​
vim lvs_mysql.conf
virtual_server 10.0.0.200 3306 {
  delay_loop 3
  lb_algo wrr
   lb_kind DR
   protocol TCP
   sorry_server 127.0.0.1 33006
   real_server 10.0.0.7 3306 {
       weight 1
       TCP_CHECK {      # 另一台主机使用TCP检测
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 3306
       }
   }
   real_server 10.0.0.17 3306 {
       weight 1
       TCP_CHECK {      # 另一台主机使用TCP检测
           connect_timeout 5
           nb_get_retry 3
           delay_before_retry 3
           connect_port 3306
       }
   }
}
  • mysql1与mysql2
ifconfig lo:2 10.0.0.200/32


防火墙标签绑定多个服务

  • 两个节点都执行以下操作
# mangle是死的，不能改
iptables -t mangle -A PREROUTING -d 10.0.0.100 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 6
​
iptables -t mangle -vnL
  • 配置keepalived文件(k1、k2)
vim /etc/keepalived/conf.d/web1_vip.conf
vrrp_instance web1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
virtual_ipaddress {
  10.0.0.100/24 dev eth0 label eth0:100
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
​
​
vim /etc/keepalived/conf.d/web1.conf
virtual_server fwmark 6 {    # 指定FWM为6
 delay_loop 3
 lb_algo wrr
 lb_kind DR
 protocol TCP
 sorry_server 127.0.0.1 80    # 端口必须指定
 real_server 10.0.0.7 80 {
     weight 1
     HTTP_GET {
         url {
             path /monitor.html
             status_code 200
         }
         connect_timeout 2
         nb_get_retry 3
         delay_before_retry 3
     }
 }
 real_server 10.0.0.17 80 {
     weight 1
     TCP_CHECK {
         connect_timeout 2
         nb_get_retry 3
         delay_before_retry 3
         connect_port 80
     }
 }
}
  • 访问测试
curl -k https://10.0.0.100/




实现其它应用的高可用性 VRRP Script

  • 分两步实现

  • vrrp_script: 自定义资源监控脚本，vrrp实现根据脚本返回值，公共定义，可被多个实例调用，定义在vrrp实例之外的独立配置块，一般放在global_defs设置块之后
  • 通常脚本用于监控指定应用的状态。一旦发现应用的状态异常，则触发对MASTER节点的权重减至低于SLAVE节点，从而实现VIP切换到SLAVE节点
vrrp_script <SCRIPT_NAME> {
   script <STRING>|<QUOTED-STRING>    # 此脚本返回值为非0时，会触发下面OPTIONS执行
   OPTIONS
}
  • 调用脚本
  • track_script：调用vrrp_script定义的脚本去监控资源，定义在VRRP实例之内，调用事先定义的vrrp_script
track_script {
   SCRIPT_NAME_1
   SCRIPT_NAME_2
}


定义vrrp script

vrrp_script <SCRIPT_NAME> {             # 定义一个检测脚本，在 global_defs 之外配置
script <STRING>|<QUOTED-STRING>      # shell命令或脚本路径
interval <INTEGER>                  # 间隔时间，单位为秒，默认1秒
timeout <INTEGER>                   # 超时时间
weight <INTEGER:-254..254>           # 默认为0，如果设置此值为负数，当上面脚本返回值为非0时，会将此值与本节点权重相加可以降低本节点权重，即表示fall，如果是正数，当脚本返回值为0，会将此值与本节点权重相加可以提高本节点权重，即表示 rise ，通常使用负值
fall <INTEGER>                     # 脚本几次失败转为失败，建议设为2以上
rise <INTEGER>                     # 脚本连续监测成功后，把服务器从失败标记为成功的次数。将优先级恢复
user USERNAME [GROUPNAME]           # 执行监测脚本的用户或组
init_fail                          # 设置默认标记为失败状态，监测成功之后再转换为成功状态
}


调用 VRRP script

vrrp_instance VI_1 {
'''
track_script {
   chk_down
}
}

  • 范例

  • 先恢复为一主一从的环境

  • k1
vim /etc/keepalived/conf.d/web_vip.conf
vrrp_instance web1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100            # 从设置为80
advert_int 1
virtual_ipaddress {
    10.0.0.100/24 dev eth0 label eth0:100
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vim /etc/keepalived/conf.d/check_down.conf
vrrp_script check_down {
   script "[ ! -f /etc/keepalived/down ]"
   interval 1
   weight -30
   fall 3
   rise 2
   timeout 2
}
  • ka1调用脚本
vim /etc/keepalived/conf.d/web_vip.conf
vrrp_instance web1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100            # 从设置为80
advert_int 1
virtual_ipaddress {
    10.0.0.100/24 dev eth0 label eth0:100
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
   check_down    # 上面脚本配置文件定义的名称
}
}
​
systemctl restart keeepalived

  • client抓包测试

tcpdump -i eth0 host 224.1.1.1
​
# ka1创建文件
touch /etc/keepalived/down     # 查看优先级是不是被减了30，如果是那么脚本设置成功
​
# ka1再删除文件查看vip是否能回来
rm -f /etc/keepalived/down




nginx的反向代理高可用

  • 利用脚本文件的方法

  • k1与k2准备环境

dnf install -y nginx
​
vim /etc/nginx/nginx.conf
''''
http {
  upstream webservers {
      server 10.0.0.7:80;
      server 10.0.0.17:80;
  }
  server {
      ''''
      location / {
          proxy_pass http://webservers/;
      }
  }
}
​
# 将lvs配置文件去掉，只配置vip,因为这里使用nginx作为反向代理
cat /etc/keepalived/conf.d/web1_vip.conf 
vrrp_instance web1 {
   state MASTER
   interface eth0
   virtual_router_id 66
   priority 100
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 123456
   }
  unicast_src_ip 10.0.0.8
  unicast_peer {
      10.0.0.18
  }
  virtual_ipaddress {
      10.0.0.100 dev eth0 label eth0:1
  }
  notify_master "/etc/keepalived/notify.sh master"
  notify_backup "/etc/keepalived/notify.sh backup"
  notify_fault "/etc/keepalived/notify.sh fault"
}
​
systemctl restart nginx keepalived
  • client测试
curl 10.0.0.100
  • 虽然keepalived可以完成自动切换了，但是当nginx挂了，nginx反向代理也就失效，vip也不会发生切换，现在配置检测脚本解决问题
  • 思路：检测nginx进程是否存在
pgrep nginx
​
# 发送0信号
killall -0 nginx
​
echo $?
0 正常
非零不正常
  • k1与k2编写keepalived脚本
cd /etc/keepalived/
​
vim check_nginx.conf
vrrp_script check_nginx {
  script "/etc/keepalived/conf.d/check_nginx.sh"
  interval 1
  weight -30
  fall 3
  rise 2
  timeout 2
}
​
vim /etc/keepalived/conf.d/check_nginx.sh
#!/bin/bash
/usr/bin/killall -0 nginx &> /dev/null
​
chmod +x /etc/keepalived/conf.d/check_nginx.sh
​
vim /etc/keepalived/conf.d/web_vip.conf
vrrp_instance web1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100            # 从设置为80
advert_int 1
virtual_ipaddress {
   10.0.0.100/24 dev eth0 label eth0:100
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
  check_nginx
}
}

  • 测试

# 监控
tcpdump -i eth0 host 224.1.1.1
​
# k1中将nginx服务停止，查看优先级和vip是否发送变动
killall nginx
​
# client是否有通
curl 10.0.0.100
​
# ka1再将nginx服务起来，查看优先级和vip是否能恢复
systemctl start nginx
  • 更改为能自愈的脚本
vim /etc/keepalived/conf.d/check_nginx.sh
#!/bin/bash
/usr/bin/killall -0 nginx &> /dev/null || systemctl restart nginx
systemctl is-active nginx
【版权声明】本文为华为云社区用户原创内容，未经允许不得转载，如需转载请自行联系原作者进行授权。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱： cloudbbs@huaweicloud.com
  • 点赞
  • 收藏
  • 关注作者

评论（0）

0/1000
抱歉，系统识别当前为高风险访问，暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称，即可参与社区互动！

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

确认 取消

加入云驻计划，成为创作者

  • 华为云周边好礼
  • 免费体验产品
  • 特殊身份标识
  • 线下官方门票
  • 内部专家零距离
  • 与10000+优质创作者共同成长
立即加入