基于多节点多硬盘部署的Minio高可用集群

先决条件

Minio高可用集群至少满足四个节点才能开启纠删码模式，此处基于官网的多节点文档，部署一个标准的Minio最小集群,并且通过负载均衡的模式代理集群地址(此处使用nginx，也可使用haproxy)

环境

四个ubuntu 服务器节点,并且每台服务器需要挂在四块同样大小的磁盘(因为minio要求作为存储的磁盘一定是要干净的，不能为/路径下的文件夹)
服务器信息:

网络设置和防火墙(每个节点)

默认当前ubuntu服务器已经开启ufw防火墙，此时需要开启minio对应端口的访问策略
部署的Minio服务器必须使用相同的监听端口，默认情况下为9000和9001
ufw allow 9000
ufw allow 9001
在每台节点上设置连续的主机名

vim /etc/hosts
192.168.28.157 minio-1
192.168.28.158 minio-2
192.168.28.159 minio-3
192.168.28.160 minio-4

创建服务器池时，Minio需要{1…4}来表示一系列连续的主机

具有连续挂在的本地存储

Minio推荐使用XFS格式的磁盘以获得最佳性能
在每台服务器节点上新增挂载硬盘，并且硬盘大小需一致
Minio中强烈建议使用/etc/fstab 挂载配置，以确保重启后的驱动器顺序不变

$ mkfs.xfs /dev/sdb -L DISK1
$ mkfs.xfs /dev/sdc -L DISK2
$ mkfs.xfs /dev/sdd -L DISK3
$ mkfs.xfs /dev/sde -L DISK4

$ nano /etc/fstab

  # <file system>  <mount point>  <type>  <options>         <dump>  <pass>
  LABEL=DISK1      /minio/disk1     xfs     defaults,noatime  0       2
  LABEL=DISK2      /minio/disk2     xfs     defaults,noatime  0       2
  LABEL=DISK3      /minio/disk3     xfs     defaults,noatime  0       2
  LABEL=DISK4      /minio/disk4     xfs     defaults,noatime  0       2

设置为连续的挂载,是为了方便使用{1…4}的方式指定整个驱动区的范围
MinIO 不支持 将具有现有MinIO数据的驱动器任意迁移到新的挂载位置， 无论是故意还是作为操作系统级别行为的结果。

部署Minio服务器

在每个节点安装Minio 安装二进制包

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

在每个节点创建systemd系统启动服务文件

sudo touch /usr/lib/systemd/system/minio.service
sudo vim /usr/lib/systemd/system/minio.service


[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

默认情况下,minio是通过minio-user用户和组运行的，所以需要创建对应用户和用户组.并且需要设置文件夹路径访问权限

groupadd -r minio-user
useradd -M -r -g minio-user minio-user
chown minio-user:minio-user /minio/disk1 /minio/disk2 /minio/disk3 /minio/disk4

创建服务环境变量文件

创建一个环境变量配置文件在/etc/default/minio文件
服务使用此文件作为所有环境变量的来源
示例如下:

MINIO_VOLUMES="http://minio-{1...4}:9000/minio/disk{1...4}"
MINIO_OPTS="--console-address :9001"
MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=admin
MINIO_SERVER_URL="https://minio.example.com"
MINIO_BROWSER_REDIRECT_URL="https://minio.example.com/minio/ui"

需要注意的是此处通过池化的写法，将MINIO_VOLUMES的地址范围，磁盘的路径通过{1…4}表达，简化了管理
MINIO_SERVER_URL设置为了负载均衡后的地址
MINIO_BROWSER_REDIRECT_URL也设置为了负载均衡后的地址

按顺序启动Minio服务

启动minio
sudo systemctl start minio.service
使用以下命令确认服务是否正常

sudo systemctl status minio.service
journalctl -f -u minio.service

minio 会记录在启动时的关键信息，如果有异常，可以查看日志进行排查
设置minio服务为自启动
sudo systemctl enable minio.service

设置负载均衡

此处通过nginx来设置负载均衡
nginx的配置如下:

upstream minio_server {
 least_conn;
 server 192.168.28.157:9000;
 server 192.168.28.158:9000;
 server 192.168.28.159:9000;
 server 192.168.28.160:9000;
}
upstream minio_console {
 least_conn;
 server 192.168.28.157:9001;
 server 192.168.28.158:9001;
 server 192.168.28.159:9001;
 server 192.168.28.160:9001;

}
server {
    listen 443 ssl;
    ssl_certificate      /root/.acme.sh/*.llmyhome.cloud/llmyhome.cloud.cert.pem;
    ssl_certificate_key  /root/.acme.sh/*.llmyhome.cloud/llmyhome.cloud.key.pem;
    server_name minio.llmyhome.cloud;
    client_max_body_size 0;
    proxy_buffering off;
    proxy_request_buffering off;
    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;
      proxy_pass http://minio_server; # This uses the upstream directive definition to load balance
        }
   location /minio/ui/ {
      rewrite ^/minio/ui/(.*) /$1 break;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;
      real_ip_header X-Real-IP;
      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      chunked_transfer_encoding off;
      proxy_pass http://minio_console; # This uses the upstream directive definition to load balance
        }
}

重启nginx

nginx -t 
nginx -s reload

通过 域名访问minio集群
并用设置好的账户密码进行登录