Kubernetes存储体系与CSI驱动开发深度实战：从StorageClass到StatefulSet到分布式数据卷的全链路管理全解析

引言

容器存储是Kubernetes中最复杂的子系统之一。从简单的emptyDir到分布式块存储，从单节点ReadWriteOnce到多节点ReadWriteMany，从静态PV到CSI动态供给，K8s存储体系覆盖了应用临时存储、配置注入、数据持久化、分布式数据共享等多种场景。CSI（Container Storage Interface）标准的出现让存储厂商可以通过统一接口对接K8s，但CSI驱动开发本身涉及身份验证、快照管理、扩容、拓扑感知等复杂特性。本文从K8s存储模型出发，系统性地讲解PV/PVC生命周期、StorageClass动态供给、StatefulSet存储管理、CSI驱动架构与开发、存储快照与克隆、数据卷扩容与迁移、分布式文件系统（CephFS/NFS/Longhorn）、生产存储选型，通过大量可运行配置和代码示例帮助读者掌握K8s存储的完整实践。

一、K8s存储模型与Volume类型

1.1 Volume类型全景

# K8s Volume类型分类
#
# 1. 临时存储（Pod生命周期绑定）
#    - emptyDir: Pod内共享临时目录，Pod删除即消失
#    - configMap: 配置文件注入
#    - secret: 敏感配置注入
#    - downwardAPI: Pod元信息注入
#
# 2. 持久存储（独立于Pod生命周期）
#    - hostPath: 主机路径（不推荐生产）
#    - local: 本地卷（比hostPath更规范）
#    - persistentVolumeClaim: 持久卷声明
#    - nfs: NFS共享存储
#    - cephfs: Ceph分布式文件系统
#    - fc/iSCSI: SAN存储
#
# 3. 投射存储（Projected Volume）
#    - projected: 组合多种数据源到一个目录

# emptyDir示例
apiVersion: v1
kind: Pod
metadata:
  name: shared-data-pod
spec:
  volumes:
  - name: shared-data
    emptyDir:
      medium: Memory          # 使用内存（tmpfs）
      sizeLimit: 512Mi
  - name: config
    configMap:
      name: app-config
      items:
      - key: config.yaml
        path: config.yaml
  - name: secrets
    secret:
      secretName: app-secrets
      defaultMode: 0400       # 只读权限
  containers:
  - name: writer
    image: writer:v1
    volumeMounts:
    - name: shared-data
      mountPath: /shared
  - name: reader
    image: reader:v1
    volumeMounts:
    - name: shared-data
      mountPath: /shared
      readOnly: true
    - name: config
      mountPath: /etc/config
      readOnly: true
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true

1.2 ConfigMap与Secret使用

# ConfigMap：非敏感配置
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  config.yaml: |
    database:
      host: postgres.production.svc
      port: 5432
      pool: 20
    redis:
      host: redis.production.svc
      port: 6379
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
---
# Secret：敏感配置（etcd中可加密存储）
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: production
type: Opaque
stringData:                      # 明文写入，K8s自动base64编码
  DATABASE_PASSWORD: "super-secret-db-pass"
  JWT_SECRET: "jwt-signing-key-2026"
  API_KEY: "sk-xxx-yyy-zzz"
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
---
# 以环境变量注入
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      containers:
      - name: app
        image: app:v1
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets
        env:
        - name: DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: DATABASE_HOST
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: DATABASE_PASSWORD
---
# 以文件挂载（支持热更新）
apiVersion: v1
kind: Pod
metadata:
  name: config-mount-pod
spec:
  volumes:
  - name: config
    configMap:
      name: app-config
      items:
      - key: config.yaml
        path: app-config.yaml
        mode: 0644
  containers:
  - name: app
    image: app:v1
    volumeMounts:
    - name: config
      mountPath: /etc/app
      readOnly: true
    # subPath：挂载单个文件而不覆盖目录
    - name: config
      mountPath: /etc/app/override.yaml
      subPath: config.yaml

二、PV/PVC生命周期管理

2.1 静态PV创建

# PersistentVolume：集群级资源
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-001
  labels:
    type: nfs
    environment: production
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteMany             # 多节点读写
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs       # 匹配StorageClass
  mountOptions:
  - hard
  - nfsvers=4.1
  - timeo=600
  - retrans=2
  nfs:
    server: nfs.example.com
    path: /export/k8s/pv-001
  # nodeAffinity：限制PVC只能在特定节点挂载
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-east-1a
---
# PersistentVolumeClaim：命名空间级，用户声明需求
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: production
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: nfs
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      type: nfs
      environment: production

2.2 AccessModes详解

# AccessModes（访问模式）
#
# ReadWriteOnce (RWO)：
#   单节点读写（最常见）
#   适用：EBS、本地盘、Ceph RBD
#   不适用：需要多Pod同时读写的场景
#
# ReadOnlyMany (ROX)：
#   多节点只读
#   适用：NFS共享只读数据
#   不适用：需要写入的场景
#
# ReadWriteMany (RWX)：
#   多节点读写
#   适用：NFS、CephFS、GlusterFS
#   不适用：块存储（EBS、本地盘）
#
# ReadWriteOncePod (RWOP)：
#   单Pod读写（K8s 1.22+）
#   比RWO更严格：同一节点也不能两个Pod同时挂载
#   适用：确保数据一致性的场景

2.3 ReclaimPolicy与回收

# ReclaimPolicy：PVC删除后PV的处理方式
#
# Retain（保留）：
#   PVC删除 → PV变为Released状态 → 数据保留
#   需要手动清理PV和底层数据
#   适用：生产数据、不可恢复数据
#
# Delete（删除）：
#   PVC删除 → PV和底层数据自动删除
#   适用：临时数据、开发环境
#
# Recycle（回收，已废弃）：
#   PVC删除 → PV变为Available → 数据被清空
#   K8s 1.15+已废弃

三、StorageClass动态供给

3.1 动态供给配置

# StorageClass：定义动态供给策略
# 用户创建PVC → CSI驱动自动创建PV → 绑定PVC

# AWS EBS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
  kmsKeyId: "arn:aws:kms:us-east-1:xxx:key/yyy"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
mountOptions:
  - noatime
---
# NFS（需nfs-subdir-external-provisioner）
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-shared
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs.example.com
  share: /export/k8s
  pathPattern: "${.PVC.namespace}-${.PVC.name}"
  onDelete: delete
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
# Longhorn（分布式块存储）
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"        # 3副本数据冗余
  staleReplicaTimeout: "30"
  diskSelector: "ssd"           # 选用SSD磁盘
  nodeSelector: "storage"
  recurringJobSelector: '[{"name":"backup","isGroup":true}]'
  dataLocality: "disabled"
  replicaAutoBalance: "best-effort"
---
# Ceph RBD（块存储）
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: "ceph-cluster-1"
  pool: "rbd-pool"
  imageFormat: "2"
  imageFeatures: "layering"
  csi.storage.k8s.io/provisioner-secret-name: ceph-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph
  csi.storage.k8s.io/node-stage-secret-name: ceph-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph
reclaimPolicy: Retain
allowVolumeExpansion: true

3.2 VolumeBindingMode策略

# volumeBindingMode：控制PV绑定时机
#
# Immediate（立即绑定）：
#   PVC创建时立即创建并绑定PV
#   问题：Pod可能调度到不同AZ，PV无法跨AZ挂载
#
# WaitForFirstConsumer（延迟绑定）：
#   等Pod调度后才创建PV（在Pod所在节点/AZ创建）
#   推荐：所有需要拓扑感知的存储

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: topology-aware
provisioner: disk.csi.azure.com
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
    - key: topology.kubernetes.io/zone
      values:
      - eastus-1
      - eastus-2
      - eastus-3

四、StatefulSet存储管理

4.1 StatefulSet与volumeClaimTemplates

# StatefulSet：有状态应用部署
# 每个Pod有唯一标识（稳定网络名+稳定存储）

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-cluster
  namespace: production
spec:
  serviceName: postgres-cluster     # 关联Headless Service
  replicas: 3
  podManagementPolicy: Parallel      # 并行管理（默认OrderedReady）
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0                  # 全部更新
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: postgres
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 300
      containers:
      - name: postgres
        image: postgres:16
        ports:
        - containerPort: 5432
          name: postgres
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
          subPath: postgres           # 子路径防止权限问题
        - name: config
          mountPath: /etc/postgresql
        resources:
          requests:
            cpu: 1000m
            memory: 2Gi
          limits:
            cpu: 4000m
            memory: 8Gi
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 60
          periodSeconds: 30
      volumes:
      - name: config
        configMap:
          name: postgres-config
  # volumeClaimTemplates：每个Pod自动创建独立PVC
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: postgres
      annotations:
        # 备份策略标注
        backup: "daily"
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 200Gi
---
# Headless Service：为StatefulSet提供稳定DNS
apiVersion: v1
kind: Service
metadata:
  name: postgres-cluster
  namespace: production
spec:
  clusterIP: None              # Headless
  selector:
    app: postgres
  ports:
  - port: 5432
    name: postgres

4.2 稳定标识与DNS

# StatefulSet Pod命名规则：{statefulset-name}-{ordinal}
# postgres-cluster-0, postgres-cluster-1, postgres-cluster-2

# DNS解析：
# {pod-name}.{service-name}.{namespace}.svc.cluster.local
# postgres-cluster-0.postgres-cluster.production.svc.cluster.local

# 测试DNS解析
kubectl run dns-test --image=busybox --rm -it -- \
  nslookup postgres-cluster-0.postgres-cluster.production.svc.cluster.local

# Pod间通信（PostgreSQL复制配置）
# 在postgres-cluster-0上：
# primary_conninfo = 'host=postgres-cluster-1.postgres-cluster.production.svc.cluster.local port=5432'

五、CSI驱动架构与开发

5.1 CSI规范与组件

# CSI（Container Storage Interface）规范
#
# 三大gRPC服务：
#   Identity Service：驱动身份信息
#   Controller Service：卷生命周期管理（创建/删除/快照/扩容）
#   Node Service：节点级卷操作（挂载/卸载/格式化）
#
# K8s CSI组件：
#   1. CSI Driver：存储厂商实现的驱动（gRPC服务端）
#   2. External Provisioner：监听PVC，调用CSI创建PV
#   3. External Attacher：监听VolumeAttachment，调用CSI挂载
#   4. External Resizer：监听PVC扩容请求
#   5. External Snapshotter：监听VolumeSnapshot
#   6. Node Driver Registrar：注册CSI到kubelet
#   7. liveness-probe：健康检查
#
# 部署模式：
#   Controller：Deployment（集群级，处理创建/删除）
#   Node：DaemonSet（每节点，处理挂载/卸载）

5.2 CSI驱动开发示例

// CSI Identity Service实现
package main

import (
    "context"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    "github.com/container-storage-interface/spec/lib/go/csi"
)

type IdentityServer struct {
    csi.UnimplementedIdentityServer
    name    string
    version string
}

func (s *IdentityServer) GetPluginInfo(ctx context.Context,
    req *csi.GetPluginInfoRequest) (*csi.GetPluginInfoResponse, error) {
    return &csi.GetPluginInfoResponse{
        Name:          s.name,
        VendorVersion: s.version,
        Manifest: map[string]string{
            "description": "Custom CSI Driver for object storage",
            "version":     s.version,
        },
    }, nil
}

func (s *IdentityServer) GetPluginCapabilities(ctx context.Context,
    req *csi.GetPluginCapabilitiesRequest) (*csi.GetPluginCapabilitiesResponse, error) {
    return &csi.GetPluginCapabilitiesResponse{
        Capabilities: []*csi.PluginCapability{
            {
                Type: &csi.PluginCapability_Service_{
                    Service: &csi.PluginCapability_Service{
                        Type: csi.PluginCapability_Service_CONTROLLER_SERVICE,
                    },
                },
            },
            {
                Type: &csi.PluginCapability_Service_{
                    Service: &csi.PluginCapability_Service{
                        Type: csi.PluginCapability_Service_VOLUME_ACCESSIBILITY_CONSTRAINTS,
                    },
                },
            },
        },
    }, nil
}

func (s *IdentityServer) Probe(ctx context.Context,
    req *csi.ProbeRequest) (*csi.ProbeResponse, error) {
    return &csi.ProbeResponse{}, nil
}

// CSI Controller Service实现
type ControllerServer struct {
    csi.UnimplementedControllerServer
    storage StorageBackend
}

// CreateVolume：创建卷
func (s *ControllerServer) CreateVolume(ctx context.Context,
    req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) {
    // 验证参数
    if req.Name == "" {
        return nil, status.Error(codes.InvalidArgument, "volume name required")
    }
    if req.CapacityRange == nil {
        return nil, status.Error(codes.InvalidArgument, "capacity required")
    }

    // 检查是否已存在
    existing, err := s.storage.GetVolume(req.Name)
    if err == nil && existing != nil {
        // 幂等：返回已有卷
        return &csi.CreateVolumeResponse{
            Volume: existing.ToCSIVolume(),
        }, nil
    }

    // 计算容量
    requiredBytes := req.CapacityRange.RequiredBytes
    limitBytes := req.CapacityRange.LimitBytes

    // 创建卷
    vol, err := s.storage.CreateVolume(req.Name, requiredBytes, limitBytes,
        req.Parameters, req.AccessibilityRequirements)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "create volume: %v", err)
    }

    // 拓扑信息
    if req.AccessibilityRequirements != nil {
        vol.AccessibleTopology = req.AccessibilityRequirements.Requisite[0]
    }

    return &csi.CreateVolumeResponse{
        Volume: &csi.Volume{
            VolumeId:      vol.ID,
            CapacityBytes: vol.Capacity,
            VolumeContext: vol.Context,
            ContentSource: req.VolumeContentSource,
            AccessibleTopology: vol.Topologies,
        },
    }, nil
}

// DeleteVolume：删除卷
func (s *ControllerServer) DeleteVolume(ctx context.Context,
    req *csi.DeleteVolumeRequest) (*csi.DeleteVolumeResponse, error) {
    if req.VolumeId == "" {
        return nil, status.Error(codes.InvalidArgument, "volume ID required")
    }
    err := s.storage.DeleteVolume(req.VolumeId)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "delete: %v", err)
    }
    return &csi.DeleteVolumeResponse{}, nil
}

// ControllerPublishVolume：将卷附加到节点
func (s *ControllerServer) ControllerPublishVolume(ctx context.Context,
    req *csi.ControllerPublishVolumeRequest) (*csi.ControllerPublishVolumeResponse, error) {
    // 块存储需要attach操作
    // 对象存储可跳过
    return &csi.ControllerPublishVolumeResponse{}, nil
}

// ControllerExpandVolume：在线扩容
func (s *ControllerServer) ControllerExpandVolume(ctx context.Context,
    req *csi.ControllerExpandVolumeRequest) (*csi.ControllerExpandVolumeResponse, error) {
    if req.VolumeId == "" {
        return nil, status.Error(codes.InvalidArgument, "volume ID required")
    }
    if req.CapacityRange == nil {
        return nil, status.Error(codes.InvalidArgument, "capacity required")
    }
    err := s.storage.ResizeVolume(req.VolumeId, req.CapacityRange.RequiredBytes)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "expand: %v", err)
    }
    return &csi.ControllerExpandVolumeResponse{
        CapacityBytes:         req.CapacityRange.RequiredBytes,
        NodeExpansionRequired: true,  // 需要Node端resize文件系统
    }, nil
}

// CreateSnapshot：创建快照
func (s *ControllerServer) CreateSnapshot(ctx context.Context,
    req *csi.CreateSnapshotRequest) (*csi.CreateSnapshotResponse, error) {
    snap, err := s.storage.CreateSnapshot(req.SourceVolumeId, req.Name)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "snapshot: %v", err)
    }
    return &csi.CreateSnapshotResponse{
        Snapshot: &csi.Snapshot{
            SnapshotId:     snap.ID,
            SourceVolumeId: snap.SourceVolumeId,
            SizeBytes:      snap.Size,
            CreationTime:   snap.CreatedAt,
            ReadyToUse:     snap.Ready,
        },
    }, nil
}

5.3 CSI Driver部署

# CSI Driver注册
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: mycs.example.com
spec:
  attachRequired: false          # 对象存储不需要attach
  podInfoOnMount: true
  fsGroupPolicy: File            # 支持fsGroup
  volumeLifecycleModes:
  - Persistent
  - Ephemeral                    # 支持临时卷
---
# Controller Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mycsi-controller
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mycsi-controller
  template:
    spec:
      serviceAccountName: mycsi-controller
      containers:
      # CSI驱动
      - name: csi-driver
        image: mycsi:v1.0.0
        args:
        - --endpoint=unix:///csi/csi.sock
        - --nodeid=$(NODE_ID)
        - --v=4
        env:
        - name: NODE_ID
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
      # External Provisioner
      - name: csi-provisioner
        image: registry.k8s.io/sig-storage/csi-provisioner:v4.0.0
        args:
        - --csi-address=/csi/csi.sock
        - --leader-election=true
        - --leader-election-lease-duration=15s
        - --feature-gates=Topology=true
        - --extra-create-metadata=true
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
      # External Attacher
      - name: csi-attacher
        image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0
        args:
        - --csi-address=/csi/csi.sock
        - --leader-election=true
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
      # External Resizer
      - name: csi-resizer
        image: registry.k8s.io/sig-storage/csi-resizer:v1.9.0
        args:
        - --csi-address=/csi/csi.sock
        - --leader-election=true
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
      # External Snapshotter
      - name: csi-snapshotter
        image: registry.k8s.io/sig-storage/csi-snapshotter:v7.0.0
        args:
        - --csi-address=/csi/csi.sock
        - --leader-election=true
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
      volumes:
      - name: socket-dir
        emptyDir: {}
---
# Node DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: mycsi-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: mycsi-node
  template:
    spec:
      hostNetwork: true
      containers:
      - name: csi-driver
        image: mycsi:v1.0.0
        args:
        - --endpoint=unix:///csi/csi.sock
        - --nodeid=$(NODE_ID)
        env:
        - name: NODE_ID
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          privileged: true
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
        - name: mount-dir
          mountPath: /var/lib/kubelet/pods
          mountPropagation: Bidirectional
        - name: device-dir
          mountPath: /dev
      # Node Driver Registrar
      - name: node-driver-registrar
        image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0
        args:
        - --csi-address=/csi/csi.sock
        - --kubelet-registration-path=/var/lib/kubelet/plugins/mycsi.example.com/csi.sock
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
        - name: registration-dir
          mountPath: /registration
      # Liveness Probe
      - name: liveness-probe
        image: registry.k8s.io/sig-storage/livenessprobe:v2.13.0
        args:
        - --csi-address=/csi/csi.sock
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
      volumes:
      - name: socket-dir
        hostPath:
          path: /var/lib/kubelet/plugins/mycsi.example.com
          type: DirectoryOrCreate
      - name: mount-dir
        hostPath:
          path: /var/lib/kubelet/pods
          type: Directory
      - name: device-dir
        hostPath:
          path: /dev
      - name: registration-dir
        hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: Directory

六、存储快照与克隆

6.1 VolumeSnapshot

# VolumeSnapshot：数据备份基础
# 前提：CSI驱动和StorageClass支持快照

# VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshot
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: ebs.csi.aws.com
deletionPolicy: Retain            # Delete | Retain
parameters:
  encrypted: "true"
---
# 创建快照
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-snap-20250115
  namespace: production
spec:
  volumeSnapshotClassName: ebs-snapshot
  source:
    persistentVolumeClaimName: postgres-data
---
# 从快照恢复
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-restored
  namespace: production
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ebs-gp3
  resources:
    requests:
      storage: 200Gi
  dataSource:
    name: postgres-snap-20250115
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

6.2 Volume Clone

# Volume克隆：从已有PVC创建新PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-clone
  namespace: production
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ebs-gp3
  resources:
    requests:
      storage: 200Gi
  dataSource:
    name: postgres-data        # 源PVC名称
    kind: PersistentVolumeClaim

6.3 定期快照备份

# VolumeSnapshot定时创建
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-auto-snap-$(date +%Y%m%d%H%M)
  namespace: production
  labels:
    type: auto-backup
spec:
  volumeSnapshotClassName: ebs-snapshot
  source:
    persistentVolumeClaimName: postgres-data
---
# CronJob触发快照
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
  namespace: production
spec:
  schedule: "0 2 * * *"          # 每天凌晨2点
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: snapshot-creator
          containers:
          - name: kubectl
            image: bitnami/kubectl:1.30
            command:
            - sh
            - -c
            - |
              SNAP_NAME="postgres-snap-$(date +%Y%m%d)"
              cat << EOF | kubectl apply -f -
              apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                name: ${SNAP_NAME}
                namespace: production
              spec:
                volumeSnapshotClassName: ebs-snapshot
                source:
                  persistentVolumeClaimName: postgres-data
              EOF
              # 清理30天前快照
              kubectl get volumesnapshot -n production \
                -l type=auto-backup \
                -o jsonpath='{.items[?(@.metadata.creationTimestamp)].metadata.name}' | \
                while read snap; do
                  AGE=$(kubectl get volumesnapshot $snap -n production \
                    -o jsonpath='{.metadata.creationTimestamp}')
                  # 逻辑：30天前的快照删除
                done
          restartPolicy: OnFailure

七、分布式存储方案

7.1 Longhorn部署

# Longhorn：K8s原生分布式块存储
# 特性：多副本、快照、备份、精简配置

# Helm安装
# helm install longhorn longhorn/longhorn \
#   --namespace longhorn-system \
#   --create-namespace \
#   --set persistence.defaultClassReplicaCount=3 \
#   --set persistence.defaultDataLocality=best-effort \
#   --set defaultSettings.backupTarget=s3://backup@us-east-1/ \
#   --set defaultSettings.backupTargetCredentialSecret=longhorn-backup-secret

# Longhorn StorageClass（高级配置）
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ha
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  diskSelector: "ssd,fast"
  nodeSelector: "storage-node"
  dataLocality: "best-effort"
  replicaAutoBalance: "best-effort"
  recurringJobSelector: |
    [
      {
        "name": "snapshot",
        "isGroup": true,
        "groups": ["default"]
      },
      {
        "name": "backup",
        "isGroup": true,
        "groups": ["default"]
      }
    ]
---
# Longhorn RecurringJob（定期快照+备份）
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: daily-backup
  namespace: longhorn-system
spec:
  name: daily-backup
  groups:
  - default
  task: backup                 # snapshot | backup
  cron: "0 2 * * *"            # 每天凌晨2点
  retain: 7                    # 保留7个备份
  concurrency: 2               # 并发数
---
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: hourly-snapshot
  namespace: longhorn-system
spec:
  name: hourly-snapshot
  task: snapshot
  cron: "0 * * * *"
  retain: 24                   # 保留24个快照

7.2 MinIO对象存储

# MinIO：S3兼容的K8s原生对象存储
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
  namespace: storage
spec:
  serviceName: minio
  replicas: 4                  # 分布式模式至少4节点
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: minio
            topologyKey: kubernetes.io/hostname
      containers:
      - name: minio
        image: minio/minio:RELEASE.2025-01-01T00-00-00Z
        args:
        - server
        - http://minio-{0...3}.minio.storage.svc.cluster.local/data{0...3}
        - --console-address=:9001
        env:
        - name: MINIO_ROOT_USER
          valueFrom:
            secretKeyRef:
              name: minio-credentials
              key: access-key
        - name: MINIO_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: minio-credentials
              key: secret-key
        ports:
        - containerPort: 9000
          name: api
        - containerPort: 9001
          name: console
        volumeMounts:
        - name: data-0
          mountPath: /data0
        - name: data-1
          mountPath: /data1
        - name: data-2
          mountPath: /data2
        - name: data-3
          mountPath: /data3
        resources:
          requests:
            cpu: 1000m
            memory: 2Gi
          limits:
            cpu: 4000m
            memory: 8Gi
        readinessProbe:
          httpGet:
            path: /minio/health/ready
            port: 9000
          initialDelaySeconds: 10
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /minio/health/live
            port: 9000
          initialDelaySeconds: 30
          periodSeconds: 30
  volumeClaimTemplates:
  - metadata:
      name: data-0
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 500Gi
  - metadata:
      name: data-1
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 500Gi

八、存储选型决策矩阵

# 存储选型决策矩阵
#
# 场景                     | 推荐方案             | AccessMode | 特点
# -------------------------|---------------------|------------|-----
# 数据库（PostgreSQL/MySQL）| EBS/io2/本地NVMe      | RWO        | 高IOPS低延迟
# Redis缓存                | 本地盘/memory         | RWO        | 极低延迟
# 日志收集（Loki/ES）       | NFS/CephFS            | RWX        | 多Pod共享
# 静态资源/图片             | MinIO/S3             | 对象存储    | 海量低成本
# 配置文件                  | ConfigMap            | 投射       | 热更新
# CI/CD工作区               | emptyDir + 缓存        | 临时       | 高性能
# 机器学习数据集            | NFS/对象存储           | RWX        | 多Pod读
# 分布式数据库（Cassandra） | local卷              | RWO        | 数据局部性
# 共享配置文件              | projected volume      | 投射       | 合并多源
# 临时计算中间结果           | emptyDir (Memory)     | 临时       | 内存速度

九、总结

Kubernetes存储体系从临时emptyDir到分布式对象存储，覆盖了应用对存储的全方位需求。理解PV/PVC的绑定机制和ReclaimPolicy是避免数据丢失的基础——生产环境必须使用Retain策略。StorageClass的动态供给极大简化了存储管理，但volumeBindingMode的选择直接影响拓扑感知能力——WaitForFirstConsumer是生产环境的标准选择。StatefulSet的volumeClaimTemplates为有状态应用提供了稳定标识+稳定存储的基石。CSI标准让存储厂商通过统一接口接入K8s生态，但CSI驱动开发涉及Controller和Node两套服务、多个External Sidecar组件协作，复杂度不低。存储快照是数据备份的基础手段，结合CronJob可以实现自动化备份策略。Longhorn作为K8s原生的分布式块存储，在自建集群中是优秀的EBS替代方案。生产存储选型的核心原则是：按需选择AccessMode（不要过度使用RWX），按IOPS需求选择存储类型（数据库用io2、缓存用本地盘、日志用NFS），按成本选择存储层级（热数据用SSD、冷数据用对象存储或归档），按可用性需求选择副本数（关键数据3副本+跨AZ备份）。