测试环境：

CPU:Kunpeng 920 8Core
MEM:16G
Storage:200G
OS:openEuler 20.03 (LTS-SP3)

1 测试磁盘性能工具——Fio

1.1 编译安装——Fio

1. 安装Fio环境

yum install libaio-devel -y

2. 通过yum源直接安装

yum list *fio*

​ 运行以上指令，回显如下即可使用yum安装。

[root@controller nmon]# yum list *fio*
Last metadata expiration check: 0:01:17 ago on Tue 03 Jan 2023 10:31:48 AM CST.
Installed Packages
fio.aarch64                                                                 3.7-7.oe1                                                     @OS
fio-help.aarch64                                                            3.7-7.oe1                                                     @OS
Available Packages
fio.src                                                                     3.7-7.oe1                                                     source
fio-debuginfo.aarch64                                                       3.7-7.oe1                                                     debuginfo
fio-debugsource.aarch64                                                     3.7-7.oe1                                                     debuginfo

3. 源码编译安装

​ 下载源代码：

wget https://brick.kernel.dk/snaps/fio-3.7.tar.gz

​ 解压并编译：

tar -zxvf fio-3.7.tar.gz
cd fio-3.7
make
make install

1.2 验证与运行——Fio

1. 执行如下命令，查看Fio版本

fio --version

​ 返回内容如下所示，表示安装已经完成：

[root@controller ~]# fio --version
fio-3.7

2. 运行

​ 参数说明：

filename=/dev/vdb        测试文件名称，通常选择需要测试的盘的data目录
direct=1                 测试过程绕过机器自带的buffer，使测试结果更真实
rw=read                  顺序读
rw=write                 顺序写
rw=readwrite             顺序混合读写
rw=randwrite             随机写
rw=randread              随机读
rw=randrw                随机混合读写
bs=16k                   单次io的块文件大小为16k
bsrange=512-2048         同上，提定数据块的大小范围
size=1g                  本次的测试文件大小为1g，以每次4k的io进行测试
numjobs=30               本次的测试线程为30
runtime=1000             测试时间为1000秒，如果不写则一直将1g文件分4k每次写完为止
ioengine=psync           io引擎使用pync方式
group_reporting          关于显示结果的，汇总每个进程的信息

lockmem=1g               只使用1g内存进行测试
zero_buffers             用0初始化系统buffer
nrfiles=8                每个进程生成文件的数量
rwmixwrite=30            在混合读写的模式下，写占30%

1）顺序读示例

fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=1G -numjobs=4 -runtime=1000 -group_reporting -name=mytest

​ 返回结果：

[root@controller ~]# fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=1G -numjobs=4 -runtime=1000 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 4 threads
Jobs: 1 (f=1): [R(1),_(3)][100.0%][r=21.4MiB/s,w=0KiB/s][r=5475,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=4): err= 0: pid=2830310: Tue Jan  3 11:12:58 2023
   read: IOPS=8562, BW=33.4MiB/s (35.1MB/s)(4096MiB/122455msec)
    clat (usec): min=200, max=80336, avg=463.72, stdev=513.63
     lat (usec): min=200, max=80336, avg=463.84, stdev=513.64
    clat percentiles (usec):
     |  1.00th=[  265],  5.00th=[  285], 10.00th=[  302], 20.00th=[  318],
     | 30.00th=[  334], 40.00th=[  351], 50.00th=[  371], 60.00th=[  392],
     | 70.00th=[  420], 80.00th=[  465], 90.00th=[  586], 95.00th=[  848],
     | 99.00th=[ 2343], 99.50th=[ 3032], 99.90th=[ 6456], 99.95th=[ 8717],
     | 99.99th=[14091]
   bw (  KiB/s): min= 4216, max=10944, per=25.12%, avg=8605.29, stdev=1153.54, samples=973
   iops        : min= 1054, max= 2736, avg=2151.32, stdev=288.39, samples=973
  lat (usec)   : 250=0.29%, 500=84.39%, 750=9.11%, 1000=2.27%
  lat (msec)   : 2=2.52%, 4=1.14%, 10=0.25%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.36%, sys=0.72%, ctx=1048596, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=33.4MiB/s (35.1MB/s), 33.4MiB/s-33.4MiB/s (35.1MB/s-35.1MB/s), io=4096MiB (4295MB), run=122455-122455msec

Disk stats (read/write):
  vdb: ios=1034729/436, merge=10932/4, ticks=473564/120, in_queue=15252, util=99.89%

2）随机混合读写示例

 fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=4k -size=1G -numjobs=30 -runtime=1000 -group_reporting -name=mytest

​ 返回结果：

[root@controller ~]# fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=4k -size=1G -numjobs=30 -runtime=1000 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 30 threads
Jobs: 6 (f=6): [m(2),_(1),m(1),_(2),m(1),_(4),E(1),_(1),m(1),_(6),m(1),_(9)][99.8%][r=16.7MiB/s,w=17.1MiB/s][r=4282,w=4372 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=30): err= 0: pid=2834861: Tue Jan  3 11:32:59 2023
   read: IOPS=7914, BW=30.9MiB/s (32.4MB/s)(14.0GiB/496774msec)
    clat (usec): min=240, max=215504, avg=1785.65, stdev=1555.16
     lat (usec): min=241, max=215504, avg=1785.77, stdev=1555.17
    clat percentiles (usec):
     |  1.00th=[  433],  5.00th=[  807], 10.00th=[  906], 20.00th=[ 1020],
     | 30.00th=[ 1106], 40.00th=[ 1221], 50.00th=[ 1336], 60.00th=[ 1483],
     | 70.00th=[ 1729], 80.00th=[ 2147], 90.00th=[ 3097], 95.00th=[ 4228],
     | 99.00th=[ 8225], 99.50th=[10814], 99.90th=[16712], 99.95th=[18744],
     | 99.99th=[24511]
   bw (  KiB/s): min=  360, max= 1856, per=3.34%, avg=1057.75, stdev=204.65, samples=29715
   iops        : min=   90, max=  464, avg=264.42, stdev=51.17, samples=29715
  write: IOPS=7916, BW=30.9MiB/s (32.4MB/s)(15.0GiB/496774msec)
    clat (usec): min=546, max=215138, avg=1991.94, stdev=1806.19
     lat (usec): min=546, max=215138, avg=1992.15, stdev=1806.19
    clat percentiles (usec):
     |  1.00th=[  775],  5.00th=[  881], 10.00th=[  947], 20.00th=[ 1057],
     | 30.00th=[ 1156], 40.00th=[ 1270], 50.00th=[ 1418], 60.00th=[ 1614],
     | 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3687], 95.00th=[ 5014],
     | 99.00th=[ 9110], 99.50th=[11207], 99.90th=[16057], 99.95th=[18482],
     | 99.99th=[25560]
   bw (  KiB/s): min=  352, max= 1736, per=3.34%, avg=1057.96, stdev=204.19, samples=29715
   iops        : min=   88, max=  434, avg=264.48, stdev=51.05, samples=29715
  lat (usec)   : 250=0.01%, 500=0.89%, 750=1.26%, 1000=14.38%
  lat (msec)   : 2=57.89%, 4=18.56%, 10=6.35%, 20=0.65%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.09%, sys=0.25%, ctx=7864440, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3931786,3932534,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=30.9MiB/s (32.4MB/s), 30.9MiB/s-30.9MiB/s (32.4MB/s-32.4MB/s), io=14.0GiB (16.1GB), run=496774-496774msec
  WRITE: bw=30.9MiB/s (32.4MB/s), 30.9MiB/s-30.9MiB/s (32.4MB/s-32.4MB/s), io=15.0GiB (16.1GB), run=496774-496774msec

Disk stats (read/write):
  vdb: ios=3931755/3934156, merge=4/20, ticks=6985720/7797261, in_queue=2774632, util=100.00%

结果说明：

bw：磁盘的吞吐量，顺序读写考察的重点，类似于下载速度。

iops：磁盘的每秒读写次数，随机读写考察的重点。

util：磁盘利用率。

1.3 其他——Fio

1. Fio文档

https://fio.readthedocs.io/en/latest/fio_doc.html

2. Fio源码仓库链接

https://brick.kernel.dk/snaps/

3. 详细参数

fio [options] [job options] <job file(s)>
  --debug=options       Enable debug logging. May be one/more of:
                        process,file,io,mem,blktrace,verify,random,parse,
                        diskutil,job,mutex,profile,time,net,rate,compress,
                        steadystate,helperthread
  --parse-only          Parse options only, don't start any IO
  --output              Write output to file
  --bandwidth-log       Generate aggregate bandwidth logs
  --minimal             Minimal (terse) output
  --output-format=type  Output format (terse,json,json+,normal)
  --terse-version=type  Set terse version output format (default 3, or 2 or 4)
  --version             Print version info and exit
  --help                Print this page
  --cpuclock-test       Perform test/validation of CPU clock
  --crctest=[type]      Test speed of checksum functions
  --cmdhelp=cmd         Print command help, "all" for all of them
  --enghelp=engine      Print ioengine help, or list available ioengines
  --enghelp=engine,cmd  Print help for an ioengine cmd
  --showcmd             Turn a job file into command line options
  --eta=when            When ETA estimate should be printed
                        May be "always", "never" or "auto"
  --eta-newline=time    Force a new line for every 'time' period passed
  --status-interval=t   Force full status dump every 't' period passed
  --readonly            Turn on safety read-only checks, preventing writes
  --section=name        Only run specified section in job file, multiple sections can be specified
  --alloc-size=kb       Set smalloc pool to this size in kb (def 16384)
  --warnings-fatal      Fio parser warnings are fatal
  --max-jobs=nr         Maximum number of threads/processes to support
  --server=args         Start a backend fio server
  --daemonize=pidfile   Background fio server, write pid to file
  --client=hostname     Talk to remote backend(s) fio server at hostname
  --remote-config=file  Tell fio server to load this local job file
  --idle-prof=option    Report cpu idleness on a system or percpu basis
                        (option=system,percpu) or run unit work
                        calibration only (option=calibrate)
  --inflate-log=log     Inflate and output compressed log
  --trigger-file=file   Execute trigger cmd when file exists
  --trigger-timeout=t   Execute trigger at this time
  --trigger=cmd         Set this command as local trigger
  --trigger-remote=cmd  Set this command as remote trigger
  --aux-path=path       Use this path for fio state generated files