目标：测试两个40g网卡对接 能否冲到40g的流量

                           测试报告

目标：测试两个40g网卡对接 能否冲到40g的流量

物理环境：

服务器A5.5.5.1------------40网卡--------5.5.5.2服务器B------172.31.1.228（可以上外网）---公司网络的2960

服务器A

硬件配置：

CPU:Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz

内存：8G (DDR3 1600 4G*2）

硬盘：128G SSD

主板信息：HP Z220 SFF Workstation

网卡：40G网卡 (Skyhawk)

软件环境：DEBIAN 8.6 升级到下面版本

root@debian:/home/andy# lsb_release -a

No LSB modules are available.

Distributor ID: Debian

Description:     Debian GNU/Linux 8.8 (jessie)

Release:  8.8

Codename:      jessie

root@debian:/home/andy#

服务器B

硬件配置：

CPU：Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz

内存：8G （DDR3 1600 2G*4）

硬盘：128G SSD

主板信息：HP Z220 SFF Workstation

网卡：40G网卡 (Skyhawk)

软件环境：debian8.4升级到下面版本

root@debian:/home/andy# lsb_release -a

No LSB modules are available.

Distributor ID: Debian

Description:     Debian GNU/Linux 8.8 (jessie)

Release:  8.8

Codename:      jessie

root@debian:/home/andy#

直接在服务器上show出来的

root@debian:/home/andy# inxi -Fx

System:    Host: debian Kernel: 3.16.0-4-amd64 x86_64 (64 bit gcc: 4.8.4) Console: tty 8

           Distro: Debian GNU/Linux 8

Machine:   System: Hewlett-Packard product: HP Z220 SFF Workstation serial: 6CR324WLCN

           Mobo: Hewlett-Packard model: 1791 serial: 6CR324WLCN

           Bios: Hewlett-Packard v: K51 v01.58 date: 02/07/2013

CPU:       Quad core Intel Core i5-3470 (-MCP-) cache: 6144 KB

           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 25541

           Clock Speeds: 1: 3595 MHz 2: 3565 MHz 3: 3440 MHz 4: 3496 MHz

Graphics:  Card: Intel Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller bus-ID: 00:02.0

           Display Server: N/A driver: N/A tty size: 224x62 Advanced Data: N/A for root out of X

Audio:     Card Intel 7 Series/C210 Series Family High Definition Audio Controller

           driver: snd_hda_intel bus-ID: 00:1b.0

           Sound: Advanced Linux Sound Architecture v: k3.16.0-4-amd64

Network:   Card-1: Emulex OneConnect NIC (Skyhawk) driver: be2net v: 10.2u port: Root bus-ID: 01:00.0

           IF: eth1 state: up speed: 40000 Mbps duplex: full mac: 00:90:fa:70:80:dc

           Card-2: Intel 82579LM Gigabit Network Connection driver: e1000e v: 2.3.2-k port: f080 bus-ID: 00:19.0

           IF: eth0 state: up speed: 1000 Mbps duplex: full mac: 88:51:fb:6f:c0:bc

Drives:    HDD Total Size: 128.0GB (4.8% used) ID-1: /dev/sda model: LITEONIT_LCS size: 128.0GB temp: 0C

Partition: ID-1: / size: 113G used: 1.1G (1%) fs: ext4 dev: /dev/sda1

           ID-2: swap-1 size: 5.23GB used: 0.00GB (0%) fs: swap dev: /dev/sda5

Sensors:   System Temperatures: cpu: 68.0C mobo: 27.8C

           Fan Speeds (in rpm): cpu: N/A

Info:      Processes: 131 Uptime: 5:25 Memory: 141.7/7877.9MB Init: systemd runlevel: 5 Gcc sys: N/A

           Client: Shell (bash 4.3.301) inxi: 2.1.28

root@debian:/home/andy#

方法2

root@debian:/home/andy# lshw -short

H/W path        Device     Class          Description

=====================================================

                           system         Computer

/0                         bus            Motherboard

/0/0                       memory         7877MiB System memory

/0/1                       processor      Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz

/0/100                     bridge         Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller

/0/100/1                   bridge         Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port

/0/100/1/0      eth1       network        OneConnect NIC (Skyhawk)

/0/100/1/0.1               bus            OneConnect FCoE Initiator (Skyhawk)

/0/100/2                   display        Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller

/0/100/14                  bus            7 Series/C210 Series Chipset Family USB xHCI Host Controller

/0/100/14/0     usb4       bus            xHCI Host Controller

/0/100/14/1     usb3       bus            xHCI Host Controller

/0/100/16                  communication  7 Series/C210 Series Chipset Family MEI Controller #1

/0/100/16.3                communication  7 Series/C210 Series Chipset Family KT Controller

/0/100/19       eth0       network        82579LM Gigabit Network Connection

/0/100/1a                  bus            7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2

/0/100/1a/1     usb1       bus            EHCI Host Controller

/0/100/1a/1/1              bus            Integrated Rate Matching Hub

/0/100/1b                  multimedia     7 Series/C210 Series Chipset Family High Definition Audio Controller

/0/100/1d                  bus            7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1

/0/100/1d/1     usb2       bus            EHCI Host Controller

/0/100/1d/1/1              bus            Integrated Rate Matching Hub

/0/100/1e                  bridge         82801 PCI Bridge

/0/100/1f                  bridge         C216 Series Chipset LPC Controller

/0/100/1f.2                storage        82801 SATA Controller [RAID mode]

/0/100/1f.3                bus            7 Series/C210 Series Chipset Family SMBus Controller

/0/2            scsi1      storage       

/0/2/0.0.0      /dev/sda   disk           128GB LITEONIT LCS-128

/0/2/0.0.0/1    /dev/sda1  volume         114GiB EXT4 volume

/0/2/0.0.0/2    /dev/sda2  volume         4986MiB Extended partition

/0/2/0.0.0/2/5  /dev/sda5  volume         4986MiB Linux swap / Solaris partition

root@debian:/home/andy#

安装完以上的基础环境后：

Step1：apt-get update 更新一下基础系统

Step2：两个服务器堵安装iperf3（版本iperf 3.1.3），安装办法如下：

sudo apt-get remove iperf3 libiperf0

wget https://iperf.fr/download/ubuntu/libiperf0_3.1.3-1_amd64.deb

wget https://iperf.fr/download/ubuntu/iperf3_3.1.3-1_amd64.deb

sudo dpkg -i libiperf0_3.1.3-1_amd64.deb iperf3_3.1.3-1_amd64.deb

rm libiperf0_3.1.3-1_amd64.deb iperf3_3.1.3-1_amd64.deb

Step 3：两个服务器安装一下ethtool_2.6.33.orig.tar.gz 安装办法如下：

tar -xvzf ethtool_2.6.33.orig.tar.gz

cd ethtool-2.6.33

./configure

make

    make install

    make clean

Step 4：优化rwnd 办法如下：能起到一定的cpu优化作用，但是影响不大，所以这个最好不做，应该是debian比较新的版本是有做了rwind的自动伸缩的了。

nano /etc/sysctl.conf

添加红色部分：

##############################################################3

# Functions previously found in netbase

#

net.ipv4.tcp_syn_retries = 1

net.ipv4.tcp_synack_retries = 1

net.ipv4.tcp_keepalive_time = 600

net.ipv4.tcp_keepalive_probes = 3

net.ipv4.tcp_keepalive_intvl =15

net.ipv4.tcp_retries2 = 5

net.ipv4.tcp_fin_timeout = 2

net.ipv4.tcp_max_tw_buckets = 36000

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_max_orphans = 32768

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_max_syn_backlog = 16384

net.ipv4.tcp_wmem = 8192 131072 16777216

net.ipv4.tcp_rmem = 327680 1310720 16777216000

net.ipv4.tcp_mem = 786432 1048576 1572864

net.ipv4.ip_local_port_range = 1024 65000

net.ipv4.ip_conntrack_max = 65536

net.ipv4.netfilter.ip_conntrack_max=65536

net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=180

net.core.somaxconn = 16384

net.core.netdev_max_backlog = 16384

net.ipv4.tcp_sack = 0

net.ipv4.tcp_moderate_rcvbuf = 1

（net.ipv4.tcp_fin_timeout = 15）

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)

# Turn on Source Address Verification in all interfaces to

# prevent some spoofing attacks

#net.ipv4.conf.default.rp_filter=1

#net.ipv4.conf.all.rp_filter=1

然后让这些起到效果：sysctl -p

Step4：优化 cwnd----修改这个没有什么意义，甚至可能有反作用。

shell> ip route | while read r; do

           ip route change $r initcwnd 10;

       done

Step 5：修过两个网卡的mtu值-修过这个可以让流量放大 效果最明显的其中的一个因素，

     办法如下： ifconfig eth1 mtu 90000

Step 5：两个服务器同时，优化interface接收包的能力

 ethtool -K eth1 tx on

ethtool -K eth1 rx on

Step6：再做一下简单的优化 详细可以参考 http://blog.csdn.net/an_zhenwei/article/details/18732081

      ethtool -K eth1 sg on

      ethtool -K eth1 tso on

   ethtool -K eth1 gso on

可以看一下修改的情况：

root@debian:/home/andy# ethtool -k eth0

Features for eth0:

rx-checksumming: on

tx-checksumming: on

         tx-checksum-ipv4: off [fixed]

         tx-checksum-ip-generic: on

         tx-checksum-ipv6: off [fixed]

         tx-checksum-fcoe-crc: off [fixed]

         tx-checksum-sctp: off [fixed]

scatter-gather: on

         tx-scatter-gather: on

         tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

         tx-tcp-segmentation: on

         tx-tcp-ecn-segmentation: off [fixed]

         tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

tx-mpls-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off

rx-all: off

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

busy-poll: off [fixed]

root@debian:/home/andy#

Step 7：内核优化

  sysctl -w net.core.rmem_max=16777216

Step 8：关闭timesstamps 可以节省cpu开销

Sysctl -w net.ipv4.tcp_timestamps=0

Step 8:基本上做上面的优化

Udp测试

服务器 5.5.5.1 运行iperf -s

客户端 5.5.5.1 运行 iperf3 -c 5.5.5.1   -t 30000   -P 6  -M 8900 -u -b 60000m 

会有以下的输出结果

 以上的结构是基本达标的，谢谢。

二次测试验证情况：

（外网ip172.31.1.182）服务器A--5.5.5.1----------5.5.5.2-------服务器B（外网ip172.31.1.228）

Step 1：基于这样的情况下

硬件还是不变的，两边都安装玩debian8.6后

apt-get update

apt-get upgrade

Step 2:还是按照上面的办法安装好iperf3

      再安装apt-get install  ethtool 这个主要是看那个网卡是40g网卡用

      apt-get install nload 看网卡流量用的

Step 3：没有修改任何东西的情况下

服务器A iperf3 -s

      服务器B iperf3 -c 5.5.5.1   -t 30000   -P 6  -M 8900 -u -b 60000m 

 这样流量可以到达大约 15G

Step 4： 我修改了两个服务器40g网卡的mtu 修改到9000

        Ifconfig eth1 mtu 9000

        重复step3 然后竟然发现流量也可以达到 38g

Nload 看也有 36g

       但是服务器b就是客户端服务器cpu有点高：

         服务器a感觉还好

Step 5：对debian进行系统优化：

做了截图上的优化后，服务器b 明显滴从接近100%到 80% 只是优化了服务器b

同样去优化服务器a的时候：cpu也稍有降低一些了 78%---73%

结论：就是安装好debian8.6后，各自升级到最新，安装好要用的测试软件 iperf3 ethtool nload

接着就是修改MTU测试就好，在debian环境下，tcp窗口可以自动伸缩，其他窗口也尽量不用动。只是这样情况下cpu有点高。

修改cpu有效的办法如下就好：-其实就是上面的二次测试情况流程