卷积核对精度的影响

在cnn中单个layer的不同卷积核对精度的影响到底是怎样的？在这里做下实验来对比下。

1       实验结论

• 不同卷积核对acc的影响有差别，有的卷积核提取的特征对模型的表现具有关键的作用。
• L1/L2、TVLOSS对卷积核的度量结果和卷积核的重要性在有的层有强的相关性，在有的层没有必然的联系。
• 干掉某些卷积核有时反而会提高模型的精度。

2       实验方法

我们首先训练一个用来识别cifar10的resnet50，精度为95.5%，然后对该模型某一层的不同卷积核分别进行mask，然后看下屏蔽后的准确率，从而对比同一层不同卷积核对acc影响。

3       实验1：第一个bottleneck的conv1

3.1       通道屏蔽后acc

屏蔽对应通道后的acc值：acc_list [95.51, 95.51, 95.51, 95.53, 95.51, 95.28, 94.55, 95.51, 95.51, 95.51, 95.37, 95.51, 95.51, 95.51, 95.51, 95.39, 95.49, 95.51, 95.51, 95.51, 95.46, 95.51, 95.51, 95.51, 95.5, 95.52, 95.5, 95.53, 95.51, 95.51, 95.51, 95.5, 95.51, 95.53, 95.51, 95.51, 95.53, 95.51, 95.51, 95.3, 95.32, 95.42, 95.5, 95.45, 95.51, 95.51, 95.42, 95.51, 95.51, 95.33, 95.51, 95.51, 95.53, 95.5, 95.45, 95.37, 95.5, 95.51, 95.46, 95.5, 95.51, 95.48, 95.51, 95.51]

可以看到通道6对acc影响最大。

3.2       通道对应的l1值

L1:

tensor([9.0696e-04, 1.0765e-04, 3.6797e-05, 4.2372e-01, 7.7838e-07, 8.8848e-01,

        1.9568e+00, 9.2843e-07, 2.7821e-01, 2.7539e-01, 8.8536e-01, 2.5309e-06,

        1.0610e-05, 1.2782e-06, 5.0379e-07, 9.6642e-01, 3.8845e-01, 8.9877e-01,

        1.5160e-07, 3.0052e-06, 1.4623e+00, 7.2491e-04, 6.5939e-07, 2.5264e-01,

        9.5067e-01, 3.6332e-01, 1.3800e-01, 3.1673e-01, 1.8118e-06, 1.8608e-03,

        2.7195e-03, 1.9264e-01, 3.8799e-03, 6.4787e-01, 9.9888e-08, 2.5409e-01,

        2.8853e-01, 3.8932e-08, 2.8015e-06, 9.2283e-01, 1.7676e+00, 1.3977e+00,

        1.7551e-01, 1.1330e+00, 6.7971e-05, 7.0355e-06, 9.4906e-01, 8.9619e-01,

        3.0095e-11, 9.8106e-01, 5.0086e-06, 3.4246e-07, 6.6274e-01, 2.7919e-01,

        1.1153e+00, 1.1972e+00, 3.0969e-01, 4.7030e-06, 1.1568e+00, 5.6171e-01,

        1.0998e-06, 1.0449e+00, 1.8734e-07, 7.6779e-07])

3.3       通道对应的l2值

L2：

tensor([2.3542e-04, 2.5970e-05, 8.3133e-06, 1.0720e-01, 1.4336e-07, 3.6386e-01,

        4.6973e-01, 2.4229e-07, 7.7052e-02, 9.4308e-02, 2.8199e-01, 5.7094e-07,

        2.6479e-06, 2.3814e-07, 1.3788e-07, 3.4913e-01, 9.5818e-02, 2.0528e-01,

        3.9279e-08, 8.1699e-07, 3.1909e-01, 1.7411e-04, 1.0670e-07, 6.5193e-02,

        2.6681e-01, 9.8394e-02, 4.1757e-02, 1.0555e-01, 3.2757e-07, 3.9119e-04,

        6.7353e-04, 7.6015e-02, 9.2085e-04, 1.7074e-01, 1.7432e-08, 7.0176e-02,

        9.6610e-02, 6.8448e-09, 5.2183e-07, 2.6660e-01, 3.9182e-01, 3.4121e-01,

        5.5723e-02, 2.7750e-01, 1.8330e-05, 1.7744e-06, 2.7980e-01, 2.4811e-01,

        3.8367e-12, 3.9042e-01, 9.5567e-07, 6.5733e-08, 1.7807e-01, 1.0319e-01,

        2.9599e-01, 3.5718e-01, 8.8296e-02, 1.0225e-06, 2.6082e-01, 1.4984e-01,

        2.1402e-07, 3.0009e-01, 3.5059e-08, 1.3561e-07])

3.4       acc与L1/L2的相关系数

4       实验2：第一个bottleneck的conv2

4.1       通道屏蔽后acc

屏蔽对应通道后的acc值：acc_list [95.0, 95.52, 95.28, 95.52, 95.51, 95.52, 95.51, 95.53, 95.48, 95.51, 95.54, 95.37, 95.48, 95.52, 94.47, 95.36, 95.45, 95.51, 95.52, 95.52, 95.52, 95.38, 95.51, 95.47, 95.54, 95.59, 95.41, 95.53, 95.51, 95.43, 95.51, 95.53, 95.53, 94.98, 95.41, 95.51, 95.51, 95.53, 95.52, 95.53, 95.49, 95.53, 95.56, 95.51, 95.54, 95.44, 95.5, 95.53, 95.51, 95.52, 95.51, 95.23, 95.51, 95.44, 95.51, 95.48, 95.38, 95.42, 95.45, 95.51, 95.36, 95.53, 95.54, 95.54]

可以看到通道14对acc影响最大。

4.2       通道对应的l1值

L1:

tensor([1.0716e+01, 1.9086e+00, 7.8853e+00, 1.5622e+00, 2.9861e-10, 3.3352e-01,

        3.9368e-01, 1.2408e+00, 3.3717e+00, 1.2252e+00, 1.1791e+00, 6.6897e+00,

        3.0702e+00, 7.9634e+00, 1.1525e+01, 7.6635e+00, 5.9539e+00, 1.3473e+00,

        3.3368e+00, 1.5067e+00, 1.3881e+00, 7.4053e+00, 1.4943e-10, 3.9277e+00,

        2.0258e+00, 4.7269e+00, 5.0097e+00, 1.7124e+00, 3.2214e+00, 4.2788e+00,

        1.0224e+00, 1.5025e+00, 1.8395e+00, 1.0824e+01, 5.7857e+00, 5.5878e-01,

        6.1386e-01, 6.2127e-01, 4.0590e-01, 4.8125e+00, 3.0765e+00, 7.0582e-01,

        5.2847e+00, 1.1769e+00, 1.8588e+00, 9.6256e+00, 6.7625e-01, 8.4935e-01,

        5.9225e-01, 4.5970e-01, 2.7215e-01, 8.8173e+00, 3.6089e-01, 5.7669e+00,

        4.8017e-01, 2.3523e+00, 4.8038e+00, 4.5273e+00, 4.7753e+00, 1.0642e-05,

        7.3642e+00, 1.5959e+00, 1.4231e+00, 3.0602e+00])

4.3       通道对应的l2值

L2：

tensor([1.0794e+00, 1.6398e-01, 7.5314e-01, 1.3647e-01, 1.2861e-11, 2.8600e-02,

        3.4727e-02, 1.1088e-01, 2.8136e-01, 1.0280e-01, 1.0981e-01, 6.6187e-01,

        2.8738e-01, 7.1218e-01, 1.2429e+00, 7.5780e-01, 5.6656e-01, 1.1207e-01,

        3.0706e-01, 1.2832e-01, 1.2030e-01, 6.5510e-01, 7.5710e-12, 3.6322e-01,

        1.7983e-01, 4.1302e-01, 4.6616e-01, 1.4823e-01, 2.9193e-01, 3.5102e-01,

        8.8890e-02, 1.2920e-01, 1.6741e-01, 1.0602e+00, 5.3704e-01, 4.8062e-02,

        5.1057e-02, 4.8576e-02, 3.3148e-02, 4.7651e-01, 2.9394e-01, 6.4850e-02,

        5.5513e-01, 1.0006e-01, 1.6000e-01, 8.7806e-01, 5.8017e-02, 7.2040e-02,

        4.9933e-02, 3.8316e-02, 2.4285e-02, 8.2778e-01, 3.5021e-02, 5.0688e-01,

        4.2325e-02, 2.4359e-01, 5.2677e-01, 3.6967e-01, 5.0588e-01, 7.6541e-07,

        7.4116e-01, 1.5700e-01, 1.2355e-01, 2.8397e-01])

4.4       通道对应的tvloss值

tvloss:

tensor([8.1483e-03, 2.3616e-04, 1.3290e-02, 1.0702e-04, 2.0741e-25, 7.1450e-06,

        7.8319e-06, 8.3272e-05, 7.4236e-04, 5.7573e-05, 8.3458e-05, 3.3662e-03,

        7.1745e-04, 5.0785e-03, 9.8108e-03, 1.1484e-02, 4.0307e-03, 8.2193e-05,

        9.7639e-04, 1.4180e-04, 8.1392e-05, 5.2845e-03, 1.9715e-25, 1.1177e-03,

        1.9397e-04, 1.3477e-03, 2.2037e-03, 1.7273e-04, 6.1949e-04, 1.7323e-03,

        4.0396e-05, 1.0451e-04, 2.6899e-04, 8.6072e-03, 4.3876e-03, 1.3627e-05,

        2.0766e-05, 1.2660e-05, 4.2458e-06, 2.0336e-03, 7.7894e-04, 3.2793e-05,

        2.0875e-03, 7.5620e-05, 1.6514e-04, 6.5552e-03, 1.4968e-05, 3.2309e-05,

        1.5248e-05, 7.3420e-06, 3.7719e-06, 1.6258e-02, 8.7880e-06, 2.7667e-03,

        1.1910e-05, 5.1394e-04, 2.3415e-03, 2.0368e-03, 2.6388e-03, 3.8714e-15,

        1.1320e-02, 1.8265e-04, 8.7671e-05, 8.7602e-04])

4.5       acc与L1/L2/TVLOSS的相关系数

5       实验3：第二个bottleneck的conv2

5.1       通道屏蔽后acc

屏蔽对应通道后的acc值：acc_list [95.52, 95.52, 95.55, 95.49, 95.55, 95.58, 95.47, 95.46, 95.52, 95.35, 95.5, 95.44, 95.48, 95.56, 95.45, 95.51, 95.44, 95.48, 95.38, 95.56, 95.47, 95.5, 95.49, 95.45, 95.54, 95.53, 95.52, 95.59, 95.49, 95.5, 95.57, 95.51, 95.45, 95.54, 95.49, 95.44, 95.5, 95.51, 95.49, 95.5, 95.49, 95.56, 95.38, 95.5, 95.55, 95.42, 95.52, 95.53, 95.42, 95.55, 95.53, 95.52, 95.51, 95.51, 95.52, 95.44, 95.48, 95.47, 95.52, 95.49, 95.54, 95.5, 95.52, 95.5, 95.58, 95.48, 95.45, 95.52, 95.48, 95.55, 95.53, 95.52, 95.51, 95.47, 95.48, 95.5, 95.52, 95.44, 95.56, 95.54, 95.29, 95.38, 95.52, 95.52, 95.5, 95.48, 95.47, 95.51, 95.47, 95.53, 95.52, 95.45, 95.52, 95.45, 95.48, 95.52, 95.57, 95.47, 95.47, 95.38, 95.5, 95.49, 95.52, 95.51, 95.57, 95.53, 95.35, 95.5, 95.52, 95.51, 95.48, 95.52, 95.52, 95.5, 95.48, 95.49, 95.5, 95.51, 95.5, 95.51, 95.45, 95.47, 95.49, 95.45, 95.5, 95.44, 95.47, 95.52]

5.2       通道对应的l1值

L1:

tensor([ 4.5902,  6.1007,  8.4471, 10.3940,  8.8796, 10.2521,  8.8904, 11.2981,

         7.7232, 10.9551,  6.9816,  7.7976,  4.8660,  5.8874,  7.0002, 12.3415,

        12.4564,  8.8947, 11.7549,  7.1401, 10.7795,  8.5514,  8.6884, 10.5477,

        11.3029,  4.1957,  3.4916, 11.5326, 10.0938, 11.7479,  7.2280,  2.8732,

         9.7092,  6.0612, 10.1461, 11.8868,  2.5059,  6.2565, 10.5787,  8.4110,

         7.2470, 10.0240, 12.8911, 11.3949, 12.4613, 11.1628,  8.8561,  9.5457,

        11.0646,  6.2736,  3.9552, 10.0440,  9.3584,  9.7429,  7.9220,  7.1326,

         8.5418,  9.6327,  4.4188, 11.4901,  9.0462,  8.6284,  2.9410,  9.5523,

         7.7476,  8.5597, 13.2373,  4.7582, 11.9483,  5.9163,  4.8952,  5.6138,

         9.9418, 10.6109, 11.1282, 12.7357, 11.0690, 10.4473, 13.2904,  9.9944,

        17.0782, 10.3416,  6.9438,  6.2926, 11.2201,  6.1278, 12.1820, 11.8296,

         9.9560,  4.8155,  8.0361,  9.2518, 10.4862, 12.2798, 10.8676,  6.2806,

        11.2317, 12.0288,  9.9416,  9.9810,  8.7164,  9.4495,  9.4815,  9.4012,

        10.4839,  7.1395, 14.7499, 10.6155,  7.0444, 11.4811, 11.3775, 13.4971,

         7.6662, 10.0595, 11.8670,  7.0470,  9.9963,  8.3277, 12.4580, 10.9683,

         9.2675,  8.0764,  5.4207, 11.0078,  9.7833, 12.9759,  6.2853,  8.8741])

5.3       通道对应的l2值

L2：

tensor([0.1787, 0.2391, 0.3342, 0.3921, 0.3440, 0.4416, 0.3498, 0.4368, 0.3053,

        0.4745, 0.2787, 0.3175, 0.1888, 0.2498, 0.2963, 0.5002, 0.4961, 0.3563,

        0.5600, 0.2841, 0.4305, 0.3648, 0.3448, 0.4190, 0.4396, 0.1705, 0.1376,

        0.4672, 0.4020, 0.4780, 0.2914, 0.1115, 0.3916, 0.2377, 0.3976, 0.4800,

        0.0977, 0.2469, 0.4925, 0.3215, 0.2989, 0.3953, 0.5386, 0.4496, 0.4931,

        0.4462, 0.3367, 0.3745, 0.4365, 0.2507, 0.1566, 0.3804, 0.3654, 0.3888,

        0.3183, 0.2892, 0.3305, 0.3884, 0.1721, 0.5207, 0.3797, 0.3375, 0.1169,

        0.3770, 0.3088, 0.3439, 0.5457, 0.1945, 0.4879, 0.2408, 0.1922, 0.2234,

        0.4317, 0.4117, 0.4434, 0.5047, 0.4493, 0.4560, 0.5692, 0.3967, 0.6973,

        0.4085, 0.2812, 0.2542, 0.4401, 0.2411, 0.5101, 0.4650, 0.3938, 0.1905,

        0.3086, 0.3571, 0.4044, 0.4873, 0.4430, 0.2432, 0.4442, 0.5425, 0.3944,

        0.3935, 0.3775, 0.4198, 0.3633, 0.3755, 0.4250, 0.2776, 0.6163, 0.4177,

        0.2753, 0.4480, 0.4460, 0.5406, 0.2966, 0.3973, 0.4547, 0.2785, 0.3854,

        0.3280, 0.4983, 0.4521, 0.3613, 0.3052, 0.2156, 0.4326, 0.4208, 0.5252,

        0.2488, 0.3467])

5.4       通道对应的tvloss值

tvloss:

tensor([1.3992e-04, 2.2695e-04, 2.7250e-04, 7.6327e-04, 7.5144e-04, 8.7256e-04,

        5.5133e-04, 1.3989e-03, 5.4804e-04, 9.4830e-04, 2.1857e-04, 1.9901e-04,

        1.4296e-04, 2.0737e-04, 1.4010e-04, 1.7853e-03, 1.1067e-03, 9.6182e-04,

        1.5229e-03, 1.6583e-04, 8.5500e-04, 5.1329e-04, 5.4607e-04, 6.2221e-04,

        9.3564e-04, 1.2324e-04, 5.7210e-05, 1.0581e-03, 1.1646e-03, 9.6945e-04,

        4.2071e-04, 4.5595e-05, 8.3459e-04, 1.9988e-04, 1.0124e-03, 1.0173e-03,

        3.6465e-05, 2.9829e-04, 9.4978e-04, 6.5539e-04, 2.3278e-04, 6.3994e-04,

        1.5582e-03, 1.3657e-03, 1.5396e-03, 9.7195e-04, 6.6808e-04, 6.2939e-04,

        7.5327e-04, 3.2658e-04, 8.0546e-05, 7.0369e-04, 6.4445e-04, 1.1580e-03,

        3.6162e-04, 2.3316e-04, 6.8859e-04, 9.7655e-04, 1.4548e-04, 1.9952e-03,

        7.2628e-04, 7.0566e-04, 5.2536e-05, 5.8076e-04, 3.9151e-04, 7.9537e-04,

        1.4251e-03, 1.5686e-04, 1.7316e-03, 2.2931e-04, 1.7838e-04, 3.2831e-04,

        4.5192e-04, 7.3333e-04, 1.3144e-03, 1.2641e-03, 1.4423e-03, 5.2904e-04,

        1.7992e-03, 1.0000e-03, 1.9789e-03, 6.4655e-04, 3.7544e-04, 2.1733e-04,

        9.0644e-04, 2.8223e-04, 1.7986e-03, 1.7908e-03, 6.4259e-04, 1.3189e-04,

        4.1928e-04, 5.2752e-04, 6.4312e-04, 9.1314e-04, 1.5107e-03, 3.0553e-04,

        1.3613e-03, 2.2215e-03, 3.5164e-04, 6.3454e-04, 6.5172e-04, 3.5056e-04,

        8.1161e-04, 3.3546e-04, 1.2511e-03, 2.8639e-04, 1.5900e-03, 8.1033e-04,

        3.9007e-04, 1.5212e-03, 1.4268e-03, 2.2759e-03, 4.3843e-04, 1.0428e-03,

        1.5551e-03, 3.1775e-04, 1.0363e-03, 8.3562e-04, 2.0071e-03, 8.1407e-04,

        4.5609e-04, 3.5050e-04, 2.5387e-04, 6.7972e-04, 3.7846e-04, 1.3381e-03,

        2.8636e-04, 4.4538e-04])

5.5       acc与L1/L2/TVLOSS的相关系数

6       TODO

resnet50对于cifar10来说参数偏多，对于imagenet是否有类似的结论？

    能否找到合适的度量来评估卷积核的重要性，而不是通过现在这种逐次测试准确率的方法。