卷积核对精度的影响
卷积核对精度的影响
在cnn中单个layer的不同卷积核对精度的影响到底是怎样的?在这里做下实验来对比下。
1 实验结论
- 不同卷积核对acc的影响有差别,有的卷积核提取的特征对模型的表现具有关键的作用。
- L1/L2、TVLOSS对卷积核的度量结果和卷积核的重要性在有的层有强的相关性,在有的层没有必然的联系。
- 干掉某些卷积核有时反而会提高模型的精度。
2 实验方法
我们首先训练一个用来识别cifar10的resnet50,精度为95.5%,然后对该模型某一层的不同卷积核分别进行mask,然后看下屏蔽后的准确率,从而对比同一层不同卷积核对acc影响。
3 实验1:第一个bottleneck的conv1
3.1 通道屏蔽后acc
屏蔽对应通道后的acc值:acc_list [95.51, 95.51, 95.51, 95.53, 95.51, 95.28, 94.55, 95.51, 95.51, 95.51, 95.37, 95.51, 95.51, 95.51, 95.51, 95.39, 95.49, 95.51, 95.51, 95.51, 95.46, 95.51, 95.51, 95.51, 95.5, 95.52, 95.5, 95.53, 95.51, 95.51, 95.51, 95.5, 95.51, 95.53, 95.51, 95.51, 95.53, 95.51, 95.51, 95.3, 95.32, 95.42, 95.5, 95.45, 95.51, 95.51, 95.42, 95.51, 95.51, 95.33, 95.51, 95.51, 95.53, 95.5, 95.45, 95.37, 95.5, 95.51, 95.46, 95.5, 95.51, 95.48, 95.51, 95.51]
可以看到通道6对acc影响最大。
3.2 通道对应的l1值
L1:
tensor([9.0696e-04, 1.0765e-04, 3.6797e-05, 4.2372e-01, 7.7838e-07, 8.8848e-01,
1.9568e+00, 9.2843e-07, 2.7821e-01, 2.7539e-01, 8.8536e-01, 2.5309e-06,
1.0610e-05, 1.2782e-06, 5.0379e-07, 9.6642e-01, 3.8845e-01, 8.9877e-01,
1.5160e-07, 3.0052e-06, 1.4623e+00, 7.2491e-04, 6.5939e-07, 2.5264e-01,
9.5067e-01, 3.6332e-01, 1.3800e-01, 3.1673e-01, 1.8118e-06, 1.8608e-03,
2.7195e-03, 1.9264e-01, 3.8799e-03, 6.4787e-01, 9.9888e-08, 2.5409e-01,
2.8853e-01, 3.8932e-08, 2.8015e-06, 9.2283e-01, 1.7676e+00, 1.3977e+00,
1.7551e-01, 1.1330e+00, 6.7971e-05, 7.0355e-06, 9.4906e-01, 8.9619e-01,
3.0095e-11, 9.8106e-01, 5.0086e-06, 3.4246e-07, 6.6274e-01, 2.7919e-01,
1.1153e+00, 1.1972e+00, 3.0969e-01, 4.7030e-06, 1.1568e+00, 5.6171e-01,
1.0998e-06, 1.0449e+00, 1.8734e-07, 7.6779e-07])
3.3 通道对应的l2值
L2:
tensor([2.3542e-04, 2.5970e-05, 8.3133e-06, 1.0720e-01, 1.4336e-07, 3.6386e-01,
4.6973e-01, 2.4229e-07, 7.7052e-02, 9.4308e-02, 2.8199e-01, 5.7094e-07,
2.6479e-06, 2.3814e-07, 1.3788e-07, 3.4913e-01, 9.5818e-02, 2.0528e-01,
3.9279e-08, 8.1699e-07, 3.1909e-01, 1.7411e-04, 1.0670e-07, 6.5193e-02,
2.6681e-01, 9.8394e-02, 4.1757e-02, 1.0555e-01, 3.2757e-07, 3.9119e-04,
6.7353e-04, 7.6015e-02, 9.2085e-04, 1.7074e-01, 1.7432e-08, 7.0176e-02,
9.6610e-02, 6.8448e-09, 5.2183e-07, 2.6660e-01, 3.9182e-01, 3.4121e-01,
5.5723e-02, 2.7750e-01, 1.8330e-05, 1.7744e-06, 2.7980e-01, 2.4811e-01,
3.8367e-12, 3.9042e-01, 9.5567e-07, 6.5733e-08, 1.7807e-01, 1.0319e-01,
2.9599e-01, 3.5718e-01, 8.8296e-02, 1.0225e-06, 2.6082e-01, 1.4984e-01,
2.1402e-07, 3.0009e-01, 3.5059e-08, 1.3561e-07])
3.4 acc与L1/L2的相关系数
|
L1 |
L2 |
相关系数 |
-0.622 |
-0.6137 |
4 实验2:第一个bottleneck的conv2
4.1 通道屏蔽后acc
屏蔽对应通道后的acc值:acc_list [95.0, 95.52, 95.28, 95.52, 95.51, 95.52, 95.51, 95.53, 95.48, 95.51, 95.54, 95.37, 95.48, 95.52, 94.47, 95.36, 95.45, 95.51, 95.52, 95.52, 95.52, 95.38, 95.51, 95.47, 95.54, 95.59, 95.41, 95.53, 95.51, 95.43, 95.51, 95.53, 95.53, 94.98, 95.41, 95.51, 95.51, 95.53, 95.52, 95.53, 95.49, 95.53, 95.56, 95.51, 95.54, 95.44, 95.5, 95.53, 95.51, 95.52, 95.51, 95.23, 95.51, 95.44, 95.51, 95.48, 95.38, 95.42, 95.45, 95.51, 95.36, 95.53, 95.54, 95.54]
可以看到通道14对acc影响最大。
4.2 通道对应的l1值
L1:
tensor([1.0716e+01, 1.9086e+00, 7.8853e+00, 1.5622e+00, 2.9861e-10, 3.3352e-01,
3.9368e-01, 1.2408e+00, 3.3717e+00, 1.2252e+00, 1.1791e+00, 6.6897e+00,
3.0702e+00, 7.9634e+00, 1.1525e+01, 7.6635e+00, 5.9539e+00, 1.3473e+00,
3.3368e+00, 1.5067e+00, 1.3881e+00, 7.4053e+00, 1.4943e-10, 3.9277e+00,
2.0258e+00, 4.7269e+00, 5.0097e+00, 1.7124e+00, 3.2214e+00, 4.2788e+00,
1.0224e+00, 1.5025e+00, 1.8395e+00, 1.0824e+01, 5.7857e+00, 5.5878e-01,
6.1386e-01, 6.2127e-01, 4.0590e-01, 4.8125e+00, 3.0765e+00, 7.0582e-01,
5.2847e+00, 1.1769e+00, 1.8588e+00, 9.6256e+00, 6.7625e-01, 8.4935e-01,
5.9225e-01, 4.5970e-01, 2.7215e-01, 8.8173e+00, 3.6089e-01, 5.7669e+00,
4.8017e-01, 2.3523e+00, 4.8038e+00, 4.5273e+00, 4.7753e+00, 1.0642e-05,
7.3642e+00, 1.5959e+00, 1.4231e+00, 3.0602e+00])
4.3 通道对应的l2值
L2:
tensor([1.0794e+00, 1.6398e-01, 7.5314e-01, 1.3647e-01, 1.2861e-11, 2.8600e-02,
3.4727e-02, 1.1088e-01, 2.8136e-01, 1.0280e-01, 1.0981e-01, 6.6187e-01,
2.8738e-01, 7.1218e-01, 1.2429e+00, 7.5780e-01, 5.6656e-01, 1.1207e-01,
3.0706e-01, 1.2832e-01, 1.2030e-01, 6.5510e-01, 7.5710e-12, 3.6322e-01,
1.7983e-01, 4.1302e-01, 4.6616e-01, 1.4823e-01, 2.9193e-01, 3.5102e-01,
8.8890e-02, 1.2920e-01, 1.6741e-01, 1.0602e+00, 5.3704e-01, 4.8062e-02,
5.1057e-02, 4.8576e-02, 3.3148e-02, 4.7651e-01, 2.9394e-01, 6.4850e-02,
5.5513e-01, 1.0006e-01, 1.6000e-01, 8.7806e-01, 5.8017e-02, 7.2040e-02,
4.9933e-02, 3.8316e-02, 2.4285e-02, 8.2778e-01, 3.5021e-02, 5.0688e-01,
4.2325e-02, 2.4359e-01, 5.2677e-01, 3.6967e-01, 5.0588e-01, 7.6541e-07,
7.4116e-01, 1.5700e-01, 1.2355e-01, 2.8397e-01])
4.4 通道对应的tvloss值
tvloss:
tensor([8.1483e-03, 2.3616e-04, 1.3290e-02, 1.0702e-04, 2.0741e-25, 7.1450e-06,
7.8319e-06, 8.3272e-05, 7.4236e-04, 5.7573e-05, 8.3458e-05, 3.3662e-03,
7.1745e-04, 5.0785e-03, 9.8108e-03, 1.1484e-02, 4.0307e-03, 8.2193e-05,
9.7639e-04, 1.4180e-04, 8.1392e-05, 5.2845e-03, 1.9715e-25, 1.1177e-03,
1.9397e-04, 1.3477e-03, 2.2037e-03, 1.7273e-04, 6.1949e-04, 1.7323e-03,
4.0396e-05, 1.0451e-04, 2.6899e-04, 8.6072e-03, 4.3876e-03, 1.3627e-05,
2.0766e-05, 1.2660e-05, 4.2458e-06, 2.0336e-03, 7.7894e-04, 3.2793e-05,
2.0875e-03, 7.5620e-05, 1.6514e-04, 6.5552e-03, 1.4968e-05, 3.2309e-05,
1.5248e-05, 7.3420e-06, 3.7719e-06, 1.6258e-02, 8.7880e-06, 2.7667e-03,
1.1910e-05, 5.1394e-04, 2.3415e-03, 2.0368e-03, 2.6388e-03, 3.8714e-15,
1.1320e-02, 1.8265e-04, 8.7671e-05, 8.7602e-04])
4.5 acc与L1/L2/TVLOSS的相关系数
|
L1 |
L2 |
Tvloss |
相关系数 |
-0.7262 |
-0.764 |
-0.6732 |
5 实验3:第二个bottleneck的conv2
5.1 通道屏蔽后acc
屏蔽对应通道后的acc值:acc_list [95.52, 95.52, 95.55, 95.49, 95.55, 95.58, 95.47, 95.46, 95.52, 95.35, 95.5, 95.44, 95.48, 95.56, 95.45, 95.51, 95.44, 95.48, 95.38, 95.56, 95.47, 95.5, 95.49, 95.45, 95.54, 95.53, 95.52, 95.59, 95.49, 95.5, 95.57, 95.51, 95.45, 95.54, 95.49, 95.44, 95.5, 95.51, 95.49, 95.5, 95.49, 95.56, 95.38, 95.5, 95.55, 95.42, 95.52, 95.53, 95.42, 95.55, 95.53, 95.52, 95.51, 95.51, 95.52, 95.44, 95.48, 95.47, 95.52, 95.49, 95.54, 95.5, 95.52, 95.5, 95.58, 95.48, 95.45, 95.52, 95.48, 95.55, 95.53, 95.52, 95.51, 95.47, 95.48, 95.5, 95.52, 95.44, 95.56, 95.54, 95.29, 95.38, 95.52, 95.52, 95.5, 95.48, 95.47, 95.51, 95.47, 95.53, 95.52, 95.45, 95.52, 95.45, 95.48, 95.52, 95.57, 95.47, 95.47, 95.38, 95.5, 95.49, 95.52, 95.51, 95.57, 95.53, 95.35, 95.5, 95.52, 95.51, 95.48, 95.52, 95.52, 95.5, 95.48, 95.49, 95.5, 95.51, 95.5, 95.51, 95.45, 95.47, 95.49, 95.45, 95.5, 95.44, 95.47, 95.52]
5.2 通道对应的l1值
L1:
tensor([ 4.5902, 6.1007, 8.4471, 10.3940, 8.8796, 10.2521, 8.8904, 11.2981,
7.7232, 10.9551, 6.9816, 7.7976, 4.8660, 5.8874, 7.0002, 12.3415,
12.4564, 8.8947, 11.7549, 7.1401, 10.7795, 8.5514, 8.6884, 10.5477,
11.3029, 4.1957, 3.4916, 11.5326, 10.0938, 11.7479, 7.2280, 2.8732,
9.7092, 6.0612, 10.1461, 11.8868, 2.5059, 6.2565, 10.5787, 8.4110,
7.2470, 10.0240, 12.8911, 11.3949, 12.4613, 11.1628, 8.8561, 9.5457,
11.0646, 6.2736, 3.9552, 10.0440, 9.3584, 9.7429, 7.9220, 7.1326,
8.5418, 9.6327, 4.4188, 11.4901, 9.0462, 8.6284, 2.9410, 9.5523,
7.7476, 8.5597, 13.2373, 4.7582, 11.9483, 5.9163, 4.8952, 5.6138,
9.9418, 10.6109, 11.1282, 12.7357, 11.0690, 10.4473, 13.2904, 9.9944,
17.0782, 10.3416, 6.9438, 6.2926, 11.2201, 6.1278, 12.1820, 11.8296,
9.9560, 4.8155, 8.0361, 9.2518, 10.4862, 12.2798, 10.8676, 6.2806,
11.2317, 12.0288, 9.9416, 9.9810, 8.7164, 9.4495, 9.4815, 9.4012,
10.4839, 7.1395, 14.7499, 10.6155, 7.0444, 11.4811, 11.3775, 13.4971,
7.6662, 10.0595, 11.8670, 7.0470, 9.9963, 8.3277, 12.4580, 10.9683,
9.2675, 8.0764, 5.4207, 11.0078, 9.7833, 12.9759, 6.2853, 8.8741])
5.3 通道对应的l2值
L2:
tensor([0.1787, 0.2391, 0.3342, 0.3921, 0.3440, 0.4416, 0.3498, 0.4368, 0.3053,
0.4745, 0.2787, 0.3175, 0.1888, 0.2498, 0.2963, 0.5002, 0.4961, 0.3563,
0.5600, 0.2841, 0.4305, 0.3648, 0.3448, 0.4190, 0.4396, 0.1705, 0.1376,
0.4672, 0.4020, 0.4780, 0.2914, 0.1115, 0.3916, 0.2377, 0.3976, 0.4800,
0.0977, 0.2469, 0.4925, 0.3215, 0.2989, 0.3953, 0.5386, 0.4496, 0.4931,
0.4462, 0.3367, 0.3745, 0.4365, 0.2507, 0.1566, 0.3804, 0.3654, 0.3888,
0.3183, 0.2892, 0.3305, 0.3884, 0.1721, 0.5207, 0.3797, 0.3375, 0.1169,
0.3770, 0.3088, 0.3439, 0.5457, 0.1945, 0.4879, 0.2408, 0.1922, 0.2234,
0.4317, 0.4117, 0.4434, 0.5047, 0.4493, 0.4560, 0.5692, 0.3967, 0.6973,
0.4085, 0.2812, 0.2542, 0.4401, 0.2411, 0.5101, 0.4650, 0.3938, 0.1905,
0.3086, 0.3571, 0.4044, 0.4873, 0.4430, 0.2432, 0.4442, 0.5425, 0.3944,
0.3935, 0.3775, 0.4198, 0.3633, 0.3755, 0.4250, 0.2776, 0.6163, 0.4177,
0.2753, 0.4480, 0.4460, 0.5406, 0.2966, 0.3973, 0.4547, 0.2785, 0.3854,
0.3280, 0.4983, 0.4521, 0.3613, 0.3052, 0.2156, 0.4326, 0.4208, 0.5252,
0.2488, 0.3467])
5.4 通道对应的tvloss值
tvloss:
tensor([1.3992e-04, 2.2695e-04, 2.7250e-04, 7.6327e-04, 7.5144e-04, 8.7256e-04,
5.5133e-04, 1.3989e-03, 5.4804e-04, 9.4830e-04, 2.1857e-04, 1.9901e-04,
1.4296e-04, 2.0737e-04, 1.4010e-04, 1.7853e-03, 1.1067e-03, 9.6182e-04,
1.5229e-03, 1.6583e-04, 8.5500e-04, 5.1329e-04, 5.4607e-04, 6.2221e-04,
9.3564e-04, 1.2324e-04, 5.7210e-05, 1.0581e-03, 1.1646e-03, 9.6945e-04,
4.2071e-04, 4.5595e-05, 8.3459e-04, 1.9988e-04, 1.0124e-03, 1.0173e-03,
3.6465e-05, 2.9829e-04, 9.4978e-04, 6.5539e-04, 2.3278e-04, 6.3994e-04,
1.5582e-03, 1.3657e-03, 1.5396e-03, 9.7195e-04, 6.6808e-04, 6.2939e-04,
7.5327e-04, 3.2658e-04, 8.0546e-05, 7.0369e-04, 6.4445e-04, 1.1580e-03,
3.6162e-04, 2.3316e-04, 6.8859e-04, 9.7655e-04, 1.4548e-04, 1.9952e-03,
7.2628e-04, 7.0566e-04, 5.2536e-05, 5.8076e-04, 3.9151e-04, 7.9537e-04,
1.4251e-03, 1.5686e-04, 1.7316e-03, 2.2931e-04, 1.7838e-04, 3.2831e-04,
4.5192e-04, 7.3333e-04, 1.3144e-03, 1.2641e-03, 1.4423e-03, 5.2904e-04,
1.7992e-03, 1.0000e-03, 1.9789e-03, 6.4655e-04, 3.7544e-04, 2.1733e-04,
9.0644e-04, 2.8223e-04, 1.7986e-03, 1.7908e-03, 6.4259e-04, 1.3189e-04,
4.1928e-04, 5.2752e-04, 6.4312e-04, 9.1314e-04, 1.5107e-03, 3.0553e-04,
1.3613e-03, 2.2215e-03, 3.5164e-04, 6.3454e-04, 6.5172e-04, 3.5056e-04,
8.1161e-04, 3.3546e-04, 1.2511e-03, 2.8639e-04, 1.5900e-03, 8.1033e-04,
3.9007e-04, 1.5212e-03, 1.4268e-03, 2.2759e-03, 4.3843e-04, 1.0428e-03,
1.5551e-03, 3.1775e-04, 1.0363e-03, 8.3562e-04, 2.0071e-03, 8.1407e-04,
4.5609e-04, 3.5050e-04, 2.5387e-04, 6.7972e-04, 3.7846e-04, 1.3381e-03,
2.8636e-04, 4.4538e-04])
5.5 acc与L1/L2/TVLOSS的相关系数
|
L1 |
L2 |
Tvloss |
相关系数 |
-0.4043 |
-0.4212 |
-0.2575 |
6 TODO
resnet50对于cifar10来说参数偏多,对于imagenet是否有类似的结论?
能否找到合适的度量来评估卷积核的重要性,而不是通过现在这种逐次测试准确率的方法。
- 点赞
- 收藏
- 关注作者
评论(0)