use_mkldnn=True表现异常 (#12131) · Issue · PaddlePaddle / Paddle

use_mkldnn=True表现异常

Created by: bipedalBit

我在MacOS系统中用docker起了一个Ubuntu 16.04的容器
严格按说明安装了mkldnn
pip install paddlepaddle安装了paddle 0.14.0
使用fluid搭建了一个3层fc的简单模型
fluid.layers.fc的use_mkldnn参数置True
使用fluid.ParallelExecutor运行模型训练程序

日志如下：

INFO: 07-13 05:43:43: model.py:107 * 139679863383808 Train: epoch 0, average cost: 3.017329, average accuracy: 0.363281
INFO: 07-13 05:43:44: model.py:107 * 139679863383808 Train: epoch 1, average cost: 2.489046, average accuracy: 0.483073
INFO: 07-13 05:43:44: model.py:107 * 139679863383808 Train: epoch 2, average cost: 1.946011, average accuracy: 0.545573
INFO: 07-13 05:43:45: model.py:107 * 139679863383808 Train: epoch 3, average cost: 1.320109, average accuracy: 0.591146
INFO: 07-13 05:43:45: model.py:107 * 139679863383808 Train: epoch 4, average cost: 2.642845, average accuracy: 0.569010
INFO: 07-13 05:43:45: model.py:107 * 139679863383808 Train: epoch 5, average cost: 4.254782, average accuracy: 0.548177
INFO: 07-13 05:43:46: model.py:107 * 139679863383808 Train: epoch 6, average cost: 4.210656, average accuracy: 0.570312
INFO: 07-13 05:43:46: model.py:107 * 139679863383808 Train: epoch 7, average cost: 3.032582, average accuracy: 0.600260
INFO: 07-13 05:43:47: model.py:107 * 139679863383808 Train: epoch 8, average cost: 4.139921, average accuracy: 0.606771
INFO: 07-13 05:43:47: model.py:107 * 139679863383808 Train: epoch 9, average cost: 2.891686, average accuracy: 0.619792
INFO: 07-13 05:43:47: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-9
INFO: 07-13 05:43:48: model.py:107 * 139679863383808 Train: epoch 10, average cost: 2.406002, average accuracy: 0.628906
INFO: 07-13 05:43:48: model.py:107 * 139679863383808 Train: epoch 11, average cost: 3.166168, average accuracy: 0.611979
INFO: 07-13 05:43:48: model.py:107 * 139679863383808 Train: epoch 12, average cost: 2.559951, average accuracy: 0.623698
INFO: 07-13 05:43:49: model.py:107 * 139679863383808 Train: epoch 13, average cost: 1.378130, average accuracy: 0.645833
INFO: 07-13 05:43:49: model.py:107 * 139679863383808 Train: epoch 14, average cost: 2.780429, average accuracy: 0.618490
INFO: 07-13 05:43:50: model.py:107 * 139679863383808 Train: epoch 15, average cost: 3.023936, average accuracy: 0.623698
INFO: 07-13 05:43:50: model.py:107 * 139679863383808 Train: epoch 16, average cost: 1.938339, average accuracy: 0.640625
INFO: 07-13 05:43:51: model.py:107 * 139679863383808 Train: epoch 17, average cost: 2.699482, average accuracy: 0.630208
INFO: 07-13 05:43:51: model.py:107 * 139679863383808 Train: epoch 18, average cost: 2.029525, average accuracy: 0.651042
INFO: 07-13 05:43:52: model.py:107 * 139679863383808 Train: epoch 19, average cost: 1.593852, average accuracy: 0.667969
INFO: 07-13 05:43:52: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-19
INFO: 07-13 05:43:52: model.py:107 * 139679863383808 Train: epoch 20, average cost: 1.240289, average accuracy: 0.658854
INFO: 07-13 05:43:52: model.py:107 * 139679863383808 Train: epoch 21, average cost: 2.560953, average accuracy: 0.652344
INFO: 07-13 05:43:53: model.py:107 * 139679863383808 Train: epoch 22, average cost: 2.253163, average accuracy: 0.638021
INFO: 07-13 05:43:53: model.py:107 * 139679863383808 Train: epoch 23, average cost: 2.898263, average accuracy: 0.648438
INFO: 07-13 05:43:54: model.py:107 * 139679863383808 Train: epoch 24, average cost: 1.063407, average accuracy: 0.683594
INFO: 07-13 05:43:54: model.py:107 * 139679863383808 Train: epoch 25, average cost: 0.960033, average accuracy: 0.692708
INFO: 07-13 05:43:55: model.py:107 * 139679863383808 Train: epoch 26, average cost: 2.196545, average accuracy: 0.631510
INFO: 07-13 05:43:55: model.py:107 * 139679863383808 Train: epoch 27, average cost: 2.570275, average accuracy: 0.657552
INFO: 07-13 05:43:56: model.py:107 * 139679863383808 Train: epoch 28, average cost: 0.908734, average accuracy: 0.695312
INFO: 07-13 05:43:56: model.py:107 * 139679863383808 Train: epoch 29, average cost: 1.389157, average accuracy: 0.695312
INFO: 07-13 05:43:56: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-29
INFO: 07-13 05:43:57: model.py:107 * 139679863383808 Train: epoch 30, average cost: 2.379101, average accuracy: 0.684896
INFO: 07-13 05:43:57: model.py:107 * 139679863383808 Train: epoch 31, average cost: 2.564972, average accuracy: 0.656250
INFO: 07-13 05:43:58: model.py:107 * 139679863383808 Train: epoch 32, average cost: 2.475330, average accuracy: 0.664062
INFO: 07-13 05:43:58: model.py:107 * 139679863383808 Train: epoch 33, average cost: 1.951182, average accuracy: 0.688802
INFO: 07-13 05:43:58: model.py:107 * 139679863383808 Train: epoch 34, average cost: 4.177868, average accuracy: 0.647135
INFO: 07-13 05:43:59: model.py:107 * 139679863383808 Train: epoch 35, average cost: 2.891385, average accuracy: 0.666667
INFO: 07-13 05:43:59: model.py:107 * 139679863383808 Train: epoch 36, average cost: 3.416686, average accuracy: 0.664062
INFO: 07-13 05:44:00: model.py:107 * 139679863383808 Train: epoch 37, average cost: 1.928989, average accuracy: 0.675781
INFO: 07-13 05:44:00: model.py:107 * 139679863383808 Train: epoch 38, average cost: 0.993063, average accuracy: 0.726562
INFO: 07-13 05:44:01: model.py:107 * 139679863383808 Train: epoch 39, average cost: 3.475044, average accuracy: 0.645833
INFO: 07-13 05:44:01: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-39
INFO: 07-13 05:44:01: model.py:107 * 139679863383808 Train: epoch 40, average cost: 2.659002, average accuracy: 0.690104
INFO: 07-13 05:44:01: model.py:107 * 139679863383808 Train: epoch 41, average cost: 3.121070, average accuracy: 0.671875
INFO: 07-13 05:44:02: model.py:107 * 139679863383808 Train: epoch 42, average cost: 3.013545, average accuracy: 0.691406
INFO: 07-13 05:44:02: model.py:107 * 139679863383808 Train: epoch 43, average cost: 2.164809, average accuracy: 0.695312
INFO: 07-13 05:44:03: model.py:107 * 139679863383808 Train: epoch 44, average cost: 2.152214, average accuracy: 0.697917
INFO: 07-13 05:44:03: model.py:107 * 139679863383808 Train: epoch 45, average cost: 1.003491, average accuracy: 0.722656
INFO: 07-13 05:44:04: model.py:107 * 139679863383808 Train: epoch 46, average cost: 2.613479, average accuracy: 0.703125
INFO: 07-13 05:44:04: model.py:107 * 139679863383808 Train: epoch 47, average cost: 1.914267, average accuracy: 0.699219
INFO: 07-13 05:44:05: model.py:107 * 139679863383808 Train: epoch 48, average cost: 3.186370, average accuracy: 0.673177
INFO: 07-13 05:44:05: model.py:107 * 139679863383808 Train: epoch 49, average cost: 3.081753, average accuracy: 0.673177
INFO: 07-13 05:44:05: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-49
INFO: 07-13 05:44:05: model.py:107 * 139679863383808 Train: epoch 50, average cost: 1.390708, average accuracy: 0.725260
INFO: 07-13 05:44:06: model.py:107 * 139679863383808 Train: epoch 51, average cost: 1.949504, average accuracy: 0.721354
INFO: 07-13 05:44:06: model.py:107 * 139679863383808 Train: epoch 52, average cost: 1.597929, average accuracy: 0.707031
INFO: 07-13 05:44:07: model.py:107 * 139679863383808 Train: epoch 53, average cost: 0.802412, average accuracy: 0.750000
INFO: 07-13 05:44:07: model.py:107 * 139679863383808 Train: epoch 54, average cost: 2.411009, average accuracy: 0.718750
INFO: 07-13 05:44:08: model.py:107 * 139679863383808 Train: epoch 55, average cost: 1.498306, average accuracy: 0.710938
INFO: 07-13 05:44:08: model.py:107 * 139679863383808 Train: epoch 56, average cost: 1.734822, average accuracy: 0.726562
INFO: 07-13 05:44:09: model.py:107 * 139679863383808 Train: epoch 57, average cost: 1.829362, average accuracy: 0.704427
INFO: 07-13 05:44:09: model.py:107 * 139679863383808 Train: epoch 58, average cost: 1.054880, average accuracy: 0.742188
INFO: 07-13 05:44:09: model.py:107 * 139679863383808 Train: epoch 59, average cost: 1.788491, average accuracy: 0.707031
INFO: 07-13 05:44:10: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-59
INFO: 07-13 05:44:10: model.py:107 * 139679863383808 Train: epoch 60, average cost: 3.511059, average accuracy: 0.712240
INFO: 07-13 05:44:10: model.py:107 * 139679863383808 Train: epoch 61, average cost: 0.976011, average accuracy: 0.740885
INFO: 07-13 05:44:11: model.py:107 * 139679863383808 Train: epoch 62, average cost: 1.044576, average accuracy: 0.743490
INFO: 07-13 05:44:11: model.py:107 * 139679863383808 Train: epoch 63, average cost: 0.899340, average accuracy: 0.757812
INFO: 07-13 05:44:12: model.py:107 * 139679863383808 Train: epoch 64, average cost: 2.477690, average accuracy: 0.717448
INFO: 07-13 05:44:12: model.py:107 * 139679863383808 Train: epoch 65, average cost: 2.377518, average accuracy: 0.699219
INFO: 07-13 05:44:13: model.py:107 * 139679863383808 Train: epoch 66, average cost: 2.021298, average accuracy: 0.703125
INFO: 07-13 05:44:13: model.py:107 * 139679863383808 Train: epoch 67, average cost: 2.554380, average accuracy: 0.708333
INFO: 07-13 05:44:14: model.py:107 * 139679863383808 Train: epoch 68, average cost: 3.546416, average accuracy: 0.694010
INFO: 07-13 05:44:14: model.py:107 * 139679863383808 Train: epoch 69, average cost: 2.458568, average accuracy: 0.722656
INFO: 07-13 05:44:14: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-69
INFO: 07-13 05:44:15: model.py:107 * 139679863383808 Train: epoch 70, average cost: 2.322152, average accuracy: 0.729167
INFO: 07-13 05:44:15: model.py:107 * 139679863383808 Train: epoch 71, average cost: 3.763601, average accuracy: 0.705729
INFO: 07-13 05:44:15: model.py:107 * 139679863383808 Train: epoch 72, average cost: 1.934551, average accuracy: 0.729167
INFO: 07-13 05:44:16: model.py:107 * 139679863383808 Train: epoch 73, average cost: 2.759411, average accuracy: 0.696615
INFO: 07-13 05:44:16: model.py:107 * 139679863383808 Train: epoch 74, average cost: 1.161568, average accuracy: 0.761719
INFO: 07-13 05:44:17: model.py:107 * 139679863383808 Train: epoch 75, average cost: 1.837180, average accuracy: 0.736979
INFO: 07-13 05:44:17: model.py:107 * 139679863383808 Train: epoch 76, average cost: 2.300429, average accuracy: 0.712240
INFO: 07-13 05:44:18: model.py:107 * 139679863383808 Train: epoch 77, average cost: 1.529922, average accuracy: 0.744792
INFO: 07-13 05:44:18: model.py:107 * 139679863383808 Train: epoch 78, average cost: 1.110254, average accuracy: 0.731771
INFO: 07-13 05:44:19: model.py:107 * 139679863383808 Train: epoch 79, average cost: 3.726367, average accuracy: 0.700521
INFO: 07-13 05:44:19: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-79
INFO: 07-13 05:44:19: model.py:107 * 139679863383808 Train: epoch 80, average cost: 5.805529, average accuracy: 0.660156
INFO: 07-13 05:44:20: model.py:107 * 139679863383808 Train: epoch 81, average cost: 1.066115, average accuracy: 0.738281
INFO: 07-13 05:44:20: model.py:107 * 139679863383808 Train: epoch 82, average cost: 0.806044, average accuracy: 0.765625
INFO: 07-13 05:44:21: model.py:107 * 139679863383808 Train: epoch 83, average cost: 2.483238, average accuracy: 0.751302
INFO: 07-13 05:44:21: model.py:107 * 139679863383808 Train: epoch 84, average cost: 1.464043, average accuracy: 0.769531
INFO: 07-13 05:44:22: model.py:107 * 139679863383808 Train: epoch 85, average cost: 3.488322, average accuracy: 0.705729
INFO: 07-13 05:44:22: model.py:107 * 139679863383808 Train: epoch 86, average cost: 3.726289, average accuracy: 0.739583
INFO: 07-13 05:44:23: model.py:107 * 139679863383808 Train: epoch 87, average cost: 1.248387, average accuracy: 0.746094
INFO: 07-13 05:44:23: model.py:107 * 139679863383808 Train: epoch 88, average cost: 3.396407, average accuracy: 0.718750
INFO: 07-13 05:44:24: model.py:107 * 139679863383808 Train: epoch 89, average cost: 1.461747, average accuracy: 0.755208
INFO: 07-13 05:44:24: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-89
INFO: 07-13 05:44:24: model.py:107 * 139679863383808 Train: epoch 90, average cost: 1.207618, average accuracy: 0.768229
INFO: 07-13 05:44:25: model.py:107 * 139679863383808 Train: epoch 91, average cost: 2.252221, average accuracy: 0.736979
INFO: 07-13 05:44:25: model.py:107 * 139679863383808 Train: epoch 92, average cost: 2.414238, average accuracy: 0.743490
INFO: 07-13 05:44:26: model.py:107 * 139679863383808 Train: epoch 93, average cost: 2.154225, average accuracy: 0.714844
INFO: 07-13 05:44:26: model.py:107 * 139679863383808 Train: epoch 94, average cost: 2.673501, average accuracy: 0.697917
INFO: 07-13 05:44:27: model.py:107 * 139679863383808 Train: epoch 95, average cost: 2.010543, average accuracy: 0.725260
INFO: 07-13 05:44:27: model.py:107 * 139679863383808 Train: epoch 96, average cost: 2.981954, average accuracy: 0.692708
INFO: 07-13 05:44:28: model.py:107 * 139679863383808 Train: epoch 97, average cost: 1.559371, average accuracy: 0.765625
INFO: 07-13 05:44:28: model.py:107 * 139679863383808 Train: epoch 98, average cost: 1.222868, average accuracy: 0.763021
INFO: 07-13 05:44:28: model.py:107 * 139679863383808 Train: epoch 99, average cost: 2.264318, average accuracy: 0.768229
INFO: 07-13 05:44:29: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-99
INFO: 07-13 05:44:29: model.py:107 * 139679863383808 Train: epoch 100, average cost: 0.973710, average accuracy: 0.778646
INFO: 07-13 05:44:29: model.py:107 * 139679863383808 Train: epoch 101, average cost: 3.790188, average accuracy: 0.716146
INFO: 07-13 05:44:30: model.py:107 * 139679863383808 Train: epoch 102, average cost: 1.439416, average accuracy: 0.742188
INFO: 07-13 05:44:30: model.py:107 * 139679863383808 Train: epoch 103, average cost: 2.400558, average accuracy: 0.716146
INFO: 07-13 05:44:31: model.py:107 * 139679863383808 Train: epoch 104, average cost: 3.936377, average accuracy: 0.717448
INFO: 07-13 05:44:31: model.py:107 * 139679863383808 Train: epoch 105, average cost: 2.873991, average accuracy: 0.718750
INFO: 07-13 05:44:32: model.py:107 * 139679863383808 Train: epoch 106, average cost: 2.645878, average accuracy: 0.718750
INFO: 07-13 05:44:33: model.py:107 * 139679863383808 Train: epoch 107, average cost: 3.573340, average accuracy: 0.725260
INFO: 07-13 05:44:33: model.py:107 * 139679863383808 Train: epoch 108, average cost: 1.902647, average accuracy: 0.703125
INFO: 07-13 05:44:34: model.py:107 * 139679863383808 Train: epoch 109, average cost: 1.849591, average accuracy: 0.740885
INFO: 07-13 05:44:34: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-109
INFO: 07-13 05:44:34: model.py:107 * 139679863383808 Train: epoch 110, average cost: 1.150515, average accuracy: 0.744792
INFO: 07-13 05:44:35: model.py:107 * 139679863383808 Train: epoch 111, average cost: 2.176013, average accuracy: 0.742188
INFO: 07-13 05:44:35: model.py:107 * 139679863383808 Train: epoch 112, average cost: 1.237140, average accuracy: 0.757812
INFO: 07-13 05:44:35: model.py:107 * 139679863383808 Train: epoch 113, average cost: 2.419860, average accuracy: 0.712240
INFO: 07-13 05:44:36: model.py:107 * 139679863383808 Train: epoch 114, average cost: 1.654727, average accuracy: 0.727865
INFO: 07-13 05:44:36: model.py:107 * 139679863383808 Train: epoch 115, average cost: 1.780760, average accuracy: 0.748698
INFO: 07-13 05:44:37: model.py:107 * 139679863383808 Train: epoch 116, average cost: 1.448064, average accuracy: 0.765625
INFO: 07-13 05:44:37: model.py:107 * 139679863383808 Train: epoch 117, average cost: 2.028671, average accuracy: 0.748698
INFO: 07-13 05:44:38: model.py:107 * 139679863383808 Train: epoch 118, average cost: 3.702690, average accuracy: 0.694010
INFO: 07-13 05:44:38: model.py:107 * 139679863383808 Train: epoch 119, average cost: 1.394666, average accuracy: 0.742188
INFO: 07-13 05:44:38: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-119
INFO: 07-13 05:44:39: model.py:107 * 139679863383808 Train: epoch 120, average cost: 3.621784, average accuracy: 0.709635
INFO: 07-13 05:44:39: model.py:107 * 139679863383808 Train: epoch 121, average cost: 2.855005, average accuracy: 0.735677
INFO: 07-13 05:44:40: model.py:107 * 139679863383808 Train: epoch 122, average cost: 2.628077, average accuracy: 0.712240
INFO: 07-13 05:44:40: model.py:107 * 139679863383808 Train: epoch 123, average cost: 2.993645, average accuracy: 0.716146
INFO: 07-13 05:44:40: model.py:107 * 139679863383808 Train: epoch 124, average cost: 2.969736, average accuracy: 0.705729
INFO: 07-13 05:44:41: model.py:107 * 139679863383808 Train: epoch 125, average cost: 4.065771, average accuracy: 0.662760
INFO: 07-13 05:44:41: model.py:107 * 139679863383808 Train: epoch 126, average cost: 1.303965, average accuracy: 0.752604
INFO: 07-13 05:44:42: model.py:107 * 139679863383808 Train: epoch 127, average cost: 1.919925, average accuracy: 0.720052
INFO: 07-13 05:44:42: model.py:107 * 139679863383808 Train: epoch 128, average cost: 1.942278, average accuracy: 0.742188
INFO: 07-13 05:44:43: model.py:107 * 139679863383808 Train: epoch 129, average cost: 0.955757, average accuracy: 0.770833
INFO: 07-13 05:44:43: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-129
INFO: 07-13 05:44:43: model.py:107 * 139679863383808 Train: epoch 130, average cost: 2.309173, average accuracy: 0.740885
INFO: 07-13 05:44:44: model.py:107 * 139679863383808 Train: epoch 131, average cost: 1.215543, average accuracy: 0.750000
INFO: 07-13 05:44:44: model.py:107 * 139679863383808 Train: epoch 132, average cost: 1.220970, average accuracy: 0.740885
INFO: 07-13 05:44:45: model.py:107 * 139679863383808 Train: epoch 133, average cost: 2.568725, average accuracy: 0.748698
INFO: 07-13 05:44:45: model.py:107 * 139679863383808 Train: epoch 134, average cost: 2.024275, average accuracy: 0.764323
INFO: 07-13 05:44:46: model.py:107 * 139679863383808 Train: epoch 135, average cost: 1.315992, average accuracy: 0.768229
INFO: 07-13 05:44:46: model.py:107 * 139679863383808 Train: epoch 136, average cost: 2.307564, average accuracy: 0.678385
INFO: 07-13 05:44:47: model.py:107 * 139679863383808 Train: epoch 137, average cost: 1.914190, average accuracy: 0.734375
INFO: 07-13 05:44:47: model.py:107 * 139679863383808 Train: epoch 138, average cost: 1.380930, average accuracy: 0.748698
INFO: 07-13 05:44:48: model.py:107 * 139679863383808 Train: epoch 139, average cost: 3.338050, average accuracy: 0.680990
INFO: 07-13 05:44:48: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-139
INFO: 07-13 05:44:48: model.py:107 * 139679863383808 Train: epoch 140, average cost: nan, average accuracy: 0.040365
INFO: 07-13 05:44:48: model.py:107 * 139679863383808 Train: epoch 141, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:49: model.py:107 * 139679863383808 Train: epoch 142, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:44:49: model.py:107 * 139679863383808 Train: epoch 143, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:50: model.py:107 * 139679863383808 Train: epoch 144, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:44:50: model.py:107 * 139679863383808 Train: epoch 145, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:51: model.py:107 * 139679863383808 Train: epoch 146, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:44:51: model.py:107 * 139679863383808 Train: epoch 147, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:52: model.py:107 * 139679863383808 Train: epoch 148, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:52: model.py:107 * 139679863383808 Train: epoch 149, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:52: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-149
INFO: 07-13 05:44:53: model.py:107 * 139679863383808 Train: epoch 150, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:53: model.py:107 * 139679863383808 Train: epoch 151, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:53: model.py:107 * 139679863383808 Train: epoch 152, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:54: model.py:107 * 139679863383808 Train: epoch 153, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:54: model.py:107 * 139679863383808 Train: epoch 154, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:44:55: model.py:107 * 139679863383808 Train: epoch 155, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:55: model.py:107 * 139679863383808 Train: epoch 156, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:44:56: model.py:107 * 139679863383808 Train: epoch 157, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:44:56: model.py:107 * 139679863383808 Train: epoch 158, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:57: model.py:107 * 139679863383808 Train: epoch 159, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:57: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-159
INFO: 07-13 05:44:57: model.py:107 * 139679863383808 Train: epoch 160, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:44:57: model.py:107 * 139679863383808 Train: epoch 161, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:44:58: model.py:107 * 139679863383808 Train: epoch 162, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:58: model.py:107 * 139679863383808 Train: epoch 163, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:59: model.py:107 * 139679863383808 Train: epoch 164, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:44:59: model.py:107 * 139679863383808 Train: epoch 165, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:00: model.py:107 * 139679863383808 Train: epoch 166, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:00: model.py:107 * 139679863383808 Train: epoch 167, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:01: model.py:107 * 139679863383808 Train: epoch 168, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:01: model.py:107 * 139679863383808 Train: epoch 169, average cost: nan, average accuracy: 0.028646
INFO: 07-13 05:45:01: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-169
INFO: 07-13 05:45:02: model.py:107 * 139679863383808 Train: epoch 170, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:02: model.py:107 * 139679863383808 Train: epoch 171, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:03: model.py:107 * 139679863383808 Train: epoch 172, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:03: model.py:107 * 139679863383808 Train: epoch 173, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:04: model.py:107 * 139679863383808 Train: epoch 174, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:04: model.py:107 * 139679863383808 Train: epoch 175, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:05: model.py:107 * 139679863383808 Train: epoch 176, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:05: model.py:107 * 139679863383808 Train: epoch 177, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:06: model.py:107 * 139679863383808 Train: epoch 178, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:06: model.py:107 * 139679863383808 Train: epoch 179, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:06: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-179
INFO: 07-13 05:45:07: model.py:107 * 139679863383808 Train: epoch 180, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:07: model.py:107 * 139679863383808 Train: epoch 181, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:08: model.py:107 * 139679863383808 Train: epoch 182, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:08: model.py:107 * 139679863383808 Train: epoch 183, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:09: model.py:107 * 139679863383808 Train: epoch 184, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:09: model.py:107 * 139679863383808 Train: epoch 185, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:10: model.py:107 * 139679863383808 Train: epoch 186, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:10: model.py:107 * 139679863383808 Train: epoch 187, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:11: model.py:107 * 139679863383808 Train: epoch 188, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:11: model.py:107 * 139679863383808 Train: epoch 189, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:11: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-189
INFO: 07-13 05:45:12: model.py:107 * 139679863383808 Train: epoch 190, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:12: model.py:107 * 139679863383808 Train: epoch 191, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:12: model.py:107 * 139679863383808 Train: epoch 192, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:13: model.py:107 * 139679863383808 Train: epoch 193, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:13: model.py:107 * 139679863383808 Train: epoch 194, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:14: model.py:107 * 139679863383808 Train: epoch 195, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:14: model.py:107 * 139679863383808 Train: epoch 196, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:15: model.py:107 * 139679863383808 Train: epoch 197, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:15: model.py:107 * 139679863383808 Train: epoch 198, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:15: model.py:107 * 139679863383808 Train: epoch 199, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:16: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-199
INFO: 07-13 05:45:16: model.py:107 * 139679863383808 Train: epoch 200, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:16: model.py:107 * 139679863383808 Train: epoch 201, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:17: model.py:107 * 139679863383808 Train: epoch 202, average cost: nan, average accuracy: 0.028646
INFO: 07-13 05:45:17: model.py:107 * 139679863383808 Train: epoch 203, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:18: model.py:107 * 139679863383808 Train: epoch 204, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:18: model.py:107 * 139679863383808 Train: epoch 205, average cost: nan, average accuracy: 0.020833
INFO: 07-13 05:45:19: model.py:107 * 139679863383808 Train: epoch 206, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:19: model.py:107 * 139679863383808 Train: epoch 207, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:20: model.py:107 * 139679863383808 Train: epoch 208, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:20: model.py:107 * 139679863383808 Train: epoch 209, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:20: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-209
INFO: 07-13 05:45:21: model.py:107 * 139679863383808 Train: epoch 210, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:21: model.py:107 * 139679863383808 Train: epoch 211, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:22: model.py:107 * 139679863383808 Train: epoch 212, average cost: nan, average accuracy: 0.028646
INFO: 07-13 05:45:22: model.py:107 * 139679863383808 Train: epoch 213, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:23: model.py:107 * 139679863383808 Train: epoch 214, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:23: model.py:107 * 139679863383808 Train: epoch 215, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:24: model.py:107 * 139679863383808 Train: epoch 216, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:24: model.py:107 * 139679863383808 Train: epoch 217, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:25: model.py:107 * 139679863383808 Train: epoch 218, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:25: model.py:107 * 139679863383808 Train: epoch 219, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:25: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-219
INFO: 07-13 05:45:26: model.py:107 * 139679863383808 Train: epoch 220, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:26: model.py:107 * 139679863383808 Train: epoch 221, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:27: model.py:107 * 139679863383808 Train: epoch 222, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:27: model.py:107 * 139679863383808 Train: epoch 223, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:28: model.py:107 * 139679863383808 Train: epoch 224, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:28: model.py:107 * 139679863383808 Train: epoch 225, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:29: model.py:107 * 139679863383808 Train: epoch 226, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:29: model.py:107 * 139679863383808 Train: epoch 227, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:29: model.py:107 * 139679863383808 Train: epoch 228, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:30: model.py:107 * 139679863383808 Train: epoch 229, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:30: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-229
INFO: 07-13 05:45:30: model.py:107 * 139679863383808 Train: epoch 230, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:31: model.py:107 * 139679863383808 Train: epoch 231, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:31: model.py:107 * 139679863383808 Train: epoch 232, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:32: model.py:107 * 139679863383808 Train: epoch 233, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:32: model.py:107 * 139679863383808 Train: epoch 234, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:33: model.py:107 * 139679863383808 Train: epoch 235, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:33: model.py:107 * 139679863383808 Train: epoch 236, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:34: model.py:107 * 139679863383808 Train: epoch 237, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:34: model.py:107 * 139679863383808 Train: epoch 238, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:35: model.py:107 * 139679863383808 Train: epoch 239, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:35: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-239
INFO: 07-13 05:45:35: model.py:107 * 139679863383808 Train: epoch 240, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:36: model.py:107 * 139679863383808 Train: epoch 241, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:36: model.py:107 * 139679863383808 Train: epoch 242, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:36: model.py:107 * 139679863383808 Train: epoch 243, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:37: model.py:107 * 139679863383808 Train: epoch 244, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:37: model.py:107 * 139679863383808 Train: epoch 245, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:38: model.py:107 * 139679863383808 Train: epoch 246, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:38: model.py:107 * 139679863383808 Train: epoch 247, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:39: model.py:107 * 139679863383808 Train: epoch 248, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:39: model.py:107 * 139679863383808 Train: epoch 249, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:39: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-249
INFO: 07-13 05:45:40: model.py:107 * 139679863383808 Train: epoch 250, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:40: model.py:107 * 139679863383808 Train: epoch 251, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:41: model.py:107 * 139679863383808 Train: epoch 252, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:42: model.py:107 * 139679863383808 Train: epoch 253, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:42: model.py:107 * 139679863383808 Train: epoch 254, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:43: model.py:107 * 139679863383808 Train: epoch 255, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:43: model.py:107 * 139679863383808 Train: epoch 256, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:44: model.py:107 * 139679863383808 Train: epoch 257, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:44: model.py:107 * 139679863383808 Train: epoch 258, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:44: model.py:107 * 139679863383808 Train: epoch 259, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:44: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-259
INFO: 07-13 05:45:45: model.py:107 * 139679863383808 Train: epoch 260, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:45: model.py:107 * 139679863383808 Train: epoch 261, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:46: model.py:107 * 139679863383808 Train: epoch 262, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:46: model.py:107 * 139679863383808 Train: epoch 263, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:47: model.py:107 * 139679863383808 Train: epoch 264, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:47: model.py:107 * 139679863383808 Train: epoch 265, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:48: model.py:107 * 139679863383808 Train: epoch 266, average cost: nan, average accuracy: 0.028646
INFO: 07-13 05:45:48: model.py:107 * 139679863383808 Train: epoch 267, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:49: model.py:107 * 139679863383808 Train: epoch 268, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:49: model.py:107 * 139679863383808 Train: epoch 269, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:49: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-269
INFO: 07-13 05:45:50: model.py:107 * 139679863383808 Train: epoch 270, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:50: model.py:107 * 139679863383808 Train: epoch 271, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:51: model.py:107 * 139679863383808 Train: epoch 272, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:45:51: model.py:107 * 139679863383808 Train: epoch 273, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:51: model.py:107 * 139679863383808 Train: epoch 274, average cost: nan, average accuracy: 0.028646
INFO: 07-13 05:45:52: model.py:107 * 139679863383808 Train: epoch 275, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:52: model.py:107 * 139679863383808 Train: epoch 276, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:53: model.py:107 * 139679863383808 Train: epoch 277, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:53: model.py:107 * 139679863383808 Train: epoch 278, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:54: model.py:107 * 139679863383808 Train: epoch 279, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:54: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-279
INFO: 07-13 05:45:54: model.py:107 * 139679863383808 Train: epoch 280, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:55: model.py:107 * 139679863383808 Train: epoch 281, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:55: model.py:107 * 139679863383808 Train: epoch 282, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:56: model.py:107 * 139679863383808 Train: epoch 283, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:56: model.py:107 * 139679863383808 Train: epoch 284, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:57: model.py:107 * 139679863383808 Train: epoch 285, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:57: model.py:107 * 139679863383808 Train: epoch 286, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:58: model.py:107 * 139679863383808 Train: epoch 287, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:45:58: model.py:107 * 139679863383808 Train: epoch 288, average cost: nan, average accuracy: 0.027344
INFO: 07-13 05:45:59: model.py:107 * 139679863383808 Train: epoch 289, average cost: nan, average accuracy: 0.026042
INFO: 07-13 05:45:59: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-289
INFO: 07-13 05:45:59: model.py:107 * 139679863383808 Train: epoch 290, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:00: model.py:107 * 139679863383808 Train: epoch 291, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:00: model.py:107 * 139679863383808 Train: epoch 292, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:46:00: model.py:107 * 139679863383808 Train: epoch 293, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:01: model.py:107 * 139679863383808 Train: epoch 294, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:46:01: model.py:107 * 139679863383808 Train: epoch 295, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:02: model.py:107 * 139679863383808 Train: epoch 296, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:02: model.py:107 * 139679863383808 Train: epoch 297, average cost: nan, average accuracy: 0.023438
INFO: 07-13 05:46:03: model.py:107 * 139679863383808 Train: epoch 298, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:03: model.py:107 * 139679863383808 Train: epoch 299, average cost: nan, average accuracy: 0.024740
INFO: 07-13 05:46:03: model_io.py:83 * 139679863383808 Saved inference model to ./models/infer-epoch-299
INFO: 07-13 05:46:03: model.py:181 * 139679863383808 Costs 140.58s.

请注意日志中的average cost变化情况：

波动比较剧烈，并非单调变化
epoch 140开始average cost变成了nan

接下来我将模型中所有fc层的use_mkldnn参数置False，重复实验，日志如下：

INFO: 07-13 05:48:12: model.py:107 * 140114397865728 Train: epoch 0, average cost: 1.909084, average accuracy: 0.386719
INFO: 07-13 05:48:13: model.py:107 * 140114397865728 Train: epoch 1, average cost: 1.525706, average accuracy: 0.515625
INFO: 07-13 05:48:13: model.py:107 * 140114397865728 Train: epoch 2, average cost: 1.335058, average accuracy: 0.589844
INFO: 07-13 05:48:14: model.py:107 * 140114397865728 Train: epoch 3, average cost: 1.217955, average accuracy: 0.623698
INFO: 07-13 05:48:14: model.py:107 * 140114397865728 Train: epoch 4, average cost: 1.132739, average accuracy: 0.647135
INFO: 07-13 05:48:15: model.py:107 * 140114397865728 Train: epoch 5, average cost: 1.065525, average accuracy: 0.662760
INFO: 07-13 05:48:15: model.py:107 * 140114397865728 Train: epoch 6, average cost: 1.010285, average accuracy: 0.691406
INFO: 07-13 05:48:16: model.py:107 * 140114397865728 Train: epoch 7, average cost: 0.963846, average accuracy: 0.704427
INFO: 07-13 05:48:16: model.py:107 * 140114397865728 Train: epoch 8, average cost: 0.924057, average accuracy: 0.720052
INFO: 07-13 05:48:16: model.py:107 * 140114397865728 Train: epoch 9, average cost: 0.889385, average accuracy: 0.723958
INFO: 07-13 05:48:16: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-9
INFO: 07-13 05:48:17: model.py:107 * 140114397865728 Train: epoch 10, average cost: 0.858741, average accuracy: 0.731771
INFO: 07-13 05:48:17: model.py:107 * 140114397865728 Train: epoch 11, average cost: 0.831309, average accuracy: 0.736979
INFO: 07-13 05:48:18: model.py:107 * 140114397865728 Train: epoch 12, average cost: 0.806434, average accuracy: 0.743490
INFO: 07-13 05:48:18: model.py:107 * 140114397865728 Train: epoch 13, average cost: 0.783616, average accuracy: 0.748698
INFO: 07-13 05:48:19: model.py:107 * 140114397865728 Train: epoch 14, average cost: 0.762484, average accuracy: 0.756510
INFO: 07-13 05:48:19: model.py:107 * 140114397865728 Train: epoch 15, average cost: 0.742775, average accuracy: 0.764323
INFO: 07-13 05:48:20: model.py:107 * 140114397865728 Train: epoch 16, average cost: 0.724286, average accuracy: 0.768229
INFO: 07-13 05:48:20: model.py:107 * 140114397865728 Train: epoch 17, average cost: 0.706880, average accuracy: 0.770833
INFO: 07-13 05:48:21: model.py:107 * 140114397865728 Train: epoch 18, average cost: 0.690451, average accuracy: 0.774740
INFO: 07-13 05:48:21: model.py:107 * 140114397865728 Train: epoch 19, average cost: 0.674930, average accuracy: 0.779948
INFO: 07-13 05:48:21: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-19
INFO: 07-13 05:48:22: model.py:107 * 140114397865728 Train: epoch 20, average cost: 0.660255, average accuracy: 0.787760
INFO: 07-13 05:48:22: model.py:107 * 140114397865728 Train: epoch 21, average cost: 0.646379, average accuracy: 0.792969
INFO: 07-13 05:48:23: model.py:107 * 140114397865728 Train: epoch 22, average cost: 0.633254, average accuracy: 0.794271
INFO: 07-13 05:48:23: model.py:107 * 140114397865728 Train: epoch 23, average cost: 0.620842, average accuracy: 0.796875
INFO: 07-13 05:48:23: model.py:107 * 140114397865728 Train: epoch 24, average cost: 0.609107, average accuracy: 0.800781
INFO: 07-13 05:48:24: model.py:107 * 140114397865728 Train: epoch 25, average cost: 0.598015, average accuracy: 0.805990
INFO: 07-13 05:48:24: model.py:107 * 140114397865728 Train: epoch 26, average cost: 0.587536, average accuracy: 0.811198
INFO: 07-13 05:48:25: model.py:107 * 140114397865728 Train: epoch 27, average cost: 0.577637, average accuracy: 0.816406
INFO: 07-13 05:48:25: model.py:107 * 140114397865728 Train: epoch 28, average cost: 0.568282, average accuracy: 0.817708
INFO: 07-13 05:48:26: model.py:107 * 140114397865728 Train: epoch 29, average cost: 0.559435, average accuracy: 0.816406
INFO: 07-13 05:48:26: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-29
INFO: 07-13 05:48:26: model.py:107 * 140114397865728 Train: epoch 30, average cost: 0.551061, average accuracy: 0.819010
INFO: 07-13 05:48:27: model.py:107 * 140114397865728 Train: epoch 31, average cost: 0.543127, average accuracy: 0.821615
INFO: 07-13 05:48:27: model.py:107 * 140114397865728 Train: epoch 32, average cost: 0.535598, average accuracy: 0.826823
INFO: 07-13 05:48:28: model.py:107 * 140114397865728 Train: epoch 33, average cost: 0.528441, average accuracy: 0.829427
INFO: 07-13 05:48:28: model.py:107 * 140114397865728 Train: epoch 34, average cost: 0.521621, average accuracy: 0.834635
INFO: 07-13 05:48:29: model.py:107 * 140114397865728 Train: epoch 35, average cost: 0.515109, average accuracy: 0.839844
INFO: 07-13 05:48:29: model.py:107 * 140114397865728 Train: epoch 36, average cost: 0.508882, average accuracy: 0.845052
INFO: 07-13 05:48:29: model.py:107 * 140114397865728 Train: epoch 37, average cost: 0.502917, average accuracy: 0.848958
INFO: 07-13 05:48:30: model.py:107 * 140114397865728 Train: epoch 38, average cost: 0.497191, average accuracy: 0.851562
INFO: 07-13 05:48:30: model.py:107 * 140114397865728 Train: epoch 39, average cost: 0.491690, average accuracy: 0.852865
INFO: 07-13 05:48:30: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-39
INFO: 07-13 05:48:31: model.py:107 * 140114397865728 Train: epoch 40, average cost: 0.486399, average accuracy: 0.854167
INFO: 07-13 05:48:31: model.py:107 * 140114397865728 Train: epoch 41, average cost: 0.481306, average accuracy: 0.856771
INFO: 07-13 05:48:32: model.py:107 * 140114397865728 Train: epoch 42, average cost: 0.476398, average accuracy: 0.858073
INFO: 07-13 05:48:32: model.py:107 * 140114397865728 Train: epoch 43, average cost: 0.471667, average accuracy: 0.859375
INFO: 07-13 05:48:33: model.py:107 * 140114397865728 Train: epoch 44, average cost: 0.467103, average accuracy: 0.861979
INFO: 07-13 05:48:33: model.py:107 * 140114397865728 Train: epoch 45, average cost: 0.462696, average accuracy: 0.860677
INFO: 07-13 05:48:34: model.py:107 * 140114397865728 Train: epoch 46, average cost: 0.458438, average accuracy: 0.860677
INFO: 07-13 05:48:34: model.py:107 * 140114397865728 Train: epoch 47, average cost: 0.454321, average accuracy: 0.861979
INFO: 07-13 05:48:35: model.py:107 * 140114397865728 Train: epoch 48, average cost: 0.450337, average accuracy: 0.861979
INFO: 07-13 05:48:35: model.py:107 * 140114397865728 Train: epoch 49, average cost: 0.446481, average accuracy: 0.863281
INFO: 07-13 05:48:35: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-49
INFO: 07-13 05:48:36: model.py:107 * 140114397865728 Train: epoch 50, average cost: 0.442744, average accuracy: 0.865885
INFO: 07-13 05:48:36: model.py:107 * 140114397865728 Train: epoch 51, average cost: 0.439119, average accuracy: 0.869792
INFO: 07-13 05:48:36: model.py:107 * 140114397865728 Train: epoch 52, average cost: 0.435600, average accuracy: 0.869792
INFO: 07-13 05:48:37: model.py:107 * 140114397865728 Train: epoch 53, average cost: 0.432183, average accuracy: 0.869792
INFO: 07-13 05:48:37: model.py:107 * 140114397865728 Train: epoch 54, average cost: 0.428858, average accuracy: 0.871094
INFO: 07-13 05:48:38: model.py:107 * 140114397865728 Train: epoch 55, average cost: 0.425620, average accuracy: 0.873698
INFO: 07-13 05:48:38: model.py:107 * 140114397865728 Train: epoch 56, average cost: 0.422464, average accuracy: 0.873698
INFO: 07-13 05:48:39: model.py:107 * 140114397865728 Train: epoch 57, average cost: 0.419382, average accuracy: 0.873698
INFO: 07-13 05:48:39: model.py:107 * 140114397865728 Train: epoch 58, average cost: 0.416369, average accuracy: 0.872396
INFO: 07-13 05:48:40: model.py:107 * 140114397865728 Train: epoch 59, average cost: 0.413414, average accuracy: 0.872396
INFO: 07-13 05:48:40: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-59
INFO: 07-13 05:48:40: model.py:107 * 140114397865728 Train: epoch 60, average cost: 0.410515, average accuracy: 0.872396
INFO: 07-13 05:48:41: model.py:107 * 140114397865728 Train: epoch 61, average cost: 0.407665, average accuracy: 0.872396
INFO: 07-13 05:48:41: model.py:107 * 140114397865728 Train: epoch 62, average cost: 0.404858, average accuracy: 0.875000
INFO: 07-13 05:48:42: model.py:107 * 140114397865728 Train: epoch 63, average cost: 0.402091, average accuracy: 0.877604
INFO: 07-13 05:48:42: model.py:107 * 140114397865728 Train: epoch 64, average cost: 0.399358, average accuracy: 0.878906
INFO: 07-13 05:48:43: model.py:107 * 140114397865728 Train: epoch 65, average cost: 0.396655, average accuracy: 0.881510
INFO: 07-13 05:48:43: model.py:107 * 140114397865728 Train: epoch 66, average cost: 0.393976, average accuracy: 0.882812
INFO: 07-13 05:48:44: model.py:107 * 140114397865728 Train: epoch 67, average cost: 0.391321, average accuracy: 0.884115
INFO: 07-13 05:48:44: model.py:107 * 140114397865728 Train: epoch 68, average cost: 0.388684, average accuracy: 0.884115
INFO: 07-13 05:48:44: model.py:107 * 140114397865728 Train: epoch 69, average cost: 0.386066, average accuracy: 0.882812
INFO: 07-13 05:48:45: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-69
INFO: 07-13 05:48:45: model.py:107 * 140114397865728 Train: epoch 70, average cost: 0.383469, average accuracy: 0.886719
INFO: 07-13 05:48:45: model.py:107 * 140114397865728 Train: epoch 71, average cost: 0.380891, average accuracy: 0.888021
INFO: 07-13 05:48:46: model.py:107 * 140114397865728 Train: epoch 72, average cost: 0.378329, average accuracy: 0.888021
INFO: 07-13 05:48:46: model.py:107 * 140114397865728 Train: epoch 73, average cost: 0.375786, average accuracy: 0.890625
INFO: 07-13 05:48:47: model.py:107 * 140114397865728 Train: epoch 74, average cost: 0.373263, average accuracy: 0.888021
INFO: 07-13 05:48:47: model.py:107 * 140114397865728 Train: epoch 75, average cost: 0.370758, average accuracy: 0.889323
INFO: 07-13 05:48:48: model.py:107 * 140114397865728 Train: epoch 76, average cost: 0.368266, average accuracy: 0.889323
INFO: 07-13 05:48:48: model.py:107 * 140114397865728 Train: epoch 77, average cost: 0.365791, average accuracy: 0.889323
INFO: 07-13 05:48:49: model.py:107 * 140114397865728 Train: epoch 78, average cost: 0.363341, average accuracy: 0.890625
INFO: 07-13 05:48:49: model.py:107 * 140114397865728 Train: epoch 79, average cost: 0.360928, average accuracy: 0.891927
INFO: 07-13 05:48:49: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-79
INFO: 07-13 05:48:50: model.py:107 * 140114397865728 Train: epoch 80, average cost: 0.358570, average accuracy: 0.894531
INFO: 07-13 05:48:51: model.py:107 * 140114397865728 Train: epoch 81, average cost: 0.356296, average accuracy: 0.894531
INFO: 07-13 05:48:51: model.py:107 * 140114397865728 Train: epoch 82, average cost: 0.354139, average accuracy: 0.893229
INFO: 07-13 05:48:52: model.py:107 * 140114397865728 Train: epoch 83, average cost: 0.352122, average accuracy: 0.890625
INFO: 07-13 05:48:53: model.py:107 * 140114397865728 Train: epoch 84, average cost: 0.350266, average accuracy: 0.890625
INFO: 07-13 05:48:54: model.py:107 * 140114397865728 Train: epoch 85, average cost: 0.348568, average accuracy: 0.891927
INFO: 07-13 05:48:54: model.py:107 * 140114397865728 Train: epoch 86, average cost: 0.347023, average accuracy: 0.893229
INFO: 07-13 05:48:55: model.py:107 * 140114397865728 Train: epoch 87, average cost: 0.345597, average accuracy: 0.894531
INFO: 07-13 05:48:55: model.py:107 * 140114397865728 Train: epoch 88, average cost: 0.344256, average accuracy: 0.894531
INFO: 07-13 05:48:56: model.py:107 * 140114397865728 Train: epoch 89, average cost: 0.342973, average accuracy: 0.894531
INFO: 07-13 05:48:56: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-89
INFO: 07-13 05:48:56: model.py:107 * 140114397865728 Train: epoch 90, average cost: 0.341739, average accuracy: 0.895833
INFO: 07-13 05:48:57: model.py:107 * 140114397865728 Train: epoch 91, average cost: 0.340549, average accuracy: 0.897135
INFO: 07-13 05:48:57: model.py:107 * 140114397865728 Train: epoch 92, average cost: 0.339428, average accuracy: 0.895833
INFO: 07-13 05:48:58: model.py:107 * 140114397865728 Train: epoch 93, average cost: 0.338405, average accuracy: 0.898438
INFO: 07-13 05:48:58: model.py:107 * 140114397865728 Train: epoch 94, average cost: 0.337501, average accuracy: 0.897135
INFO: 07-13 05:48:59: model.py:107 * 140114397865728 Train: epoch 95, average cost: 0.336680, average accuracy: 0.901042
INFO: 07-13 05:48:59: model.py:107 * 140114397865728 Train: epoch 96, average cost: 0.335774, average accuracy: 0.899740
INFO: 07-13 05:48:59: model.py:107 * 140114397865728 Train: epoch 97, average cost: 0.334488, average accuracy: 0.895833
INFO: 07-13 05:49:00: model.py:107 * 140114397865728 Train: epoch 98, average cost: 0.332654, average accuracy: 0.894531
INFO: 07-13 05:49:00: model.py:107 * 140114397865728 Train: epoch 99, average cost: 0.330539, average accuracy: 0.891927
INFO: 07-13 05:49:00: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-99
INFO: 07-13 05:49:01: model.py:107 * 140114397865728 Train: epoch 100, average cost: 0.328448, average accuracy: 0.893229
INFO: 07-13 05:49:01: model.py:107 * 140114397865728 Train: epoch 101, average cost: 0.326209, average accuracy: 0.895833
INFO: 07-13 05:49:02: model.py:107 * 140114397865728 Train: epoch 102, average cost: 0.323677, average accuracy: 0.899740
INFO: 07-13 05:49:02: model.py:107 * 140114397865728 Train: epoch 103, average cost: 0.320724, average accuracy: 0.899740
INFO: 07-13 05:49:03: model.py:107 * 140114397865728 Train: epoch 104, average cost: 0.317199, average accuracy: 0.901042
INFO: 07-13 05:49:03: model.py:107 * 140114397865728 Train: epoch 105, average cost: 0.313265, average accuracy: 0.907552
INFO: 07-13 05:49:04: model.py:107 * 140114397865728 Train: epoch 106, average cost: 0.309318, average accuracy: 0.910156
INFO: 07-13 05:49:04: model.py:107 * 140114397865728 Train: epoch 107, average cost: 0.305740, average accuracy: 0.910156
INFO: 07-13 05:49:05: model.py:107 * 140114397865728 Train: epoch 108, average cost: 0.302637, average accuracy: 0.911458
INFO: 07-13 05:49:05: model.py:107 * 140114397865728 Train: epoch 109, average cost: 0.299950, average accuracy: 0.911458
INFO: 07-13 05:49:05: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-109
INFO: 07-13 05:49:06: model.py:107 * 140114397865728 Train: epoch 110, average cost: 0.297540, average accuracy: 0.912760
INFO: 07-13 05:49:06: model.py:107 * 140114397865728 Train: epoch 111, average cost: 0.295327, average accuracy: 0.911458
INFO: 07-13 05:49:07: model.py:107 * 140114397865728 Train: epoch 112, average cost: 0.293345, average accuracy: 0.910156
INFO: 07-13 05:49:07: model.py:107 * 140114397865728 Train: epoch 113, average cost: 0.291716, average accuracy: 0.910156
INFO: 07-13 05:49:07: model.py:107 * 140114397865728 Train: epoch 114, average cost: 0.290582, average accuracy: 0.911458
INFO: 07-13 05:49:08: model.py:107 * 140114397865728 Train: epoch 115, average cost: 0.290137, average accuracy: 0.908854
INFO: 07-13 05:49:08: model.py:107 * 140114397865728 Train: epoch 116, average cost: 0.290795, average accuracy: 0.910156
INFO: 07-13 05:49:09: model.py:107 * 140114397865728 Train: epoch 117, average cost: 0.293188, average accuracy: 0.907552
INFO: 07-13 05:49:09: model.py:107 * 140114397865728 Train: epoch 118, average cost: 0.297799, average accuracy: 0.901042
INFO: 07-13 05:49:10: model.py:107 * 140114397865728 Train: epoch 119, average cost: 0.301919, average accuracy: 0.899740
INFO: 07-13 05:49:10: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-119
INFO: 07-13 05:49:10: model.py:107 * 140114397865728 Train: epoch 120, average cost: 0.295988, average accuracy: 0.898438
INFO: 07-13 05:49:11: model.py:107 * 140114397865728 Train: epoch 121, average cost: 0.280668, average accuracy: 0.908854
INFO: 07-13 05:49:12: model.py:107 * 140114397865728 Train: epoch 122, average cost: 0.271416, average accuracy: 0.917969
INFO: 07-13 05:49:12: model.py:107 * 140114397865728 Train: epoch 123, average cost: 0.262373, average accuracy: 0.919271
INFO: 07-13 05:49:12: model.py:107 * 140114397865728 Train: epoch 124, average cost: 0.257423, average accuracy: 0.925781
INFO: 07-13 05:49:13: model.py:107 * 140114397865728 Train: epoch 125, average cost: 0.255082, average accuracy: 0.928385
INFO: 07-13 05:49:13: model.py:107 * 140114397865728 Train: epoch 126, average cost: 0.253528, average accuracy: 0.930990
INFO: 07-13 05:49:14: model.py:107 * 140114397865728 Train: epoch 127, average cost: 0.251863, average accuracy: 0.933594
INFO: 07-13 05:49:14: model.py:107 * 140114397865728 Train: epoch 128, average cost: 0.249926, average accuracy: 0.932292
INFO: 07-13 05:49:15: model.py:107 * 140114397865728 Train: epoch 129, average cost: 0.247772, average accuracy: 0.932292
INFO: 07-13 05:49:15: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-129
INFO: 07-13 05:49:15: model.py:107 * 140114397865728 Train: epoch 130, average cost: 0.245534, average accuracy: 0.933594
INFO: 07-13 05:49:16: model.py:107 * 140114397865728 Train: epoch 131, average cost: 0.243272, average accuracy: 0.933594
INFO: 07-13 05:49:16: model.py:107 * 140114397865728 Train: epoch 132, average cost: 0.241031, average accuracy: 0.933594
INFO: 07-13 05:49:17: model.py:107 * 140114397865728 Train: epoch 133, average cost: 0.238807, average accuracy: 0.933594
INFO: 07-13 05:49:17: model.py:107 * 140114397865728 Train: epoch 134, average cost: 0.236601, average accuracy: 0.933594
INFO: 07-13 05:49:18: model.py:107 * 140114397865728 Train: epoch 135, average cost: 0.234447, average accuracy: 0.933594
INFO: 07-13 05:49:18: model.py:107 * 140114397865728 Train: epoch 136, average cost: 0.232368, average accuracy: 0.933594
INFO: 07-13 05:49:19: model.py:107 * 140114397865728 Train: epoch 137, average cost: 0.230380, average accuracy: 0.938802
INFO: 07-13 05:49:19: model.py:107 * 140114397865728 Train: epoch 138, average cost: 0.228487, average accuracy: 0.940104
INFO: 07-13 05:49:20: model.py:107 * 140114397865728 Train: epoch 139, average cost: 0.226699, average accuracy: 0.940104
INFO: 07-13 05:49:20: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-139
INFO: 07-13 05:49:20: model.py:107 * 140114397865728 Train: epoch 140, average cost: 0.225019, average accuracy: 0.938802
INFO: 07-13 05:49:21: model.py:107 * 140114397865728 Train: epoch 141, average cost: 0.223448, average accuracy: 0.936198
INFO: 07-13 05:49:21: model.py:107 * 140114397865728 Train: epoch 142, average cost: 0.221990, average accuracy: 0.940104
INFO: 07-13 05:49:22: model.py:107 * 140114397865728 Train: epoch 143, average cost: 0.220646, average accuracy: 0.940104
INFO: 07-13 05:49:22: model.py:107 * 140114397865728 Train: epoch 144, average cost: 0.219399, average accuracy: 0.940104
INFO: 07-13 05:49:23: model.py:107 * 140114397865728 Train: epoch 145, average cost: 0.218204, average accuracy: 0.940104
INFO: 07-13 05:49:23: model.py:107 * 140114397865728 Train: epoch 146, average cost: 0.216982, average accuracy: 0.937500
INFO: 07-13 05:49:24: model.py:107 * 140114397865728 Train: epoch 147, average cost: 0.215618, average accuracy: 0.936198
INFO: 07-13 05:49:24: model.py:107 * 140114397865728 Train: epoch 148, average cost: 0.213996, average accuracy: 0.937500
INFO: 07-13 05:49:25: model.py:107 * 140114397865728 Train: epoch 149, average cost: 0.212039, average accuracy: 0.937500
INFO: 07-13 05:49:25: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-149
INFO: 07-13 05:49:25: model.py:107 * 140114397865728 Train: epoch 150, average cost: 0.209745, average accuracy: 0.938802
INFO: 07-13 05:49:26: model.py:107 * 140114397865728 Train: epoch 151, average cost: 0.207219, average accuracy: 0.940104
INFO: 07-13 05:49:26: model.py:107 * 140114397865728 Train: epoch 152, average cost: 0.204637, average accuracy: 0.940104
INFO: 07-13 05:49:27: model.py:107 * 140114397865728 Train: epoch 153, average cost: 0.202251, average accuracy: 0.941406
INFO: 07-13 05:49:27: model.py:107 * 140114397865728 Train: epoch 154, average cost: 0.200310, average accuracy: 0.940104
INFO: 07-13 05:49:27: model.py:107 * 140114397865728 Train: epoch 155, average cost: 0.199096, average accuracy: 0.936198
INFO: 07-13 05:49:28: model.py:107 * 140114397865728 Train: epoch 156, average cost: 0.198498, average accuracy: 0.933594
INFO: 07-13 05:49:28: model.py:107 * 140114397865728 Train: epoch 157, average cost: 0.198319, average accuracy: 0.933594
INFO: 07-13 05:49:29: model.py:107 * 140114397865728 Train: epoch 158, average cost: 0.202391, average accuracy: 0.934896
INFO: 07-13 05:49:29: model.py:107 * 140114397865728 Train: epoch 159, average cost: 0.218657, average accuracy: 0.924479
INFO: 07-13 05:49:29: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-159
INFO: 07-13 05:49:30: model.py:107 * 140114397865728 Train: epoch 160, average cost: 0.225229, average accuracy: 0.925781
INFO: 07-13 05:49:30: model.py:107 * 140114397865728 Train: epoch 161, average cost: 0.205415, average accuracy: 0.929688
INFO: 07-13 05:49:31: model.py:107 * 140114397865728 Train: epoch 162, average cost: 0.204412, average accuracy: 0.929688
INFO: 07-13 05:49:31: model.py:107 * 140114397865728 Train: epoch 163, average cost: 0.198788, average accuracy: 0.936198
INFO: 07-13 05:49:32: model.py:107 * 140114397865728 Train: epoch 164, average cost: 0.193269, average accuracy: 0.940104
INFO: 07-13 05:49:32: model.py:107 * 140114397865728 Train: epoch 165, average cost: 0.190245, average accuracy: 0.942708
INFO: 07-13 05:49:33: model.py:107 * 140114397865728 Train: epoch 166, average cost: 0.186108, average accuracy: 0.941406
INFO: 07-13 05:49:33: model.py:107 * 140114397865728 Train: epoch 167, average cost: 0.181401, average accuracy: 0.944010
INFO: 07-13 05:49:34: model.py:107 * 140114397865728 Train: epoch 168, average cost: 0.177324, average accuracy: 0.945312
INFO: 07-13 05:49:34: model.py:107 * 140114397865728 Train: epoch 169, average cost: 0.174355, average accuracy: 0.944010
INFO: 07-13 05:49:34: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-169
INFO: 07-13 05:49:35: model.py:107 * 140114397865728 Train: epoch 170, average cost: 0.172266, average accuracy: 0.946615
INFO: 07-13 05:49:35: model.py:107 * 140114397865728 Train: epoch 171, average cost: 0.170699, average accuracy: 0.949219
INFO: 07-13 05:49:36: model.py:107 * 140114397865728 Train: epoch 172, average cost: 0.169309, average accuracy: 0.947917
INFO: 07-13 05:49:36: model.py:107 * 140114397865728 Train: epoch 173, average cost: 0.167879, average accuracy: 0.947917
INFO: 07-13 05:49:37: model.py:107 * 140114397865728 Train: epoch 174, average cost: 0.166327, average accuracy: 0.947917
INFO: 07-13 05:49:37: model.py:107 * 140114397865728 Train: epoch 175, average cost: 0.164675, average accuracy: 0.947917
INFO: 07-13 05:49:38: model.py:107 * 140114397865728 Train: epoch 176, average cost: 0.162992, average accuracy: 0.947917
INFO: 07-13 05:49:38: model.py:107 * 140114397865728 Train: epoch 177, average cost: 0.161334, average accuracy: 0.947917
INFO: 07-13 05:49:39: model.py:107 * 140114397865728 Train: epoch 178, average cost: 0.159728, average accuracy: 0.947917
INFO: 07-13 05:49:39: model.py:107 * 140114397865728 Train: epoch 179, average cost: 0.158195, average accuracy: 0.949219
INFO: 07-13 05:49:39: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-179
INFO: 07-13 05:49:40: model.py:107 * 140114397865728 Train: epoch 180, average cost: 0.156748, average accuracy: 0.947917
INFO: 07-13 05:49:40: model.py:107 * 140114397865728 Train: epoch 181, average cost: 0.155421, average accuracy: 0.946615
INFO: 07-13 05:49:41: model.py:107 * 140114397865728 Train: epoch 182, average cost: 0.154250, average accuracy: 0.949219
INFO: 07-13 05:49:41: model.py:107 * 140114397865728 Train: epoch 183, average cost: 0.153260, average accuracy: 0.950521
INFO: 07-13 05:49:42: model.py:107 * 140114397865728 Train: epoch 184, average cost: 0.152467, average accuracy: 0.949219
INFO: 07-13 05:49:42: model.py:107 * 140114397865728 Train: epoch 185, average cost: 0.151896, average accuracy: 0.950521
INFO: 07-13 05:49:43: model.py:107 * 140114397865728 Train: epoch 186, average cost: 0.151789, average accuracy: 0.950521
INFO: 07-13 05:49:43: model.py:107 * 140114397865728 Train: epoch 187, average cost: 0.153187, average accuracy: 0.949219
INFO: 07-13 05:49:43: model.py:107 * 140114397865728 Train: epoch 188, average cost: 0.159610, average accuracy: 0.947917
INFO: 07-13 05:49:44: model.py:107 * 140114397865728 Train: epoch 189, average cost: 0.164103, average accuracy: 0.946615
INFO: 07-13 05:49:44: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-189
INFO: 07-13 05:49:44: model.py:107 * 140114397865728 Train: epoch 190, average cost: 0.153677, average accuracy: 0.958333
INFO: 07-13 05:49:45: model.py:107 * 140114397865728 Train: epoch 191, average cost: 0.153694, average accuracy: 0.960938
INFO: 07-13 05:49:45: model.py:107 * 140114397865728 Train: epoch 192, average cost: 0.164557, average accuracy: 0.949219
INFO: 07-13 05:49:46: model.py:107 * 140114397865728 Train: epoch 193, average cost: 0.164621, average accuracy: 0.944010
INFO: 07-13 05:49:46: model.py:107 * 140114397865728 Train: epoch 194, average cost: 0.148349, average accuracy: 0.958333
INFO: 07-13 05:49:47: model.py:107 * 140114397865728 Train: epoch 195, average cost: 0.157004, average accuracy: 0.951823
INFO: 07-13 05:49:47: model.py:107 * 140114397865728 Train: epoch 196, average cost: 0.156266, average accuracy: 0.957031
INFO: 07-13 05:49:48: model.py:107 * 140114397865728 Train: epoch 197, average cost: 0.147878, average accuracy: 0.959635
INFO: 07-13 05:49:48: model.py:107 * 140114397865728 Train: epoch 198, average cost: 0.138221, average accuracy: 0.964844
INFO: 07-13 05:49:49: model.py:107 * 140114397865728 Train: epoch 199, average cost: 0.128520, average accuracy: 0.966146
INFO: 07-13 05:49:49: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-199
INFO: 07-13 05:49:49: model.py:107 * 140114397865728 Train: epoch 200, average cost: 0.121668, average accuracy: 0.967448
INFO: 07-13 05:49:50: model.py:107 * 140114397865728 Train: epoch 201, average cost: 0.117491, average accuracy: 0.967448
INFO: 07-13 05:49:50: model.py:107 * 140114397865728 Train: epoch 202, average cost: 0.115036, average accuracy: 0.968750
INFO: 07-13 05:49:51: model.py:107 * 140114397865728 Train: epoch 203, average cost: 0.113341, average accuracy: 0.970052
INFO: 07-13 05:49:51: model.py:107 * 140114397865728 Train: epoch 204, average cost: 0.111901, average accuracy: 0.968750
INFO: 07-13 05:49:52: model.py:107 * 140114397865728 Train: epoch 205, average cost: 0.110533, average accuracy: 0.967448
INFO: 07-13 05:49:52: model.py:107 * 140114397865728 Train: epoch 206, average cost: 0.109177, average accuracy: 0.968750
INFO: 07-13 05:49:53: model.py:107 * 140114397865728 Train: epoch 207, average cost: 0.107819, average accuracy: 0.970052
INFO: 07-13 05:49:53: model.py:107 * 140114397865728 Train: epoch 208, average cost: 0.106465, average accuracy: 0.968750
INFO: 07-13 05:49:54: model.py:107 * 140114397865728 Train: epoch 209, average cost: 0.105129, average accuracy: 0.970052
INFO: 07-13 05:49:54: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-209
INFO: 07-13 05:49:54: model.py:107 * 140114397865728 Train: epoch 210, average cost: 0.103818, average accuracy: 0.970052
INFO: 07-13 05:49:54: model.py:107 * 140114397865728 Train: epoch 211, average cost: 0.102523, average accuracy: 0.972656
INFO: 07-13 05:49:55: model.py:107 * 140114397865728 Train: epoch 212, average cost: 0.101254, average accuracy: 0.972656
INFO: 07-13 05:49:55: model.py:107 * 140114397865728 Train: epoch 213, average cost: 0.100004, average accuracy: 0.972656
INFO: 07-13 05:49:56: model.py:107 * 140114397865728 Train: epoch 214, average cost: 0.098774, average accuracy: 0.973958
INFO: 07-13 05:49:56: model.py:107 * 140114397865728 Train: epoch 215, average cost: 0.097565, average accuracy: 0.975260
INFO: 07-13 05:49:57: model.py:107 * 140114397865728 Train: epoch 216, average cost: 0.096376, average accuracy: 0.975260
INFO: 07-13 05:49:57: model.py:107 * 140114397865728 Train: epoch 217, average cost: 0.095215, average accuracy: 0.975260
INFO: 07-13 05:49:58: model.py:107 * 140114397865728 Train: epoch 218, average cost: 0.094087, average accuracy: 0.972656
INFO: 07-13 05:49:58: model.py:107 * 140114397865728 Train: epoch 219, average cost: 0.093006, average accuracy: 0.973958
INFO: 07-13 05:49:58: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-219
INFO: 07-13 05:49:59: model.py:107 * 140114397865728 Train: epoch 220, average cost: 0.091997, average accuracy: 0.973958
INFO: 07-13 05:49:59: model.py:107 * 140114397865728 Train: epoch 221, average cost: 0.091084, average accuracy: 0.975260
INFO: 07-13 05:50:00: model.py:107 * 140114397865728 Train: epoch 222, average cost: 0.090321, average accuracy: 0.973958
INFO: 07-13 05:50:00: model.py:107 * 140114397865728 Train: epoch 223, average cost: 0.089819, average accuracy: 0.975260
INFO: 07-13 05:50:01: model.py:107 * 140114397865728 Train: epoch 224, average cost: 0.089848, average accuracy: 0.973958
INFO: 07-13 05:50:01: model.py:107 * 140114397865728 Train: epoch 225, average cost: 0.091241, average accuracy: 0.973958
INFO: 07-13 05:50:02: model.py:107 * 140114397865728 Train: epoch 226, average cost: 0.096940, average accuracy: 0.970052
INFO: 07-13 05:50:02: model.py:107 * 140114397865728 Train: epoch 227, average cost: 0.106213, average accuracy: 0.963542
INFO: 07-13 05:50:03: model.py:107 * 140114397865728 Train: epoch 228, average cost: 0.111724, average accuracy: 0.968750
INFO: 07-13 05:50:03: model.py:107 * 140114397865728 Train: epoch 229, average cost: 0.125354, average accuracy: 0.963542
INFO: 07-13 05:50:03: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-229
INFO: 07-13 05:50:04: model.py:107 * 140114397865728 Train: epoch 230, average cost: 0.143199, average accuracy: 0.950521
INFO: 07-13 05:50:04: model.py:107 * 140114397865728 Train: epoch 231, average cost: 0.148402, average accuracy: 0.936198
INFO: 07-13 05:50:05: model.py:107 * 140114397865728 Train: epoch 232, average cost: 0.115296, average accuracy: 0.947917
INFO: 07-13 05:50:05: model.py:107 * 140114397865728 Train: epoch 233, average cost: 0.097335, average accuracy: 0.963542
INFO: 07-13 05:50:06: model.py:107 * 140114397865728 Train: epoch 234, average cost: 0.092931, average accuracy: 0.971354
INFO: 07-13 05:50:06: model.py:107 * 140114397865728 Train: epoch 235, average cost: 0.090197, average accuracy: 0.972656
INFO: 07-13 05:50:06: model.py:107 * 140114397865728 Train: epoch 236, average cost: 0.087572, average accuracy: 0.972656
INFO: 07-13 05:50:07: model.py:107 * 140114397865728 Train: epoch 237, average cost: 0.084076, average accuracy: 0.975260
INFO: 07-13 05:50:07: model.py:107 * 140114397865728 Train: epoch 238, average cost: 0.081117, average accuracy: 0.977865
INFO: 07-13 05:50:08: model.py:107 * 140114397865728 Train: epoch 239, average cost: 0.078545, average accuracy: 0.979167
INFO: 07-13 05:50:08: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-239
INFO: 07-13 05:50:08: model.py:107 * 140114397865728 Train: epoch 240, average cost: 0.075903, average accuracy: 0.979167
INFO: 07-13 05:50:09: model.py:107 * 140114397865728 Train: epoch 241, average cost: 0.073264, average accuracy: 0.980469
INFO: 07-13 05:50:09: model.py:107 * 140114397865728 Train: epoch 242, average cost: 0.070776, average accuracy: 0.981771
INFO: 07-13 05:50:10: model.py:107 * 140114397865728 Train: epoch 243, average cost: 0.068556, average accuracy: 0.988281
INFO: 07-13 05:50:10: model.py:107 * 140114397865728 Train: epoch 244, average cost: 0.066624, average accuracy: 0.988281
INFO: 07-13 05:50:11: model.py:107 * 140114397865728 Train: epoch 245, average cost: 0.064953, average accuracy: 0.988281
INFO: 07-13 05:50:11: model.py:107 * 140114397865728 Train: epoch 246, average cost: 0.063499, average accuracy: 0.986979
INFO: 07-13 05:50:12: model.py:107 * 140114397865728 Train: epoch 247, average cost: 0.062218, average accuracy: 0.986979
INFO: 07-13 05:50:12: model.py:107 * 140114397865728 Train: epoch 248, average cost: 0.061073, average accuracy: 0.988281
INFO: 07-13 05:50:13: model.py:107 * 140114397865728 Train: epoch 249, average cost: 0.060034, average accuracy: 0.988281
INFO: 07-13 05:50:13: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-249
INFO: 07-13 05:50:13: model.py:107 * 140114397865728 Train: epoch 250, average cost: 0.059080, average accuracy: 0.986979
INFO: 07-13 05:50:14: model.py:107 * 140114397865728 Train: epoch 251, average cost: 0.058196, average accuracy: 0.988281
INFO: 07-13 05:50:14: model.py:107 * 140114397865728 Train: epoch 252, average cost: 0.057366, average accuracy: 0.988281
INFO: 07-13 05:50:15: model.py:107 * 140114397865728 Train: epoch 253, average cost: 0.056579, average accuracy: 0.988281
INFO: 07-13 05:50:15: model.py:107 * 140114397865728 Train: epoch 254, average cost: 0.055825, average accuracy: 0.989583
INFO: 07-13 05:50:16: model.py:107 * 140114397865728 Train: epoch 255, average cost: 0.055094, average accuracy: 0.989583
INFO: 07-13 05:50:16: model.py:107 * 140114397865728 Train: epoch 256, average cost: 0.054381, average accuracy: 0.989583
INFO: 07-13 05:50:17: model.py:107 * 140114397865728 Train: epoch 257, average cost: 0.053679, average accuracy: 0.989583
INFO: 07-13 05:50:17: model.py:107 * 140114397865728 Train: epoch 258, average cost: 0.052985, average accuracy: 0.989583
INFO: 07-13 05:50:18: model.py:107 * 140114397865728 Train: epoch 259, average cost: 0.052299, average accuracy: 0.989583
INFO: 07-13 05:50:18: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-259
INFO: 07-13 05:50:18: model.py:107 * 140114397865728 Train: epoch 260, average cost: 0.051618, average accuracy: 0.989583
INFO: 07-13 05:50:19: model.py:107 * 140114397865728 Train: epoch 261, average cost: 0.050952, average accuracy: 0.990885
INFO: 07-13 05:50:19: model.py:107 * 140114397865728 Train: epoch 262, average cost: 0.050317, average accuracy: 0.990885
INFO: 07-13 05:50:20: model.py:107 * 140114397865728 Train: epoch 263, average cost: 0.049732, average accuracy: 0.990885
INFO: 07-13 05:50:20: model.py:107 * 140114397865728 Train: epoch 264, average cost: 0.049227, average accuracy: 0.990885
INFO: 07-13 05:50:20: model.py:107 * 140114397865728 Train: epoch 265, average cost: 0.048857, average accuracy: 0.990885
INFO: 07-13 05:50:21: model.py:107 * 140114397865728 Train: epoch 266, average cost: 0.048690, average accuracy: 0.990885
INFO: 07-13 05:50:21: model.py:107 * 140114397865728 Train: epoch 267, average cost: 0.048727, average accuracy: 0.990885
INFO: 07-13 05:50:22: model.py:107 * 140114397865728 Train: epoch 268, average cost: 0.048668, average accuracy: 0.990885
INFO: 07-13 05:50:22: model.py:107 * 140114397865728 Train: epoch 269, average cost: 0.048280, average accuracy: 0.990885
INFO: 07-13 05:50:22: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-269
INFO: 07-13 05:50:23: model.py:107 * 140114397865728 Train: epoch 270, average cost: 0.048154, average accuracy: 0.990885
INFO: 07-13 05:50:23: model.py:107 * 140114397865728 Train: epoch 271, average cost: 0.049162, average accuracy: 0.989583
INFO: 07-13 05:50:24: model.py:107 * 140114397865728 Train: epoch 272, average cost: 0.052616, average accuracy: 0.989583
INFO: 07-13 05:50:24: model.py:107 * 140114397865728 Train: epoch 273, average cost: 0.062186, average accuracy: 0.986979
INFO: 07-13 05:50:25: model.py:107 * 140114397865728 Train: epoch 274, average cost: 0.064216, average accuracy: 0.985677
INFO: 07-13 05:50:25: model.py:107 * 140114397865728 Train: epoch 275, average cost: 0.073691, average accuracy: 0.981771
INFO: 07-13 05:50:26: model.py:107 * 140114397865728 Train: epoch 276, average cost: 0.082758, average accuracy: 0.977865
INFO: 07-13 05:50:26: model.py:107 * 140114397865728 Train: epoch 277, average cost: 0.089908, average accuracy: 0.968750
INFO: 07-13 05:50:27: model.py:107 * 140114397865728 Train: epoch 278, average cost: 0.117145, average accuracy: 0.947917
INFO: 07-13 05:50:27: model.py:107 * 140114397865728 Train: epoch 279, average cost: 0.089141, average accuracy: 0.968750
INFO: 07-13 05:50:27: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-279
INFO: 07-13 05:50:28: model.py:107 * 140114397865728 Train: epoch 280, average cost: 0.073769, average accuracy: 0.976562
INFO: 07-13 05:50:28: model.py:107 * 140114397865728 Train: epoch 281, average cost: 0.063202, average accuracy: 0.984375
INFO: 07-13 05:50:29: model.py:107 * 140114397865728 Train: epoch 282, average cost: 0.063268, average accuracy: 0.981771
INFO: 07-13 05:50:29: model.py:107 * 140114397865728 Train: epoch 283, average cost: 0.061688, average accuracy: 0.983073
INFO: 07-13 05:50:30: model.py:107 * 140114397865728 Train: epoch 284, average cost: 0.059305, average accuracy: 0.984375
INFO: 07-13 05:50:30: model.py:107 * 140114397865728 Train: epoch 285, average cost: 0.057513, average accuracy: 0.984375
INFO: 07-13 05:50:31: model.py:107 * 140114397865728 Train: epoch 286, average cost: 0.056200, average accuracy: 0.986979
INFO: 07-13 05:50:31: model.py:107 * 140114397865728 Train: epoch 287, average cost: 0.054862, average accuracy: 0.986979
INFO: 07-13 05:50:31: model.py:107 * 140114397865728 Train: epoch 288, average cost: 0.053535, average accuracy: 0.986979
INFO: 07-13 05:50:32: model.py:107 * 140114397865728 Train: epoch 289, average cost: 0.052065, average accuracy: 0.988281
INFO: 07-13 05:50:32: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-289
INFO: 07-13 05:50:32: model.py:107 * 140114397865728 Train: epoch 290, average cost: 0.050557, average accuracy: 0.988281
INFO: 07-13 05:50:33: model.py:107 * 140114397865728 Train: epoch 291, average cost: 0.049154, average accuracy: 0.988281
INFO: 07-13 05:50:33: model.py:107 * 140114397865728 Train: epoch 292, average cost: 0.047842, average accuracy: 0.989583
INFO: 07-13 05:50:34: model.py:107 * 140114397865728 Train: epoch 293, average cost: 0.046646, average accuracy: 0.989583
INFO: 07-13 05:50:34: model.py:107 * 140114397865728 Train: epoch 294, average cost: 0.045621, average accuracy: 0.990885
INFO: 07-13 05:50:35: model.py:107 * 140114397865728 Train: epoch 295, average cost: 0.044743, average accuracy: 0.990885
INFO: 07-13 05:50:35: model.py:107 * 140114397865728 Train: epoch 296, average cost: 0.043806, average accuracy: 0.990885
INFO: 07-13 05:50:36: model.py:107 * 140114397865728 Train: epoch 297, average cost: 0.042730, average accuracy: 0.992188
INFO: 07-13 05:50:36: model.py:107 * 140114397865728 Train: epoch 298, average cost: 0.041650, average accuracy: 0.993490
INFO: 07-13 05:50:37: model.py:107 * 140114397865728 Train: epoch 299, average cost: 0.040772, average accuracy: 0.993490
INFO: 07-13 05:50:37: model_io.py:83 * 140114397865728 Saved inference model to ./models/infer-epoch-299
INFO: 07-13 05:50:37: model.py:181 * 140114397865728 Costs 144.99s.

可以看到：

average accuracy单调增长
average cost单调下降
300 epoch耗时仅比使用mkldnn多了不到5秒

模型搭建代码如下：

    def _build_model(self, is_test=False):
        """模型搭建"""
        data = fluid.layers.data(name='data', shape=[128], dtype='float32')
        label = fluid.layers.data(name='label', shape=[1], dtype='int64')

        fc_1 = fluid.layers.fc(input=data, size=470, act='elu', use_mkldnn=False,
            is_test=is_test)
        fc_2 = fluid.layers.fc(input=fc_1, size=470, act='elu', use_mkldnn=False,
            is_test=is_test)

        prediction = fluid.layers.fc(input=fc_2, size=self.class_dim,
            act='softmax', name='prediction', use_mkldnn=False, is_test=is_test)
        cost = fluid.layers.cross_entropy(input=prediction, label=label)
        avg_cost = fluid.layers.mean(x=cost, name='avg_cost')
        acc = fluid.layers.accuracy(input=prediction, label=label)

模型训练代码如下：

    def fit(self, train_reader):
        """模型训练
        执行模型训练过程。

        Args:
            train_reader: generator, 训练数据batch generator
        """
        # 创建模型
        cost, acc, prediction = self._build_model()
        # 优化器配置
        optimizer = fluid.optimizer.Adamax(learning_rate=self.lr)
        optimizer.minimize(cost)
        # 初始化模型
        exe = fluid.Executor(self.place)
        exe.run(fluid.default_startup_program())
        # 准备模型训练环境
        train_exe = fluid.ParallelExecutor(use_cuda=False, loss_name=cost.name)
        fetch_names = [cost.name, acc.name, prediction.name]
        feeder = fluid.DataFeeder(place=self.place, feed_list=['data', 'label'])
        # 训练模型
        for epoch_id in range(self.epoch_num):
            # 训练一轮
            total_acc = 0
            total_cost = 0
            data_count = 0
            for batch_id, data in enumerate(train_reader()):
                cost, acc, pred = train_exe.run(
                    fetch_list=fetch_names,
                    feed=feeder.feed(data))
                cost = np.array(cost)[0]
                acc = np.array(acc)[0]
                data_size = len(data)
                total_acc += data_size * acc
                total_cost += data_size * cost
                data_count += data_size
            avg_cost = total_cost / data_count
            avg_acc = total_acc / data_count
            self.logger.info('Train: epoch %d, average cost: %f, average accuracy: %f' %
                (epoch_id, avg_cost, avg_acc))
            # 训练速度很快故只保存推理用模型
            if (epoch_id + 1) % 10 == 0:
                self.infer_model_io.save(epoch_id=epoch_id, feed_names=['data'],
                    fetch_targets=[prediction], executor=exe)

我的实验中mkldnn优化的可用性值得怀疑，请问我有什么操作不当吗？

PaddlePaddle / Paddle 大约 2 年 前同步成功

use_mkldnn=True表现异常

PaddlePaddle / Paddle
大约 2 年前同步成功