Check failed: square_sum_x > 0 && square_sum_y > 0
Created by: youan1
RT,问题如题,在paddle里有计算两个embedding向量相似度的操作,第一次运行成功,但是在新数据上运行时,出如上错误,脚本如下,请问是什么问题导致的:
脚本如下:
# embedding layer
AvgPooling = paddle.pooling.Avg
emb1_1 = paddle.layer.embedding(name="emb1_1", input=data1, size=64, param_attr=paddle.attr.Param(
initial_std=emb_layer_init_std[1]),
layer_attr=paddle.attr.ExtraAttr(error_clipping_threshold=10.0))
emb1_1_pooling = paddle.layer.pooling(name="emb1_1_pooling", input=emb1_1,
pooling_type=AvgPooling())
emb1_1_size=64
#emb2_1 = paddle.layer.fc(input=emb1_1, size=64,
# act=paddle.activation.Relu(),
# param_attr=paddle.attr.Param(
# initial_std=emb_layer_init_std[1]))
#emb2_1_size=64
emb1_2 = paddle.layer.embedding(name="emb1_2",input=data2, size=64, param_attr=paddle.attr.Param(
initial_std=emb_layer_init_std[1]),
layer_attr=paddle.attr.ExtraAttr(error_clipping_threshold=10.0))
emb1_2_pooling = paddle.layer.pooling(name="emb1_2_pooling",input=emb1_2,
pooling_type=AvgPooling())
emb1_2_size=64
#emb2_2 = paddle.layer.fc(input=emb1_2, size=64,
# act=paddle.activation.Relu(),
# param_attr=paddle.attr.Param(
# initial_std=emb_layer_init_std[1]))
#emb2_2_size=64
emb1_2_cos_sim=paddle.layer.cos_sim(name="emb1_2_cos_sim",
a=emb1_1_pooling,
b=emb1_2_pooling,
scale=1)
emb1_2_cos_sim_size = 1
错误为:
Mon Jan 8 16:29:54 2018[1,43]:+ '[' --local == --ports_num_for_sparse ']' Mon Jan 8 16:29:54 2018[1,43]:+ export PADDLE_NUM_GRADIENT_SERVERS=100 Mon Jan 8 16:29:54 2018[1,43]:+ PADDLE_NUM_GRADIENT_SERVERS=100 Mon Jan 8 16:29:54 2018[1,43]:+ export PADDLE_TRAINER_ID=43 Mon Jan 8 16:29:54 2018[1,43]:+ PADDLE_TRAINER_ID=43 Mon Jan 8 16:29:54 2018[1,43]:+ export PADDLE_PSERVERS=10.86.93.39,10.86.90.39,10.86.90.40,10.86.90.42,10.86.90.43,10.86.91.11,10.86.91.12,10.86.91.13,10.86.91.14,10.86.91.15,10.86.92.40,10.86.92.42,10.86.92.43,10.86.93.11,10.86.93.12,10.86.93.13,10.86.93.16,10.86.93.17,10.86.93.18,10.86.93.19,10.86.93.20,10.86.93.21,10.86.93.22,10.86.93.24,10.86.93.25,10.86.93.27,10.86.93.28,10.86.93.29,10.86.93.31,10.86.93.32,10.86.93.33,10.86.93.34,10.86.93.35,10.86.93.36,10.86.93.37,10.86.93.38,10.86.90.37,10.86.93.40,10.86.93.42,10.86.93.43,10.86.94.11,10.86.94.13,10.86.94.14,10.86.94.15,10.86.94.17,10.86.94.18,10.86.94.19,10.86.94.21,10.86.94.23,10.86.94.26,10.86.94.27,10.86.94.28,10.86.94.29,10.86.94.31,10.86.94.32,10.86.94.35,10.86.94.38,10.86.94.39,10.86.94.40,10.86.94.41,10.86.94.43,10.86.95.11,10.86.95.13,10.86.95.14,10.86.95.18,10.86.95.19,10.86.95.20,10.86.95.21,10.86.95.23,10.86.95.25,10.86.95.26,10.86.95.27,10.86.95.28,10.86.95.29,10.86.95.30,10.86.95.32,10.86.95.33,10.86.95.34,10.86.95.36,10.86.95.37,10.86.95.38,10.86.95.39,10.86.95.42,10.86.96.11,10.86.96.12,10.86.96.13,10.86.96.14,10.86.96.15,10.86.96.16,10.86.96.17,10.86.96.18,10.86.96.20,10.86.96.21,10.86.96.24,10.86.96.25,10.86.96.27,10.86.96.28,10.86.96.29,10.86.96.30,10.86.96.31 Mon Jan 8 16:29:54 2018[1,43]:+ PADDLE_PSERVERS=10.86.93.39,10.86.90.39,10.86.90.40,10.86.90.42,10.86.90.43,10.86.91.11,10.86.91.12,10.86.91.13,10.86.91.14,10.86.91.15,10.86.92.40,10.86.92.42,10.86.92.43,10.86.93.11,10.86.93.12,10.86.93.13,10.86.93.16,10.86.93.17,10.86.93.18,10.86.93.19,10.86.93.20,10.86.93.21,10.86.93.22,10.86.93.24,10.86.93.25,10.86.93.27,10.86.93.28,10.86.93.29,10.86.93.31,10.86.93.32,10.86.93.33,10.86.93.34,10.86.93.35,10.86.93.36,10.86.93.37,10.86.93.38,10.86.90.37,10.86.93.40,10.86.93.42,10.86.93.43,10.86.94.11,10.86.94.13,10.86.94.14,10.86.94.15,10.86.94.17,10.86.94.18,10.86.94.19,10.86.94.21,10.86.94.23,10.86.94.26,10.86.94.27,10.86.94.28,10.86.94.29,10.86.94.31,10.86.94.32,10.86.94.35,10.86.94.38,10.86.94.39,10.86.94.40,10.86.94.41,10.86.94.43,10.86.95.11,10.86.95.13,10.86.95.14,10.86.95.18,10.86.95.19,10.86.95.20,10.86.95.21,10.86.95.23,10.86.95.25,10.86.95.26,10.86.95.27,10.86.95.28,10.86.95.29,10.86.95.30,10.86.95.32,10.86.95.33,10.86.95.34,10.86.95.36,10.86.95.37,10.86.95.38,10.86.95.39,10.86.95.42,10.86.96.11,10.86.96.12,10.86.96.13,10.86.96.14,10.86.96.15,10.86.96.16,10.86.96.17,10.86.96.18,10.86.96.20,10.86.96.21,10.86.96.24,10.86.96.25,10.86.96.27,10.86.96.28,10.86.96.29,10.86.96.30,10.86.96.31 Mon Jan 8 16:29:54 2018[1,43]:+ python27-gcc482/bin/python conf/trainer_config.conf Mon Jan 8 16:42:26 2018[1,62]:F0108 16:42:26.696537 3819 CosSimOp.cpp:54] Check failed: square_sum_x > 0 && square_sum_y > 0 Mon Jan 8 16:42:26 2018[1,29]:F0108 16:42:26.713567 60664 CosSimOp.cpp:54] Check failed: square_sum_x > 0 && square_sum_y > 0 Mon Jan 8 16:42:26 2018[1,62]:*** Check failure stack trace: *** Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080170a27d google::LogMessage::Fail() Mon Jan 8 16:42:26 2018[1,29]:*** Check failure stack trace: *** Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080170dd2c google::LogMessage::SendToLog() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f0801709da3 google::LogMessage::Flush() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a87827d google::LogMessage::Fail() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a87bd2c google::LogMessage::SendToLog() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080170f23e google::LogMessageFatal::~LogMessageFatal() Mon Jan 8 16:42:26 2018[1,62]:F0108 16:42:26.752230 3815 CosSimOp.cpp:54] Check failed: square_sum_x > 0 && square_sum_y > 0 Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a877da3 google::LogMessage::Flush() Mon Jan 8 16:42:26 2018[1,62]:*** Check failure stack trace: *** Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f0801535970 paddle::CosSimForward<>() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a87d23e google::LogMessageFatal::~LogMessageFatal() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080170a27d google::LogMessage::Fail() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080153a8e5 paddle::CosSimForwardFunc<>::calc() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a6a3970 paddle::CosSimForward<>() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080170dd2c google::LogMessage::SendToLog() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080147ea2f paddle::CosSimLayer::forward() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a6a88e5 paddle::CosSimForwardFunc<>::calc() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f0801709da3 google::LogMessage::Flush() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f08014ca562 paddle::NeuralNetwork::forward() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a5eca2f paddle::CosSimLayer::forward() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a638562 paddle::NeuralNetwork::forward() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f080170f23e google::LogMessageFatal::~LogMessageFatal() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f08014d31e4 paddle::TrainerThread::forward() Mon Jan 8 16:42:26 2018[1,62]:F0108 16:42:26.752230 3815 CosSimOp.cpp:54] Check failed: square_sum_x > 0 && square_sum_y > 0 F0108 16:42:26.797132 3816 CosSimOp.cpp:54] Check failed: square_sum_x > 0 && square_sum_y > 0 Mon Jan 8 16:42:26 2018[1,62]:*** Check failure stack trace: *** Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f0801535970 paddle::CosSimForward<>() Mon Jan 8 16:42:26 2018[1,29]: @ 0x7f338a6411e4 paddle::TrainerThread::forward() Mon Jan 8 16:42:26 2018[1,62]: @ 0x7f08014d4665 paddle::TrainerThread::computeThread()