Created by: yeyupiaoling

• 系统：Ubuntu 16.04
• 显卡：2张 2080ti
• 版本：PaddlePaddle-GPU 1.8.3.post107

F0823 14:54:47.975309 16507 all_reduce_op_handle.cc:192] cudaStreamSynchronize misaligned address
*** Check failure stack trace: ***
    @     0x7ff2ccc25d4d  google::LogMessage::Fail()
    @     0x7ff2ccc297fc  google::LogMessage::SendToLog()
    @     0x7ff2ccc25873  google::LogMessage::Flush()
    @     0x7ff2ccc2ad0e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7ff2cfedb35c  paddle::framework::details::AllReduceOpHandle::SyncNCCLAllReduce()
    @     0x7ff2cfedb3f4  paddle::framework::details::AllReduceOpHandle::NCCLAllReduceFunc()
    @     0x7ff2cfedbf66  paddle::framework::details::AllReduceOpHandle::AllReduceFunc()
    @     0x7ff2cfe3bf1c  paddle::framework::details::FusedAllReduceOpHandle::FusedAllReduceFunc()
    @     0x7ff2cfe3d4fe  paddle::framework::details::FusedAllReduceOpHandle::RunImpl()
    @     0x7ff2cfe77c31  paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync()
    @     0x7ff2cfe7572f  paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp()
    @     0x7ff2cfe759f4  _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data
    @     0x7ff2ccc83213  std::_Function_handler<>::_M_invoke()
    @     0x7ff2cca7ee67  std::__future_base::_State_base::_M_do_set()
    @     0x7ff3105d8a99  __pthread_once_slow
    @     0x7ff2cfe71bc2  _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv
    @     0x7ff2cca812c4  _ZZN10ThreadPoolC1EmENKUlvE_clEv
    @     0x7ff3011d7421  execute_native_thread_routine_compat
    @     0x7ff3105d16ba  start_thread
    @     0x7ff30fbf74dd  clone
    @              (nil)  (unknown)