Add multi-process version of xmap_reader to take place of multi-thread version for a significant acceleration and add seqbin data parser for internal 1w data training. !355
Created by: xinghai-sun
Resolved #354 (closed)
Done:
- Add multi-process version of
xmap_reader_mpto take place of multi-thread version ofpaddle.reader.xmap_reader. - Add seqbin data parser to adapt training with internal 1w English dataset.
This speeds up the training with the internal 1w dataset by more than 3X, and now the GPU utilization rises back to 81% (from 15%).