diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index f60c3b843d20f8cd0144f44065a2fbf7fd43d4a7..241902cca405c2b6df102257ea8ac8c5c292d4c8 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -23,8 +23,8 @@ 在数据集可以被训练之前,文件需要预先被转换成PaddlePaddle集群内部的存储格式(RecordIO)。我们提供两个转换方式: -- 提供给用户本地转换的库,用户可以编写程序完成转换。 -- 用户可以上传自己的数据集,在集群运行MapReduce job完成转换。 +1. 用户在本地转换好再上传 +1. 用户上传数据后,在机群上运行转换程序 转换生成的文件名会是以下格式: diff --git a/doc/design/cluster_train/master_process.md b/doc/design/cluster_train/master_process.md index 949811b4f76415c0f0130f3a06e0f40a18cd5557..e0be8df6343bc55c4a2422910dafbd8092d0d7ee 100644 --- a/doc/design/cluster_train/master_process.md +++ b/doc/design/cluster_train/master_process.md @@ -1,12 +1,12 @@ # Design Doc: Master Process -For an overview of master process' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master process in more details. The master will be implemented in [golang](https://golang.org/). +For an overview of master process' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master process in more details. The master will be implemented in [Go](https://golang.org/). ## Dataset -A dataset is represented by a list of files in *RecordIO* format on the distributed filesystem, each RecordIO file consists of multiple *blocks*, and each block has multiple data instances. +A dataset is a list of files in *RecordIO* format. A RecordIO file consists of chunks, whereas each chunk consists some records. ## Task Queue @@ -14,7 +14,7 @@ As mentioned in [distributed training design doc](./README.md), a *task* is a da ### Task Queue Creation -1. Each trainer will make an RPC call (using [golang rpc](https://golang.org/pkg/net/rpc/)) to the master process, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master process the same dataset, only the first RPC call will be honored. +1. Each trainer will make an RPC call (using Go's [rpc](https://golang.org/pkg/net/rpc/) package) to the master process, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master process the same dataset, only the first RPC call will be honored. The RPC interface is: ```go