提交 adaa9c5b 编写于 作者: Y Yancey1989

update by comments

上级 d05071f0
...@@ -2,29 +2,29 @@ ...@@ -2,29 +2,29 @@
We introduced how to create a PaddlePaddle Job with a single node on Kuberentes in the We introduced how to create a PaddlePaddle Job with a single node on Kuberentes in the
previous document. previous document.
In this article, we will introduce how to craete a PaddlePaddle job with multiple nodes In this article, we will introduce how to create a PaddlePaddle job with multiple nodes
on Kubernetes cluster. on Kubernetes cluster.
## Overall Architecture ## Overall Architecture
Before creating a training job, the users need to deploy the Python scripts and Before creating a training job, the users need to slice the training data and deploy
training data which have already been sliced on the precast path in the distributed file the Python scripts along with it into the distributed file system
system(We can use the different type of Kuberentes Volumes to mount different distributed (We can use the different type of Kuberentes Volumes to mount different distributed
file system). Before start training, The program would copy the training data into the file systems). Before training starts, The program will copy the training data into the
Container and also save the models at the same path during training. The global architecture Container and also save the models at the same path during training. The global architecture
is as follows: is as follows:
![PaddlePaddle on Kubernetes Architecture](src/k8s-paddle-arch.png) ![PaddlePaddle on Kubernetes Architecture](src/k8s-paddle-arch.png)
The above figure describes a distributed training architecture which contains 3 nodes, each The above figure describes a distributed training architecture which contains 3 nodes, each
Pod would mount a folder of the distributed file system to save training data and models Pod mounts a folder of the distributed file system to save training data and models
by Kubernetes Volume. Kubernetes created 3 Pod for this training phase and scheduled these on by Kubernetes Volume. Kubernetes created 3 Pods for this training phase and scheduled these on
3 nodes, each Pod has a PaddlePaddle container. After the containers have been created, 3 nodes, each Pod has a PaddlePaddle container. After the containers car created,
PaddlePaddle would start up the communication between PServer and Trainer and read training PaddlePaddle starts up the communication between PServer and Trainer and read training
data for this training job. data for this training job.
As the description above, we can start up a PaddlePaddle distributed training job on a ready As the description above, we can start up a PaddlePaddle distributed training job on a
Kubernetes cluster as the following steps: Kubernetes ready cluster with the following steps:
1. [Build PaddlePaddle Docker Image](#Build a Docker Image) 1. [Build PaddlePaddle Docker Image](#Build a Docker Image)
1. [Split training data and upload to the distributed file system](#Upload Training Data) 1. [Split training data and upload to the distributed file system](#Upload Training Data)
...@@ -35,16 +35,13 @@ We will introduce these steps as follows: ...@@ -35,16 +35,13 @@ We will introduce these steps as follows:
### Build a Docker Image ### Build a Docker Image
PaddlePaddle Docker Image needs to support the runtime environment of `Paddle PServer` and Training docker image needs to package the paddle pserver and paddle trainer runtimes, as well as two more processes before we can kick off the training:
`Paddle Trainer` process and this Docker Image has the two import features:
- Copy the training data into the container. - Copying the training data into container.
- Generate the start arguments of `Paddle PServer` and `Paddle Training` process. - Generating the initialization arguments for `Paddle PServer` and `Paddle Training` processes.
Because of the official Docker Image `paddlepaddle/paddle:latest` has already included the Since the paddlepaddle official docker image already has the runtimes we need, we'll take it as the base image and pack some additional scripts for the processes mentioned above to build our training image. for more detail, please find from the following link:
PaddlePaddle executable file, but above features so that we can use the official Docker Image as - https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/usage/cluster/src/k8s_train/Dockerfile
a base Image and add some additional scripts to finish the work of building a new image.
You can reference [Dockerfile](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/usage/cluster/src/k8s_train/Dockerfile).
```bash ```bash
...@@ -58,17 +55,17 @@ And then upload the new Docker Image to a Docker hub: ...@@ -58,17 +55,17 @@ And then upload the new Docker Image to a Docker hub:
docker push [YOUR_REPO]/paddle:mypaddle docker push [YOUR_REPO]/paddle:mypaddle
``` ```
**[NOTE]**, in the above command arguments, `[YOUR_REPO]` representative your Docker repository, **[NOTE]**, in the above command arguments, `[YOUR_REPO]` represents your Docker repository,
you need to use your repository instead of it. We will use `[YOUR_REPO]/paddle:mypaddle` to you need to use your repository instead of it. We will replace it with your respository name to
represent the Docker Image which built in this step. represent the Docker Image which built in this step.
### Prepare Training Data ### Prepare Training Data
We can download and split the training job by creating a Kubernetes Job, or custom your image We can download and split the training job by creating a Kubernetes Job, or custom your image
by editing [k8s_train](./src/k8s_train/README.md). by editing [k8s_train](./src/k8s_train/).
Before creating a Job, we need to bind a [persistenVolumeClaim](https://kubernetes.io/docs/user-guide/persistent-volumes) by the different type of Before creating a Job, we need to bind a [persistenVolumeClaim](https://kubernetes.io/docs/user-guide/persistent-volumes) by the different type of
the different distributed file system, the generated dataset would be saved on this volume. the different file system, the generated dataset would be saved on this volume.
```yaml ```yaml
apiVersion: batch/v1 apiVersion: batch/v1
...@@ -100,7 +97,13 @@ spec: ...@@ -100,7 +97,13 @@ spec:
restartPolicy: Never restartPolicy: Never
``` ```
If success, you can see some information like this: Create the Job with the following command:
```bash
> kubectl create -f xxx.yaml
```
If created successfully, you can see some information like this:
```base ```base
[root@paddle-kubernetes-node0 nfsdir]$ tree -d [root@paddle-kubernetes-node0 nfsdir]$ tree -d
...@@ -117,13 +120,13 @@ If success, you can see some information like this: ...@@ -117,13 +120,13 @@ If success, you can see some information like this:
``` ```
The `paddle-cluster-job` above is the job name for this training job; we need 3 The `paddle-cluster-job` above is the job name for this training job; we need 3
PaddlePaddle training node and save the split training data on `paddle-cluster-job` path, PaddlePaddle training nodes and save the split training data in `paddle-cluster-job` path,
the folder `0`, `1` and `2` representative the `training_id` on each node, `quick_start` folder is used to store training data, `output` folder is used to store the models and logs. the folder `0`, `1` and `2` represents the `training_id` on each node, `quick_start` folder is used to store training data, `output` folder is used to store the models and logs.
### Create a Job ### Create a Job
Kubernetes allow users to create an object with YAML files, and we can use a command-line tool Kubernetes allow users to create objects with YAML files, and we can use a command-line tool
to create it. to create it.
The Job YAML file describes that which Docker Image would be used in this training job, how much nodes would be created, what's the startup arguments of `Paddle PServer/Trainer` process and what's the type of Volumes. You can find the details of the YAML filed in The Job YAML file describes that which Docker Image would be used in this training job, how much nodes would be created, what's the startup arguments of `Paddle PServer/Trainer` process and what's the type of Volumes. You can find the details of the YAML filed in
...@@ -177,8 +180,8 @@ spec: ...@@ -177,8 +180,8 @@ spec:
In the above YAML file: In the above YAML file:
- `metadata.name`, The job name. - `metadata.name`, The job name.
- `parallelism`, The Kubernetes Job would create `parallelism` Pods at the same time. - `parallelism`, Whether the Kubernetes Job would create `parallelism` Pods at the same time.
- `completions`, The Job would become the success status only the number of successful Pod(the exit code is 0) - `completions`, The Job would become the success status only when the number of successful Pod(the exit code is 0)
is equal to `completions`. is equal to `completions`.
- `volumeMounts`, the name field `jobpath` is a key, the `mountPath` field represents - `volumeMounts`, the name field `jobpath` is a key, the `mountPath` field represents
the path in the container, and we can define the `jobpath` in `volumes` filed, use `hostPath` the path in the container, and we can define the `jobpath` in `volumes` filed, use `hostPath`
...@@ -209,13 +212,15 @@ kubectl create -f job.yaml ...@@ -209,13 +212,15 @@ kubectl create -f job.yaml
``` ```
Upon successful creation, Kubernetes would create 3 Pods as PaddlePaddle training node, Upon successful creation, Kubernetes would create 3 Pods as PaddlePaddle training node,
, pull the Docker image and begin to train. pull the Docker image and begin to train.
### Checkout the Output ### Checkout the Output
At the process of training, we can check the logs and the output models, such as we store At the process of training, we can check the logs and the output models which is stored in
the output on `output` folder. **NOTE**, `node_0`, `node_1` and `node_2` represent the the `output` folder.
**NOTE**, `node_0`, `node_1` and `node_2` represent the
`trainer_id` of the PaddlePaddle training job rather than the node id of Kubernetes. `trainer_id` of the PaddlePaddle training job rather than the node id of Kubernetes.
```bash ```bash
...@@ -292,7 +297,7 @@ PADDLE_SERVER_NUM = os.getenv("CONF_PADDLE_GRADIENT_NUM") ...@@ -292,7 +297,7 @@ PADDLE_SERVER_NUM = os.getenv("CONF_PADDLE_GRADIENT_NUM")
### Communication between Pods ### Communication between Pods
At the begin of `start_paddle.py`, it would initialize and parse the arguments. At the begin of `start_paddle.py`, it would initializes and parses the arguments.
```python ```python
parser = argparse.ArgumentParser(prog="start_paddle.py", parser = argparse.ArgumentParser(prog="start_paddle.py",
...@@ -314,11 +319,12 @@ And then query the status of all the other Pods of this Job by the function `get ...@@ -314,11 +319,12 @@ And then query the status of all the other Pods of this Job by the function `get
idMap = getIdMap(podlist) idMap = getIdMap(podlist)
``` ```
**NOTE**: `getPodList()` would fetch all the pod in the current namespace, if some Pods are running, may cause some error. We will use [statfulesets](https://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets) instead of **NOTE**: `getPodList()` would prefetch all the Pods in the current namespace, if some
Pods are alreay running, it may cause some error. We will use [statfulesets](https://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets) instead of
Kubernetes Pod or Replicaset in the future. Kubernetes Pod or Replicaset in the future.
For the implement of `getIdMap(podlist)`, this function would fetch each IP address of The function `getIdMap(podlist)` fetches IPs addresses of `podlist` and then sort them
`podlist` and then sort them to generate `trainer_id`. to generate `trainer_id`.
```python ```python
def getIdMap(podlist): def getIdMap(podlist):
...@@ -340,9 +346,10 @@ so that we can start up them by `startPaddle(idMap, train_args_dict)`. ...@@ -340,9 +346,10 @@ so that we can start up them by `startPaddle(idMap, train_args_dict)`.
### Create Job ### Create Job
The main goal of `startPaddle` is generating the arguments of `Paddle PServer` and `Paddle Trainer` processes. Such as `Paddle Trainer`, we parse the environment variable and then get The main goal of `startPaddle` is generating the arguments of `Paddle PServer` and
`PADDLE_NIC`, `PADDLE_PORT`, `PADDLE_PORTS_NUM` and etc..., finally find `trainerId` from `Paddle Trainer` processes. Take `Paddle Trainer` as an example, we parse the
`idMap` according to its IP address. environment variable and then get `PADDLE_NIC`, `PADDLE_PORT`, `PADDLE_PORTS_NUM` and etc...,
finally find `trainerId` from `idMap` according to its IP address.
```python ```python
program = 'paddle train' program = 'paddle train'
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册