You need an Amazon account and your user account needs the following privileges to continue:
* AmazonEC2FullAccess
* AmazonS3FullAccess
* AmazonRoute53FullAccess
* AmazonRoute53DomainsFullAccess
* AmazonVPCFullAccess
* IAMUserSSHKeys
* IAMFullAccess
* NetworkAdministrator
If you are not in Unites States, we also recommend creating a jump server instance with default amazon AMI in the same available zone as your cluster, otherwise there will be some issue on creating the cluster.
##For people new to Kubernetes and AWS
If you are new to Kubernetes or AWS and just want to run PaddlePaddle, you can follow these steps to start up a new cluster.
Once the cluster is up, the IP addresses of your master and node(s) will be printed, as well as information about the default services running in the cluster (monitoring, logging, dns).
User credentials and security tokens are written in `~/.kube/config`, they will be necessary to use the CLI or the HTTP Basic Auth.
Using SSH key with (AWS) fingerprint: 08:9f:6b:82:3d:b5:ba:a0:f3:db:ab:94:1b:a7:a4:c7
Creating vpc.
Adding tag to vpc-fad1139d: Name=kubernetes-vpc
Adding tag to vpc-fad1139d: KubernetesCluster=kubernetes
Using VPC vpc-fad1139d
Adding tag to dopt-e43a7180: Name=kubernetes-dhcp-option-set
Adding tag to dopt-e43a7180: KubernetesCluster=kubernetes
Using DHCP option set dopt-e43a7180
Creating subnet.
Adding tag to subnet-fc16fa9b: KubernetesCluster=kubernetes
Using subnet subnet-fc16fa9b
Creating Internet Gateway.
Using Internet Gateway igw-fc0d9398
Associating route table.
Creating route table
Adding tag to rtb-bd8512da: KubernetesCluster=kubernetes
Associating route table rtb-bd8512da to subnet subnet-fc16fa9b
Adding route to route table rtb-bd8512da
Using Route Table rtb-bd8512da
Creating master security group.
Creating security group kubernetes-master-kubernetes.
Adding tag to sg-d9280ba0: KubernetesCluster=kubernetes
Creating minion security group.
Creating security group kubernetes-minion-kubernetes.
Adding tag to sg-dc280ba5: KubernetesCluster=kubernetes
Using master security group: kubernetes-master-kubernetes sg-d9280ba0
Using minion security group: kubernetes-minion-kubernetes sg-dc280ba5
Creating master disk: size 20GB, type gp2
Adding tag to vol-04d71a810478dec0d: Name=kubernetes-master-pd
Adding tag to vol-04d71a810478dec0d: KubernetesCluster=kubernetes
Allocated Elastic IP for master: 35.162.175.115
Adding tag to vol-04d71a810478dec0d: kubernetes.io/master-ip=35.162.175.115
Generating certs for alternate-names: IP:35.162.175.115,IP:172.20.0.9,IP:10.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local,DNS:kubernetes-master
Starting Master
Adding tag to i-042488375c2ca1e3e: Name=kubernetes-master
Adding tag to i-042488375c2ca1e3e: Role=kubernetes-master
Adding tag to i-042488375c2ca1e3e: KubernetesCluster=kubernetes
Waiting for master to be ready
Attempt 1 to check for master nodeWaiting for instance i-042488375c2ca1e3e to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-042488375c2ca1e3e to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-042488375c2ca1e3e to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-042488375c2ca1e3e to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-042488375c2ca1e3e to be running (currently pending)
Sleeping for 3 seconds...
[master running]
Attaching IP 35.162.175.115 to instance i-042488375c2ca1e3e
Attaching persistent data volume (vol-04d71a810478dec0d) to master
Kubernetes master is running at https://35.162.175.115
Elasticsearch is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Grafana is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at https://35.162.175.115/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Kubernetes binaries at /home/ec2-user/kubernetes/cluster/
You may want to add this directory to your PATH in $HOME/.profile
Installation successful!
```
By default, the script will provision a new VPC and a 4 node k8s cluster in us-west-2a (Oregon) with EC2 instances running on Debian. You can override the variables defined in `<path/to/kubernetes-directory>/cluster/config-default.sh` to change this behavior as follows:
Now you can use administration tool kubectl to operate the cluster.
By default, kubectl will use the kubeconfig file generated during the cluster startup for authenticating against the API, the location is in `~/.kube/config`.
For running PaddlePaddle training with Kubernetes on AWS, you can refer to [this article](https://github.com/drinktee/Paddle/blob/k8s/doc_cn/cluster/k8s/distributed_training_on_kubernetes.md).
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-09d7e8824ef1f8384 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
All instances deleted
Releasing Elastic IP: 35.162.175.115
Deleting volume vol-04d71a810478dec0d
Cleaning up resources in VPC: vpc-fad1139d
Cleaning up security group: sg-d9280ba0
Cleaning up security group: sg-dc280ba5
Deleting security group: sg-d9280ba0
Deleting security group: sg-dc280ba5
Deleting VPC: vpc-fad1139d
Done
```
## For experts with Kubernetes and AWS
Sometimes we might need to create or manage the cluster on AWS manually with limited privileges, so here we will explain more on what’s going on with the Kubernetes setup script.
### Some Presumptions
* Instances run on Debian, the official IAM, and the filesystem is aufs instead of ext4.
* Kubernetes node use instance storage, no EBS get mounted. Master use a persistent volume for etcd.
* Nodes are running in an Auto Scaling Group on AWS, auto-scaling itself is disabled, but if some node get terminated, it will launch another node instead.
* For networking, we use ip-per-pod model here, each pod get assigned a /24 CIDR. And the whole vpc is a /16 CIDR, No overlay network at this moment, we will add Calico solution later on.
* When you create a service with Type=LoadBalancer, Kubernetes will create and ELB, and create a security group for the ELB.
* Kube-proxy sets up two IAM roles, one for master called kubernetes-master, one for nodes called kubernetes-node.
* All AWS resources are tagged with a tag named "KubernetesCluster", with a value that is the unique cluster-id.
###Script Details
* Create an s3 bucket for binaries and scripts.
* Create two iam roles: kubernetes-master, kubernetes-node.
* Create an AWS SSH key named kubernetes-YOUR_RSA_FINGERPRINT.
* Create a vpc with 172.20.0.0/16 CIDR, and enables dns-support and dns-hostnames options in vpc settings.
* Create Internet gateway, route table, a subnet with CIDR of 172.20.0.0/24, and associate the subnet to the route table.
* Create and configure security group for master and nodes.
* Create an EBS for master, it will be attached after the master node get up.
* Launch the master with fixed ip address 172.20.0.9, and the node is initialized with Salt script, all the components get started as docker containers.
* Create an auto-scaling group, it has the min and max size, it can be changed by using aws api or console, it will auto launch the kubernetes node and configure itself, connect to master, assign an internal CIDR, and the master configures the route table with the assigned CIDR.
