# 启动 Kubernetes 控制飞机

在这个实验室中，你将在三个计算实例中引导 Kubernetes 控制平面，并将其配置为高可用性。你还将创建一个外部负载均衡器，该负载均衡器向远程客户端公开 Kubernetes API 服务器。将在每个节点上安装以下组件:Kubernetes API 服务器、调度器和控制器管理器。

## 先决条件

该实验室中的命令必须在每个控制器实例上运行:`controller-0`、`controller-1` 和 `controller-2`。使用 `gcloud` 命令登录到每个控制器实例。示例:

```
gcloud compute ssh controller-0
```

### 与 TMUX 并行运行命令

[tmux](https://github.com/tmux/tmux/wiki) 可用于同时在多个计算实例上运行命令。请参阅先决条件实验室中的 [与 TMUX 并行运行命令](01-prerequisites.md#running-commands-in-parallel-with-tmux) 部分。

## 提供 Kubernetes 控制飞机

创建 Kubernetes 配置目录:

```
sudo mkdir -p /etc/kubernetes/config
```

### 下载并安装 Kubernetes 控制器二进制文件

下载官方的 Kubernetes 发布二进制文件:

```
wget -q --show-progress --https-only --timestamping \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kube-apiserver" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kube-controller-manager" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kube-scheduler" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kubectl"
```

安装 Kubernetes 二进制文件:

```
{
  chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl
  sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
}
```

### 配置 Kubernetes API 服务器

```
{
  sudo mkdir -p /var/lib/kubernetes/

  sudo mv ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
    service-account-key.pem service-account.pem \
    encryption-config.yaml /var/lib/kubernetes/
}
```

实例内部 IP 地址将用于向集群的成员宣传 API 服务器。检索当前计算实例的内部 IP 地址:

```
INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
```

```
REGION=$(curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/project/attributes/google-compute-default-region)
```

```
KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  --region $REGION \
  --format 'value(address)')
```

创建 `kube-apiserver.service`Systemd 单元文件:

```
cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
  --advertise-address=${INTERNAL_IP} \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/audit.log \\
  --authorization-mode=Node,RBAC \\
  --bind-address=0.0.0.0 \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --etcd-cafile=/var/lib/kubernetes/ca.pem \\
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\
  --etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \\
  --event-ttl=1h \\
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
  --runtime-config='api/all=true' \\
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
  --service-account-signing-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-account-issuer=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --service-node-port-range=30000-32767 \\
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
```

### 配置 Kubernetes 控制器管理器

将 `kube-controller-manager`kubeconfig 移至适当位置:

```
sudo mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
```

创建 `kube-controller-manager.service`Systemd 单元文件:

```
cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
  --bind-address=0.0.0.0 \\
  --cluster-cidr=10.200.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --leader-elect=true \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --use-service-account-credentials=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
```

### 配置 Kubernetes 调度程序

将 `kube-scheduler`kubeconfig 移至适当位置:

```
sudo mv kube-scheduler.kubeconfig /var/lib/kubernetes/
```

创建 `kube-scheduler.yaml` 配置文件:

```
cat <<EOF | sudo tee /etc/kubernetes/config/kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
EOF
```

创建 `kube-scheduler.service`Systemd 单元文件:

```
cat <<EOF | sudo tee /etc/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
  --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
```

### 启动控制器服务

```
{
  sudo systemctl daemon-reload
  sudo systemctl enable kube-apiserver kube-controller-manager kube-scheduler
  sudo systemctl start kube-apiserver kube-controller-manager kube-scheduler
}
```

> 允许长达 10 秒的时间让 Kubernetes API 服务器完全初始化。

### 启用 HTTP 健康检查

将使用 [谷歌网络负载均衡器](https://cloud.google.com/compute/docs/load-balancing/network) 在三个 API 服务器之间分配流量，并允许每个 API 服务器终止 TLS 连接并验证客户端证书。网络负载均衡器只支持 HTTP 健康检查，这意味着 API 服务器公开的 HTTPS 端点不能使用。作为一种解决方法，可以使用 Nginx Webserver 来代理 HTTP 健康检查。在本节中，将安装和配置 Nginx，以接受端口 `80` 上的 HTTP 健康检查，并代理到 `https://127.0.0.1:6443/healthz` 上的 API 服务器的连接。

> 默认情况下，`/healthz`API 服务器端点不需要身份验证。

安装一个基本的 Web 服务器来处理 HTTP 健康检查:

```
sudo apt-get update
sudo apt-get install -y nginx
```

```
cat > kubernetes.default.svc.cluster.local <<EOF
server {
  listen      80;
  server_name kubernetes.default.svc.cluster.local;

  location /healthz {
     proxy_pass                    https://127.0.0.1:6443/healthz;
     proxy_ssl_trusted_certificate /var/lib/kubernetes/ca.pem;
  }
}
EOF
```

```
{
  sudo mv kubernetes.default.svc.cluster.local \
    /etc/nginx/sites-available/kubernetes.default.svc.cluster.local

  sudo ln -s /etc/nginx/sites-available/kubernetes.default.svc.cluster.local /etc/nginx/sites-enabled/
}
```

```
sudo systemctl restart nginx
```

```
sudo systemctl enable nginx
```

### 验证

```
kubectl cluster-info --kubeconfig admin.kubeconfig
```

```
Kubernetes control plane is running at https://127.0.0.1:6443
```

测试 Nginx HTTP 健康检查代理:

```
curl -H "Host: kubernetes.default.svc.cluster.local" -i http://127.0.0.1/healthz
```

```
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sun, 02 May 2021 04:19:29 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 2
Connection: keep-alive
Cache-Control: no-cache, private
X-Content-Type-Options: nosniff
X-Kubernetes-Pf-Flowschema-Uid: c43f32eb-e038-457f-9474-571d43e5c325
X-Kubernetes-Pf-Prioritylevel-Uid: 8ba5908f-5569-4330-80fd-c643e7512366

ok
```

> 请记住在每个控制器节点上运行上述命令:`controller-0`、`controller-1` 和 `controller-2`。

## 用于 Kubelet 授权的 RBAC 

在本节中，你将配置 RBAC 权限，以允许 Kubernetes API 服务器访问每个工作节点上的 KubeletAPI。在 POD 中检索度量、日志和执行命令时，需要访问 Kubelet API。

> 本教程将 Kubelet`--authorization-mode` 标志设置为 `Webhook`。Webhook 模式使用 [SubjectAccessReview ](https://kubernetes.io/docs/admin/authorization/#checking-api-access)API 来确定授权。

本节中的命令将影响整个集群，并且只需要从一个控制器节点运行一次。

```
gcloud compute ssh controller-0
```

创建 `system:kube-apiserver-to-kubelet`[ClusterRole](https://kubernetes.io/docs/admin/authorization/rbac/#role-and-clusterrole)，具有访问 Kubelet API 和执行与管理 pod 相关的最常见任务的权限:

```
cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
EOF
```

Kubernetes API 服务器使用 `--kubelet-client-certificate` 标志所定义的客户机证书对 Kubelet 进行 `kubernetes` 用户身份验证。

将 `system:kube-apiserver-to-kubelet`clusterrole 绑定到 `kubernetes` 用户:

```
cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF
```

## Kubernetes 前端负载均衡器

在本节中，你将为 Kubernetes API 服务器提供一个外部负载均衡器。`kubernetes-the-hard-way` 静态 IP 地址将附加到生成的负载均衡器。

> 本教程中创建的计算实例将不具有完成本节的权限。** 在用于创建计算实例的同一台机器上运行以下命令。

### 提供网络负载均衡器

创建外部负载均衡器网络资源:

```
{
  KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
    --region $(gcloud config get-value compute/region) \
    --format 'value(address)')

  gcloud compute http-health-checks create kubernetes \
    --description "Kubernetes Health Check" \
    --host "kubernetes.default.svc.cluster.local" \
    --request-path "/healthz"

  gcloud compute firewall-rules create kubernetes-the-hard-way-allow-health-check \
    --network kubernetes-the-hard-way \
    --source-ranges 209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 \
    --allow tcp

  gcloud compute target-pools create kubernetes-target-pool \
    --http-health-check kubernetes

  gcloud compute target-pools add-instances kubernetes-target-pool \
   --instances controller-0,controller-1,controller-2

  gcloud compute forwarding-rules create kubernetes-forwarding-rule \
    --address ${KUBERNETES_PUBLIC_ADDRESS} \
    --ports 6443 \
    --region $(gcloud config get-value compute/region) \
    --target-pool kubernetes-target-pool
}
```

### 验证

> 本教程中创建的计算实例将不具有完成本节的权限。** 在用于创建计算实例的同一台机器上运行以下命令。

检索 `kubernetes-the-hard-way` 静态 IP 地址:

```
KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  --region $(gcloud config get-value compute/region) \
  --format 'value(address)')
```

对 Kubernetes 版本信息进行 HTTP 请求:

```
curl --cacert ca.pem https://${KUBERNETES_PUBLIC_ADDRESS}:6443/version
```

> 输出

```
{
  "major": "1",
  "minor": "21",
  "gitVersion": "v1.21.0",
  "gitCommit": "cb303e613a121a29364f75cc67d3d580833a7479",
  "gitTreeState": "clean",
  "buildDate": "2021-04-08T16:25:06Z",
  "goVersion": "go1.16.1",
  "compiler": "gc",
  "platform": "linux/amd64"
}
```

下一条:[引导 Kubernetes 工作者节点](09-bootstrapping-kubernetes-workers.md)