kubernetes 1.13.1 二进制包安装

环境信息

操作系统 CentOS 7.6.1810
Kubernetes v 1.13.1
Flanneld v0.10.0
Etcd v3.3.10
Cfssl v1.2.0
Docker 18.06.1-ce

架构信息

机器名 IP 组件 角色 配置
k8s-master-80 192.168.1.80 kube-apiserver, kube-controller-manager,kube-schedule,flannel,etcd,docker Master 8核8G
k8s-node-81 192.168.1.81 kubelet,kube-proxy,flannel,etcd,docker Node 8核8G
k8s-node-82 192.168.1.82 kubelet,kube-proxy,flannel,etcd,docker Node 8核8G

一. 需要准备的东西

  • HTTP 代理服务器
  • Kubernetes
  • Etcd
  • Flanneld

二. 软件包下载

1. 在Master节点上下载所需的软件包

kubernetes: > # 定义环境变量
$ export KUBE_URL="https://storage.googleapis.com/kubernetes-release/release/v1.13.1/bin/linux/amd64" > > # 快速下载 kubernetes 组件 $ cd /usr/local/bin/ && wget ${KUBE_URL}/{kubectl,kube-controller-manager,kube-scheduler,kube-apiserver} > > # 设置执行权限 $ chmod +x /usr/local/bin/{kubectl,kube-controller-manager,kube-scheduler,kube-apiserver}

cfssl: > # 下载 cfssl $ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl > > # 下载 cfssljson wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson > > # 下载 cfssl-certinfo wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo > > # 设置执行权限 $ chmod +x /usr/local/bin/{cfssl,cfssljson,cfssl-certinfo}

etcd:
> # 下载etcd软件包
$ wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz -O /opt/etcd-v3.3.10-linux-amd64.tar.gz > > # 解压etcd软件包
$ tar xf etcd-v3.3.10-linux-amd64.tar.gz -C /opt/ > > # 复制etcd文件
$ cp /opt/etcd-v3.3.10-linux-amd64/{etcd,etcdctl} /usr/local/bin/ > > # 设置执行权限
$ chmod +x /usr/local/bin/{etcd,etcdctl} > > # 同步到Node节点
$ scp -r /usr/local/bin/{etcd,etcdctl} root@{NodeIP}:/usr/local/bin/

flanneld: > # 从Github 下载 Flanneld Release 版本
$ wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz -O /opt/flannel-v0.10.0-linux-amd64.tar.gz > > # 创建一个目录存放flannel解压的文件
$ mkdir -p /opt/flannel_bin/
> > # 解压文件
$ tar xf /opt/flannel-v0.10.0-linux-amd64.tar.gz -C /opt/flannel_bin/
> > # 复制二进制文件到bin目录
$ cp -av /opt/flanneld_bin/{flanneld,mk-docker-opts.sh} /usr/local/bin/
> > # 同步二进制文件到 Node 节点
$ scp -r /opt/flanneld_bin/{flanneld,mk-docker-opts.sh} root@{NodeIP}:/usr/local/bin/

config:
> # 下载配置文件
wget https://github.com/Tomey/kubernetes_config/archive/v2.0.tar.gz -O /opt/v2.0.tar.gz
>
> # 解压配置文件
tar xf /opt/v2.0.tar.gz -C /opt/ && rm -f /opt/v2.0.tar.gz > > # 将配置文件同步到 Node 节点
$ scp -r /opt/kubernetes_config-2.0/ root@{NodeIP}:/opt/

2. 在Node节点上下载所需的软件包

kubernetes: > # 定义环境变量
$ export KUBE_URL="https://storage.googleapis.com/kubernetes-release/release/v1.13.1/bin/linux/amd64" > > # 快速下载 kubernetes 组件 $ cd /usr/local/bin/ && wget ${KUBE_URL}/{kubelet,kube-proxy} > ># 设置执行权限 $ chmod +x /usr/local/bin/{kubelet,kube-proxy}

PS: 其他文件已从Master节点复制过来 所以这边不在进行下载

三. 证书配置 (Master)

1. 准备操作

# 创建目录
mkdir /opt/ssl/ && cd /opt/ssl/

2. 创建CA证书

# 创建CA证书配置
> ca-config.json: {"signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } }

# 创建CA证书请求文件 > ca-csr.json: > { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "FUJIAN", "L": "JINJIANG", "O": "k8s", "OU": "System" } ] }

# 生成CA证书和私钥 cfssl gencert -initca ca-csr.json | cfssljson -bare ca

3. 创建server证书

# 创建server证书签名请求 > server-csr.json(注意替换IP及主机名): > { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.80", "192.168.1.81", "192.168.1.82", "10.254.0.1", "kubernetes", "k8s_master_80", "k8s_node_81", "k8s_node_82", "cluster.local", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "FUJIAN", "L": "JINJIANG", "O": "k8s", "OU": "System" } ] }

# 生成kubernetes证书及私钥 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

4. 创建admin证书签名请求

admin-csr.json:
{ "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "FUJIAN", "L": "JINJIANG", "O": "system:masters", "OU": "System" } ] }

# 生成admin证书及私钥 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

5. 创建 kube-proxy 证书

# 创建 kube-proxy 证书签名请求 > kube-proxy-csr.json: > { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "FUJIAN", "L": "JINJIANG", "O": "k8s", "OU": "System" } ] }

# 生成证书及私钥 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

6.分发证书

# 创建证书目录
mkdir -p /etc/kubernetes/ssl

# 拷贝证书至目录
cp -r *.pem /etc/kubernetes/ssl

# 同步证书
scp -r /etc/kubernetes ${USER}@${HOST}:/etc/ 其中${USER}为登录用户名, ${HOST}为机器IP

四、安装 etcd

1. Master 配置

# 创建服务启动文件
cp /opt//opt/kubernetes_config-2.0/master/etcd/etcd.service /usr/lib/systemd/system/etcd.service (需要手动修改 --initial-cluster 为你的集群节点信息)

# 创建目录
mkdir -p /var/lib/etcd/
mkdir /etc/etcd

# 创建 etcd 配置文件 cp /opt/kubernetes_config-2.0/master/etcd/etcd.conf /etc/etcd/etcd.conf

# 修改配置信息
$ sed -i 's/etcd1/${YouEtcdName}/g' (此名称要和服务文件的 –initial-cluster 的变量名对应上) $ sed -i 's/192.168.1.80/${YouIP}/g'

# 启动服务
systemctl daemon-reload
systemctl enable etcd && systemctl start etcd

# 查看服务是否正常启动
systemctl status etcd
netstat -ntpl| grep etcd

2. Node 配置

# 从Master服务启动文件
scp root@{MasterIP}:/usr/lib/systemd/system/etcd.service /usr/lib/systemd/system/etcd.service

# 创建目录
mkdir -p /var/lib/etcd/
mkdir /etc/etcd

# 创建 etcd 配置文件 cp /opt/kubernetes_config-2.0/node/etcd/etcd.conf /etc/etcd/etcd.conf

# 修改配置信息 $ sed -i 's/etcd2/${YouEtcdName}/g' (此名称要和服务文件的 –initial-cluster 的变量名对应上) $ sed -i 's/192.168.1.81/${YouIP}/g'

# 启动服务
systemctl daemon-reload
systemctl enable etcd && systemctl start etcd

# 查看服务是否正常启动
systemctl status etcd
netstat -ntpl| grep etcd

3. 测试服务是否正常

etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/server.pem --key-file=/etc/kubernetes/ssl/server-key.pem cluster-health

五、 配置Kubernetes 参数(Master)

1.创建程序目录

mkdir -p /etc/kubernetes/

2.创建 TLS Bootstrapping Token

# 生成随机数 export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')

# 写入至配置文件 echo ${BOOTSTRAP_TOKEN}',kubelet-bootstrap,10001,"system:kubelet-bootstrap"' >> /etc/kubernetes/token.csv

3. 创建 kubelet bootstrapping kubeconfig 文件

# 进入程序目录
cd /etc/kubernetes

# 配置API服务地址
export KUBE_APISERVER="https://192.168.1.80:6443"

# 设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

4. 创建 kube-proxy kubeconfig 文件

# 配置API服务地址
export KUBE_APISERVER="https://192.168.1.80:6443"

# 设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-proxy.kubeconfig

# 设置客户端认证参数 kubectl config set-credentials kube-proxy --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

# 设置上下文参数 kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

# 设置默认上下文 kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

5. 创建 kubectl kubeconfig 文件

# 配置API服务地址
export KUBE_APISERVER="https://192.168.1.80:6443"

# 设置集群参数 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISERVER}

# 设置客户端认证参数 kubectl config set-credentials admin --client-certificate=/etc/kubernetes/ssl/admin.pem --embed-certs=true --client-key=/etc/kubernetes/ssl/admin-key.pem

# 设置上下文参数 kubectl config set-context kubernetes --cluster=kubernetes --user=admin

# 设置默认上下文 kubectl config use-context kubernetes

6. 将{bootstrap,kube-proxy}.kubeconfig复制至Node

scp -r {kube-proxy,bootstrap}.kubeconfig ${NodeIP}:/etc/kubernetes/

六、 创建 APIServer (Master)

1. 创建 ApiServer 配置

# 复制 APIServer 启动文件 cp /opt/kubernetes_config-2.0/master/kubernetes/kube-apiserver_config/kube-apiserver.service /usr/lib/systemd/system/kube-apiserver.service

# 复制 APIServer 配置文件 cp /opt/kubernetes_config-2.0/master/kubernetes/kube-apiserver_config/apiserver /etc/kubernetes/apiserver

2. 启动 ApiServer

systemctl daemon-reload systemctl enable kube-apiserver && systemctl start kube-apiserver

3. 查看服务启动是否正常

systemctl status kube-apiserver
netstat -ntpl| grep apiserver

七、 创建 Controller-Manager(Master)

# 复制 controller-manager 服务启动文件
cp /opt/kubernetes_config-2.0/master/kubernetes/kube-controller-manager_config/kube-controller-manager.service /usr/lib/systemd/system/kube-controller-manager.service

# 复制 controller-manager服务配置文件
cp /opt/kubernetes_config-2.0/master/kubernetes/kube-controller-manager_config/controller-manager /etc/kubernetes/controller-manager

# 启动 controller-manager 服务
systemctl daemon-reload
systemctl enable kube-controller-manager && systemctl start kube-controller-manager

八、 创建Scheduler (Master)

# 复制 scheduler 服务启动文件
cp /opt/kubernetes_config-2.0/master/kubernetes/kube-scheduler_config/kube-scheduler.service /usr/lib/systemd/system/kube-scheduler.service

# 复制 scheduler 服务配置文件
cp /opt/kubernetes_config-2.0/master/kubernetes/kube-scheduler_config/scheduler /etc/kubernetes/scheduler

# 启动 scheduler 服务
systemctl daemon-reload
systemctl enable kube-scheduler && systemctl start kube-scheduler

九、 Flanneld 配置 (ALL Node)

# 复制 flanneld 启动文件 cp /opt/kubernetes_config-2.0/master/flanneld/flanneld.service /usr/lib/systemd/system/flanneld.service

# 复制 flanneld 配置文件 cp /opt/kubernetes_config-2.0/master/flanneld/flanneld /etc/kubernetes/flanneld

# 在etcd中创建网络配置 etcdctl --endpoints=https://192.168.1.80:2379,https://192.168.1.81:2379,https://192.168.1.82:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/server.pem --key-file=/etc/kubernetes/ssl/server-key.pem mkdir /kube-centos/network etcdctl --endpoints=https://192.168.1.80:2379,https://192.168.1.81:2379,https://192.168.1.82:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/server.pem --key-file=/etc/kubernetes/ssl/server-key.pem mk /kube-centos/network/config '{"Network":"172.18.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'

# 启动 flanneld 服务
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

十、 在Docker中集成Flannel

# 编辑 docker 启动文件
vim /usr/lib/systemd/system/docker.service

# 在ExecStart上增加
EnvironmentFile=-/run/flannel/docker
EnvironmentFile=-/run/docker_opts.env
EnvironmentFile=-/run/flannel/subnet.env

# 修改 ExecStart 为
ExecStart=/usr/bin/dockerd --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}

# 或者可以直接覆盖配置文件(可选)
cp /opt/kubernetes_config-2.0/master/docker/docker.service /usr/lib/systemd/system/docker.service

# 重新启动 Docker 服务
systemctl daemon-reload
systemctl restart docker

十一、 安装 kubelet (Node)

# 复制 kubelet 启动文件 /opt/kubernetes_config-2.0/node/kubernetes/kubelet_config/kubelet.service /usr/lib/systemd/system/kubelet.service

# 复制 kubelet 配置文件 cp /opt/kubernetes_config-2.0/node/kubernetes/kubelet_config/{kubelet,kubelet.yaml} /etc/kubernetes/

# 创建必备目录 mkdir -p /var/lib/kubelet && mkdir -p /etc/kubernetes/manifests

# 启动服务
systemctl daemon-reload
systemctl enable kubelet && systemctl start kubelet

十二、 安装 kube-proxy (Node)

# 复制 kube-proxy 启动文件
cp /opt/kubernetes_config-2.0/node/kubernetes/kube-proxy_config/kube-proxy.service /usr/lib/systemd/system/kube-proxy.service

# 复制 kube-proxy 配置文件
cp /opt/kubernetes_config-2.0/node/kubernetes/kube-proxy_config/proxy /etc/kubernetes/proxy

# 启动kube-proxy服务
systemctl daemon-reload
systemctl enable kube-proxy && systemctl start kube-proxy

十三、 授权

# 将 kubelet-bootstrap 用户 加入 system:node-bootstrapper 权限角色
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

# 查看证书请求
kubectl get csr

# 同意Node节点的证书请求
kubectl certificate approve ${Name}

十四、故障处理

Failed to list *v1.Node: nodes “192.168.1.81” is forbidden: User “kubelet-bootstrap” cannot list resource “nodes” in API group “” at the cluster scope:
这个是权限问题,解决:
kubectl delete clusterrolebinding kubelet-bootstrap kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node --user=kubelet-bootstrap

-

去除节点 taint 未初始化标记

kubectl taint nodes --all node.cloudprovider.kubernetes.io/uninitialized-
具体使用方法可参考: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

十五、备注

** 1. 请自行替换配置文件IP(此配置用到的IP为 192.168.1.80-192.168.1.82)**
** 2. 安装此服务基本上镜像都是来源于谷歌,所以必须翻墙 **