先介绍一下CNI 相关工作

1. k8s 的 kubelet 在启动一个容器之前，会先做一些预先检查以及 csi 的操作
2. 然后 kubelet 调用 CRI 的接口，通过 grpc 的方式和 CRI runtime 通信，告知 CRI 要创建 pod 了
3. 随后 CRI 的 Server 端收到通知后调用 OCI 的接口去真正的执行拉起 Pod 的操作
4. 不过在真的拉起 pod 之前，会先给 pod 创建一个 pause 容器，这个 pause 容器是一个特别小又特别稳定的进程，主要用来挂载网络命名空间和存储资源
5. 然后 CRI 调用 CNI 提供的接口，先在/etc/cni/net.d 目录中获取网络插件配置(这个配置由每个插件自己通过 daemonset 的方式拷贝到主机上)，然后把插件的配置作为标准输入，再把容器的运行时信息作为环境变量，最后执行插件
6. CNI 插件执行完毕后，把执行结果(结果要包含一些关键信息比如 Pod IP 等)直接干到标准输出上
7. CRI 从标准输出上读取插件执行结果再做后续操作，后续操作就是拉起真正的容器等

安装cni插件

每个worker节点上安装CNI插件驱动,先安装cni驱动，否则node一直是no ready状态

#CNI插件下载地址
https://github.com/containernetworking/plugins
# cni插件flannel 会去默认目录里找 cni
mkdir /opt/cni/bin /etc/cni/net.d
tar -zxvf cni-xxx.tgz -C /opt/cni/bin

/etc/cni/net.d 是配置目录

我这里用的是docker作为k8s的容器运行时所以插件是cri-docker

cri-docker 的配置里必须要配置上--network-plugin=cni

下面部署flannel网络

下载地址：
https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yaml
kubectl get pods -n kube-system

#查看日志没有权限
kubectl logs -f kube-flannel-ds-s8rfn -n kube-system
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log kube-flannel-ds-s8rfn)

授权Apiserver访问kubelet

cat >apiserver-to-kubelet-rbac.yaml<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f apiserver-to-kubelet-rbac.yaml

上面的flannel可能会启动失败

需要注意以下配置

#服务器上配置好hosts，要不然会启动失败
192.168.56.101  master
192.168.56.102  node1
192.168.56.103  node2

网卡配置

#网关也要配置好，要不然提示找不到默认路由
GATEWAY=192.168.56.1

批准node节点加入集群

#查看csr请求
kubectl get csr --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig  -A -o wide
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-XLjK0kUVExr3qp_Gon55QkKu2zjCLT_yJXi1OUQOMR4   10m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending
#批准csr请求，允许node加入k8s集群
kubectl certificate approve node-csr-XLjK0kUVExr3qp_Gon55QkKu2zjCLT_yJXi1OUQOMR4 --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig
certificatesigningrequest.certificates.k8s.io/node-csr-XLjK0kUVExr3qp_Gon55QkKu2zjCLT_yJXi1OUQOMR4 approved

#查看node状态
kubectl get node --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig -o wide -n kube-flannel
NAME     STATUS     ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
master   Ready      <none>   2d1h   v1.24.1   192.168.56.101   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://20.10.21
node1    Ready      <none>   95m    v1.24.1   192.168.56.102   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://20.10.21
node2    NotReady   <none>   18s    v1.24.1   192.168.56.103   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://20.10.21

#查看 node2上面的flannel日志，发现一直在 PodInitializing
kubectl logs kube-flannel-ds-zwwlk -n kube-flannel --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig 
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-zwwlk" is waiting to start: PodInitializing

#显示特定pod资源的详细信息，我发现pull 镜像失败
kubectl describe pod kube-flannel-ds-zwwlk -n kube-flannel --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig
  Warning  FailedCreatePodSandBox  117s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "registry.k8s.io/pause:3.6": Error response from daemon: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:38258->[::1]:53: read: connection refused

遇到上不去网或者服务器离线的情况

#先把镜像下载到本地，通过save把镜像打包
docker save > xxx.tar
#上传到服务器，再load到服务器上
docker load < xxx.tar

当然建个镜像仓库私服最好不过了