环境准备

实验环境规划

  • 操作系统：rockylinux8
  • 网络：NAT模式，可以确保机器在任何网络都能ssh连接
K8S集群角色 IP 主机名
控制节点 172.16.183.100 rm1
工作节点 172.16.183.101 rn1

安装前准备

  • 软件包安装
yum install -y device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo libaio-devel wget vim ncurses-devel autoconf automake epel-release openssh-server telnet vim
yum update -y
  • 配置网络
#查看网卡名
ip addr

#编辑网卡配置文件
vim /etc/sysconfig/network-scripts/ifcfg-ens160
#修改成如下内容：
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
IPADDR=172.16.183.100
NETMASK=255.255.255.0
GATEWAY=172.16.183.2
DNS1=172.16.183.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens160
DEVICE=ens160
ONBOOT=yes

nmcli connection reload 
nmcli c up ens160

也可以用nmcli来修改(rockylinux上不太好用，改配置文件好用)

nmcli connection modify ens160 ipv4.method manual connection.autoconnect yes #修改IP地址为手动配置，并且设置开机启动
nmcli connection modify ens160 ipv4.addresses 172.16.183.100/24 #修改IP地址和掩码
nmcli connection modify ens160 ipv4.gateway 172.16.183.2 #修改网关
nmcli connection modify ens160 ipv4.dns 172.16.183.2 #修改DNS，多个DNS以逗号分隔
nmcli connection down ens160 
nmcli connection up ens160
  • 关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
reboot
  • 配置主机名
hostnamectl set-hostname rm1 && bash
  • 配置hosts
tee -a /etc/hosts <<'EOF'
172.16.183.100   rm1  
172.16.183.101   rn1
EOF
  • 配置ssh互信
ssh-keygen
ssh-copy-id rn1
  • 关闭swap，并把swap的lv重新分配到/
swapoff -a
vim /etc/fstab   
#/dev/mapper/centos-swap swap      swap    defaults        0 0

lvs
#看到swap占用了3.91G的LVM
  #root rl -wi-ao---- 35.08g                                                    
  #swap rl -wi-a-----  3.91g
lvdisplay
#获取到LV Path /dev/rl/swap
#删除这个lv
lvremove /dev/rl/swap
#获取vg剩余空间
vgs
#VFree = 3.91G
#扩容lv
lvextend -L +3.91G /dev/rl/root
#扩容文件系统（xfs系统）
xfs_growfs /
#检查根分区
df -h

删除交换分区后必须重新配置 grub、BLS 引导条目、dracut 和 initramfs，否则重启会卡住或报 "找不到 swap" 错误：

# 1. 查看当前 GRUB_CMDLINE_LINUX 的完整内容（确认要删的参数）
grep GRUB_CMDLINE_LINUX /etc/default/grub
# 典型输出（VG 名可能含主机名，如 rl_rm1，也可能是 rl）：
# GRUB_CMDLINE_LINUX="crashkernel=... resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet"

# 2. 修改 /etc/default/grub，删除 swap 相关参数（影响未来新装内核的默认参数）
#    删除 resume= 参数（指向 swap 设备，用于休眠恢复）
sed -i 's/ resume=[^ "]*//' /etc/default/grub
#    删除 rd.lvm.lv=.../swap 参数（告诉 dracut 激活 swap LV）
sed -i 's/ rd\.lvm\.lv=[^ "]*swap//' /etc/default/grub

# 3. 验证 /etc/default/grub 修改结果
grep GRUB_CMDLINE_LINUX /etc/default/grub

# 4.【关键】使用 grubby 更新已有的 BLS 引导条目
#    Rocky Linux 使用 BLS（Boot Loader Specification），内核启动参数存储在
#    /boot/loader/entries/*.conf 中。仅修改 /etc/default/grub 不会更新已有的 BLS 条目！
#    必须用 grubby 删除所有已有内核条目中的 swap 相关参数：
grubby --update-kernel=ALL --remove-args="resume rd.lvm.lv=rl/swap"
# 注意：rd.lvm.lv=rl/swap 中的 rl 替换为实际 VG 名（通过 vgs 查看）

# 验证 BLS 条目已更新（确认 resume 和 swap 参数已消失）
grubby --info=ALL | grep -E "resume|swap" || echo "BLS 条目已清理干净"

# 5. 清除 dracut 的 resume 模块配置
#    dracut 有独立的 resume 配置，仅清理引导参数不够！
#    dracut -f 时 resume 模块仍会自动将已删除的 swap 设备写入 initramfs，
#    导致启动时 initramfs 尝试激活不存在的 swap LV 用于休眠恢复，系统卡住。
rm -f /etc/dracut.conf.d/resume.conf
# 显式禁用 resume 模块，防止 dracut 自动探测残留的 swap 引用
echo 'omit_dracutmodules+=" resume "' > /etc/dracut.conf.d/noresume.conf
# 如果存在 systemd-hibernate-resume 服务也一并禁用
systemctl disable systemd-hibernate-resume@*.service 2>/dev/null || true

# 6. 重新生成 initramfs（此步不可跳过，必须在上一步清除 resume 配置之后执行）
dracut -f

# 7. 重新生成 grub 配置（无论 UEFI 还是 Legacy BIOS 统一用此命令）
grub2-mkconfig -o /boot/grub2/grub.cfg

# 8. 重启验证
init 6

如果有扩盘需求，关机后在VMWare中扩盘，进系统扩容LVM：

#lsblk看到nvme0n1盘容量已经扩容了，现在新建分区，扩容到LVM
fdisk /dev/nvme0n1
#新建一个nvme0n1p3分区，type为8e
#w保存
partprobe /dev/nvme0n1
pvcreate /dev/nvme0n1p3
vgextend rl /dev/nvme0n1p3
lvextend -L +40G /dev/rl/root
xfs_growfs /dev/rl/root
  • 修改内核参数
#将br_netfilter模块添加到 Linux 内核中。这是Linux网桥的核心模块，提供网络包转发和过滤。Kubernetes需要网络插件（如 Flannel、Calico）实现容器间通信，而这些网络插件通常会用Linux网桥来进行数据包转发和过滤。
modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/profile
#开启包过滤
tee /etc/sysctl.d/k8s.conf <<'EOF'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl -p /etc/sysctl.d/k8s.conf
  • 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
  • 配置时间同步
yum -y install chrony
systemctl enable chronyd --now 

tee -a /etc/chrony.conf <<'EOF'  
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp1.tencent.com iburst
server ntp2.tencent.com iburst
EOF
#Iburst: 如果设置该选项，前四轮询之间的间隔将是2秒，而不是minpoll。 这对于在chronyd启动后快速获得时钟的第一次更新非常有用。即：重启chronyd后，2秒后，直接同步一次时间。
crontab -e
* * * * * /usr/bin/systemctl restart chronyd
systemctl restart crond
  • 配置aliyun docker repo源
#配置国内安装docker和containerd的阿里云在线源
yum install yum-utils -y
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum clean all && yum makecache

组件安装

docker

#docker也要安装，docker跟containerd不冲突，安装docker是为了能基于dockerfile构建镜像
#安装24.0.6的docker，是为了不与containerd版本冲突。否则docker版本过高，containerd也跟着变成高版本了。
yum install docker-ce-24.0.6 -y
systemctl enable docker --now
#配置docker镜像加速器，k8s所有节点均按照以下配置
tee /etc/docker/daemon.json <<'EOF'
{
 "registry-mirrors":["https://docker.m.daocloud.io"]
} 
EOF
#重启docker：
systemctl daemon-reload && systemctl restart docker && systemctl status docker

containerd

  • 当前k8s版本推荐1.6.22版本的containerd，比较稳定。
yum install containerd.io-1.6.22*  -y
cd /etc/containerd
#rm -rf *
  • 如果有harbor环境，修改config.toml为以下。（将\替换为安装harbor机器的IP），除此之外，还修改了以下内容：
  • SystemdCgroup = true
  • sandbox_image修改成了aliyun镜像库
tee /etc/containerd/config.toml <<'EOF'
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = true

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."172.16.183.100:32002".tls]
            insecure_skip_verify = true
            ca_file = ""
            cert_file = ""
            key_file = ""
        [plugins."io.containerd.grpc.v1.cri".registry.configs."172.16.183.100:32002".auth]
            username = "admin"
            password = "Harbor12345"

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
         [plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.16.183.100:32002"]
            endpoint = ["http://172.16.183.100:32002"]
          [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
             endpoint = ["https://docker.m.daocloud.io"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = ""
    insecure = false
    protocol = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
EOF

systemctl daemon-reload && systemctl start containerd && systemctl enable containerd

#修改/etc/crictl.yaml文件，指定k8s的运行时和拉取镜像都使用containerd
#这个文件是 crictl 工具的配置文件，用于指定与 containerd 交互的相关设置：
#指定unix:///run/containerd/containerd.sock 是为了告诉 crictl 使用 Unix 域套接字的方式来连接 containerd 的 API。containerd 提供了一个 socket 文件 /run/containerd/containerd.sock，crictl 通过连接该 socket 文件，可以与 containerd 进行通信，管理容器和镜像等操作。
#runtime-endpoint：指定 containerd 的运行时接口地址：crictl 可以与 containerd 通信来管理容器生命周期和资源隔离。
#image-endpoint：指定 containerd 的镜像接口地址：以便 crictl 可以与 containerd 通信来管理镜像的拉取和推送。
#timeout：指定 crictl 等待 containerd 响应的最大时间，避免出现无响应的情况。
#debug：开启或关闭 crictl 的调试模式，方便排查问题。

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
systemctl restart containerd

k8s-1.32.3

#配置安装k8s组件需要的阿里云的repo源 
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/repodata/repomd.xml.key
EOF
#其中baseurl是指定仓库的URL地址。这个仓库是Kubernetes针对CentOS 7 x86_64架构的yum仓库，yum会从这个指定的URL地址获取Kubernetes的安装包和依赖包，从而进行安装。使用阿里云镜像站可以提高安装速度和稳定性。
yum clean all && yum makecache
yum install -y kubelet-1.32.3 kubeadm-1.32.3 kubectl-1.32.3
systemctl enable kubelet

kubeadm初始化k8s集群

初始化集群

kubeadm config print init-defaults > kubeadm.yaml

tee kubeadm.yaml <<'EOF'
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.16.183.100 #控制节点的ip
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock  #指定containerd容器运行时
  imagePullPolicy: IfNotPresent
  name:  rm1 #控制节点主机名
  taints: null
---
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.32.0 
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16 
  serviceSubnet: 10.96.0.0/12 
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

kubeadm init --config=kubeadm.yaml --ignore-preflight-errors=SystemVerification

配置kubeconfig

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

#1. 设置kubectl的alias为k：
whereis kubectl ## kubectl: /usr/bin/kubectl
echo "alias k=/usr/bin/kubectl">>/etc/profile
source /etc/profile
#2. 设置alias的自动补全：
source <(kubectl completion bash | sed 's/kubectl/k/g')
#3. 解决每次启动k都会失效，要重新刷新环境变量（source /etc/profile）的问题：在~/.bashrc文件中添加以下代码：source /etc/profile
echo "source /etc/profile" >> ~/.bashrc
#4. 给alias k也配置补全
tee -a ~/.bashrc <<'EOF'
alias k='kubectl'
complete -o default -F __start_kubectl k
source <(kubectl completion bash)
EOF
##使命令生效
source ~/.bashrc

扩容工作节点

#master上
kubeadm token create --print-join-command
#把rnode1加入k8s集群，加上参数：--ignore-preflight-errors=SystemVerification
#打工作节点label
kubectl label nodes rn1 node-role.kubernetes.io/work=worker

安装calico

#上传calico镜像包和yaml文件，解压
##在线下载地址为：https://docs.projectcalico.org/manifests/calico.yaml
ctr -n=k8s.io images import calico.tar.gz
#修改calico.yaml中网卡名称
#- name: CLUSTER_TYPE
#  value: "k8s,bgp"
# 上面这一段的下面添加：（指定calico从ens33跨节点通信，不指定的话，会走lo的IP，没办法跨节点通信，也就没办法给pod划分IP）
- name: IP_AUTODETECTION_METHOD
  value: "interface=ens160"

kubectl apply -f calico.yaml

#测试网络访问
kubectl run busybox --image busybox:1.28  --image-pull-policy=IfNotPresent --restart=Never --rm -it busybox -- sh
# ping www.baidu.com
# nslookup kubernetes.default.svc.cluster.local

延长k8s证书

查看证书过期时间

#在master节点上执行
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text | grep -i Not
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -i Not

延长证书有效期的脚本

#!/bin/bash
#hanxianchao,微信: luckylucky421302
set -o errexit
set -o pipefail
# set -o xtrace

log::err() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%N%z')]: \033[31mERROR: \033[0m$@\n"
}

log::info() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%N%z')]: \033[32mINFO: \033[0m$@\n"
}

log::warning() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%N%z')]: \033[33mWARNING: \033[0m$@\n"
}

check_file() {
  if [[ ! -r  ${1} ]]; then
    log::err "can not find ${1}"
    exit 1
  fi
}

# get x509v3 subject alternative name from the old certificate
cert::get_subject_alt_name() {
  local cert=${1}.crt
  check_file "${cert}"
  local alt_name=$(openssl x509 -text -noout -in ${cert} | grep -A1 'Alternative' | tail -n1 | sed 's/[[:space:]]*Address//g')
  printf "${alt_name}\n"
}

# get subject from the old certificate
cert::get_subj() {
  local cert=${1}.crt
  check_file "${cert}"
  local subj=$(openssl x509 -text -noout -in ${cert}  | grep "Subject:" | sed 's/Subject:/\//g;s/\,/\//;s/[[:space:]]//g')
  printf "${subj}\n"
}

cert::backup_file() {
  local file=${1}
  if [[ ! -e ${file}.old-$(date +%Y%m%d) ]]; then
    cp -rp ${file} ${file}.old-$(date +%Y%m%d)
    log::info "backup ${file} to ${file}.old-$(date +%Y%m%d)"
  else
    log::warning "does not backup, ${file}.old-$(date +%Y%m%d) already exists"
  fi
}

# generate certificate whit client, server or peer
# Args:
#   $1 (the name of certificate)
#   $2 (the type of certificate, must be one of client, server, peer)
#   $3 (the subject of certificates)
#   $4 (the validity of certificates) (days)
#   $5 (the x509v3 subject alternative name of certificate when the type of certificate is server or peer)
cert::gen_cert() {
  local cert_name=${1}
  local cert_type=${2}
  local subj=${3}
  local cert_days=${4}
  local alt_name=${5}
  local cert=${cert_name}.crt
  local key=${cert_name}.key
  local csr=${cert_name}.csr
  local csr_conf="distinguished_name = dn\n[dn]\n[v3_ext]\nkeyUsage = critical, digitalSignature, keyEncipherment\n"

  check_file "${key}"
  check_file "${cert}"

  # backup certificate when certificate not in ${kubeconf_arr[@]}
  # kubeconf_arr=("controller-manager.crt" "scheduler.crt" "admin.crt" "kubelet.crt")
  # if [[ ! "${kubeconf_arr[@]}" =~ "${cert##*/}" ]]; then
  #   cert::backup_file "${cert}"
  # fi

  case "${cert_type}" in
    client)
      openssl req -new  -key ${key} -subj "${subj}" -reqexts v3_ext \
        -config <(printf "${csr_conf} extendedKeyUsage = clientAuth\n") -out ${csr}
      openssl x509 -in ${csr} -req -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -extensions v3_ext \
        -extfile <(printf "${csr_conf} extendedKeyUsage = clientAuth\n") -days ${cert_days} -out ${cert}
      log::info "generated ${cert}"
    ;;
    server)
      openssl req -new  -key ${key} -subj "${subj}" -reqexts v3_ext \
        -config <(printf "${csr_conf} extendedKeyUsage = serverAuth\nsubjectAltName = ${alt_name}\n") -out ${csr}
      openssl x509 -in ${csr} -req -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -extensions v3_ext \
        -extfile <(printf "${csr_conf} extendedKeyUsage = serverAuth\nsubjectAltName = ${alt_name}\n") -days ${cert_days} -out ${cert}
      log::info "generated ${cert}"
    ;;
    peer)
      openssl req -new  -key ${key} -subj "${subj}" -reqexts v3_ext \
        -config <(printf "${csr_conf} extendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = ${alt_name}\n") -out ${csr}
      openssl x509 -in ${csr} -req -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -extensions v3_ext \
        -extfile <(printf "${csr_conf} extendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = ${alt_name}\n") -days ${cert_days} -out ${cert}
      log::info "generated ${cert}"
    ;;
    *)
      log::err "unknow, unsupported etcd certs type: ${cert_type}, supported type: client, server, peer"
      exit 1
  esac

  rm -f ${csr}
}

cert::update_kubeconf() {
  local cert_name=${1}
  local kubeconf_file=${cert_name}.conf
  local cert=${cert_name}.crt
  local key=${cert_name}.key

  # generate  certificate
  check_file ${kubeconf_file}
  # get the key from the old kubeconf
  grep "client-key-data" ${kubeconf_file} | awk {'print$2'} | base64 -d > ${key}
  # get the old certificate from the old kubeconf
  grep "client-certificate-data" ${kubeconf_file} | awk {'print$2'} | base64 -d > ${cert}
  # get subject from the old certificate
  local subj=$(cert::get_subj ${cert_name})
  cert::gen_cert "${cert_name}" "client" "${subj}" "${CAER_DAYS}"
  # get certificate base64 code
  local cert_base64=$(base64 -w 0 ${cert})

  # backup kubeconf
  # cert::backup_file "${kubeconf_file}"

  # set certificate base64 code to kubeconf
  sed -i 's/client-certificate-data:.*/client-certificate-data: '${cert_base64}'/g' ${kubeconf_file}

  log::info "generated new ${kubeconf_file}"
  rm -f ${cert}
  rm -f ${key}

  # set config for kubectl
  if [[ ${cert_name##*/} == "admin" ]]; then
    mkdir -p ~/.kube
    cp -fp ${kubeconf_file} ~/.kube/config
    log::info "copy the admin.conf to ~/.kube/config for kubectl"
  fi
}

cert::update_etcd_cert() {
  PKI_PATH=${KUBE_PATH}/pki/etcd
  CA_CERT=${PKI_PATH}/ca.crt
  CA_KEY=${PKI_PATH}/ca.key

  check_file "${CA_CERT}"
  check_file "${CA_KEY}"

  # generate etcd server certificate
  # /etc/kubernetes/pki/etcd/server
  CART_NAME=${PKI_PATH}/server
  subject_alt_name=$(cert::get_subject_alt_name ${CART_NAME})
  cert::gen_cert "${CART_NAME}" "peer" "/CN=etcd-server" "${CAER_DAYS}" "${subject_alt_name}"

  # generate etcd peer certificate
  # /etc/kubernetes/pki/etcd/peer
  CART_NAME=${PKI_PATH}/peer
  subject_alt_name=$(cert::get_subject_alt_name ${CART_NAME})
  cert::gen_cert "${CART_NAME}" "peer" "/CN=etcd-peer" "${CAER_DAYS}" "${subject_alt_name}"

  # generate etcd healthcheck-client certificate
  # /etc/kubernetes/pki/etcd/healthcheck-client
  CART_NAME=${PKI_PATH}/healthcheck-client
  cert::gen_cert "${CART_NAME}" "client" "/O=system:masters/CN=kube-etcd-healthcheck-client" "${CAER_DAYS}"

  # generate apiserver-etcd-client certificate
  # /etc/kubernetes/pki/apiserver-etcd-client
  check_file "${CA_CERT}"
  check_file "${CA_KEY}"
  PKI_PATH=${KUBE_PATH}/pki
  CART_NAME=${PKI_PATH}/apiserver-etcd-client
  cert::gen_cert "${CART_NAME}" "client" "/O=system:masters/CN=kube-apiserver-etcd-client" "${CAER_DAYS}"

  # restart etcd
  docker ps | awk '/k8s_etcd/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted etcd"
}

cert::update_master_cert() {
  PKI_PATH=${KUBE_PATH}/pki
  CA_CERT=${PKI_PATH}/ca.crt
  CA_KEY=${PKI_PATH}/ca.key

  check_file "${CA_CERT}"
  check_file "${CA_KEY}"

  # generate apiserver server certificate
  # /etc/kubernetes/pki/apiserver
  CART_NAME=${PKI_PATH}/apiserver
  subject_alt_name=$(cert::get_subject_alt_name ${CART_NAME})
  cert::gen_cert "${CART_NAME}" "server" "/CN=kube-apiserver" "${CAER_DAYS}" "${subject_alt_name}"

  # generate apiserver-kubelet-client certificate
  # /etc/kubernetes/pki/apiserver-kubelet-client
  CART_NAME=${PKI_PATH}/apiserver-kubelet-client
  cert::gen_cert "${CART_NAME}" "client" "/O=system:masters/CN=kube-apiserver-kubelet-client" "${CAER_DAYS}"

  # generate kubeconf for controller-manager,scheduler,kubectl and kubelet
  # /etc/kubernetes/controller-manager,scheduler,admin,kubelet.conf
  cert::update_kubeconf "${KUBE_PATH}/controller-manager"
  cert::update_kubeconf "${KUBE_PATH}/scheduler"
  cert::update_kubeconf "${KUBE_PATH}/admin"
  set +e
  grep kubelet-client-current.pem /etc/kubernetes/kubelet.conf > /dev/null 2>&1
  kubelet_cert_auto_update=$?
  set -e
  if [[ "$kubelet_cert_auto_update" == "0" ]]; then
    log::warning "does not need to update kubelet.conf"
  else
    cert::update_kubeconf "${KUBE_PATH}/kubelet"
  fi

  # generate front-proxy-client certificate
  # use front-proxy-client ca
  CA_CERT=${PKI_PATH}/front-proxy-ca.crt
  CA_KEY=${PKI_PATH}/front-proxy-ca.key
  check_file "${CA_CERT}"
  check_file "${CA_KEY}"
  CART_NAME=${PKI_PATH}/front-proxy-client
  cert::gen_cert "${CART_NAME}" "client" "/CN=front-proxy-client" "${CAER_DAYS}"

  # restart apiserve, controller-manager, scheduler and kubelet
  docker ps | awk '/k8s_kube-apiserver/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted kube-apiserver"
  docker ps | awk '/k8s_kube-controller-manager/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted kube-controller-manager"
  docker ps | awk '/k8s_kube-scheduler/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted kube-scheduler"
  systemctl restart kubelet
  log::info "restarted kubelet"
}

main() {
  local node_tpye=$1

  KUBE_PATH=/etc/kubernetes
  CAER_DAYS=36500

  # backup $KUBE_PATH to $KUBE_PATH.old-$(date +%Y%m%d)
  cert::backup_file "${KUBE_PATH}"

  case ${node_tpye} in
    etcd)
      # update etcd certificates
      cert::update_etcd_cert
    ;;
    master)
      # update master certificates and kubeconf
      cert::update_master_cert
    ;;
    all)
      # update etcd certificates
      cert::update_etcd_cert
      # update master certificates and kubeconf
      cert::update_master_cert
    ;;
    *)
      log::err "unknow, unsupported certs type: ${cert_type}, supported type: all, etcd, master"
      printf "Documentation: hanxianchao
  example:
    '\033[32m./update-kubeadm-cert.sh all\033[0m' update all etcd certificates, master certificates and kubeconf
      /etc/kubernetes
      ├── admin.conf
      ├── controller-manager.conf
      ├── scheduler.conf
      ├── kubelet.conf
      └── pki
          ├── apiserver.crt
          ├── apiserver-etcd-client.crt
          ├── apiserver-kubelet-client.crt
          ├── front-proxy-client.crt
          └── etcd
              ├── healthcheck-client.crt
              ├── peer.crt
              └── server.crt

    '\033[32m./update-kubeadm-cert.sh etcd\033[0m' update only etcd certificates
      /etc/kubernetes
      └── pki
          ├── apiserver-etcd-client.crt
          └── etcd
              ├── healthcheck-client.crt
              ├── peer.crt
              └── server.crt

    '\033[32m./update-kubeadm-cert.sh master\033[0m' update only master certificates and kubeconf
      /etc/kubernetes
      ├── admin.conf
      ├── controller-manager.conf
      ├── scheduler.conf
      ├── kubelet.conf
      └── pki
          ├── apiserver.crt
          ├── apiserver-kubelet-client.crt
          └── front-proxy-client.crt
"
      exit 1
    esac
}

main "$@"

运行脚本

chmod +x update-kubeadm-cert.sh
./update-kubeadm-cert.sh all

#检查kube-system pod
kubectl get pods -n kube-system

#检查证书有效期
#在master节点上
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text | grep -i Not
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -i Not