K8s架构与核心组件

master组件

API server
集群对外的统一入口，以RESTful API方式做操作。
scheduler
通过算法算出pod调度到哪个节点
controller manager
管理一系列的controller（deployment、stateful set、namespace等）
etcd
存储系统
Calico：网络插件（给pod提供IP、提供网络policy等）
kubeproxy

Worker Node组件

kubelet
kube-proxy
coreDns
calico
runtime（docker/containerd）

环境准备

实验环境规划

操作系统：rockylinux8（yum update -y升级后会变成rockyulinux9），下载地址：Rocky Linux下载（选择Minimal ISO）
网络：NAT模式，可以确保机器在任何网络都能ssh连接

K8S集群角色	IP	主机名
控制节点	192.168.40.180	rm1
工作节点	192.168.40.181	rn1

安装前准备

配置基础源

sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
    -i.bak \
    /etc/yum.repos.d/rocky-*.repo

dnf makecache

可以去阿里云镜像源网站：[developer.aliyun.com/mirror](https://developer.aliyun.com/mirror) 找到对应系统的换源操作。

软件包安装

yum update -y
    yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo libaio-devel wget vim ncurses-devel autoconf automake epel-release openssh-server telnet vim coreutils iputils iproute nmap-ncat jq psmisc telnet git bash-completion

配置网络

#查看网卡名
ip addr

nmcli connection modify ens33 ipv4.method manual connection.autoconnect yes #修改IP地址为手动配置，并且设置开机启动
nmcli connection modify ens33 ipv4.addresses 192.168.40.180/24 #修改IP地址和掩码
nmcli connection modify ens33 ipv4.gateway 192.168.40.2 #修改网关
nmcli connection modify ens33 ipv4.dns 192.168.40.2 #修改DNS，多个DNS以逗号分隔
nmcli connection down ens33 && nmcli connection up ens33 
# systemctl restart NetworkManager 也行

NetworkManager 会把修改写入 `/etc/NetworkManager/system-connections/ens160.nmconnection`；手动编辑该文件也可，但改完需 `nmcli connection reload`

关闭selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
reboot

配置主机名

hostnamectl set-hostname rm1 && bash

配置hosts

tee -a /etc/hosts <<'EOF'
192.168.40.180   rm1  
192.168.40.181   rn1
192.168.40.182   rn2
EOF

配置ssh互信

ssh-keygen
ssh-copy-id rn1

关闭swap，并把swap的lv重新分配到/ 【注意这一步执行之前先打快照。有可能删掉之后无法启动，那就别删了】

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
systemctl daemon-reload

vim /etc/fstab
#/dev/mapper/centos-swap swap      swap    defaults        0 0

lvs
#看到swap占用了3.91G的LVM
  #root rl -wi-ao---- 35.08g                                                    
  #swap rl -wi-a-----  3.91g
lvdisplay
#获取到LV Path /dev/rl/swap
#删除这个lv
lvremove /dev/rl/swap
#获取vg剩余空间
vgs
#VFree = 3.91G
#扩容lv
lvextend -L +3.91G /dev/rl/root
#扩容文件系统（xfs系统）
xfs_growfs /
#检查根分区
df -h

删除交换分区后必须重新配置 grub、BLS 引导条目、dracut 和 initramfs，否则重启会卡住或报 "找不到 swap" 错误

# 1. 查看当前 GRUB_CMDLINE_LINUX 的完整内容（确认要删的参数）
grep GRUB_CMDLINE_LINUX /etc/default/grub
# 典型输出（VG 名可能含主机名，如 rl_rm1，也可能是 rl）：
# GRUB_CMDLINE_LINUX="crashkernel=... resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet"

# 2. 修改 /etc/default/grub，删除 swap 相关参数（影响未来新装内核的默认参数）
#    删除 resume= 参数（指向 swap 设备，用于休眠恢复）
sed -i 's/ resume=[^ "]*//' /etc/default/grub
#    删除 rd.lvm.lv=.../swap 参数（告诉 dracut 激活 swap LV）
sed -i 's/ rd\.lvm\.lv=[^ "]*swap//' /etc/default/grub

# 3. 验证 /etc/default/grub 修改结果
grep GRUB_CMDLINE_LINUX /etc/default/grub

# 4.【关键】使用 grubby 更新已有的 BLS 引导条目
#    Rocky Linux 8/9 也使用 BLS（Boot Loader Specification），内核启动参数存储在
#    /boot/loader/entries/*.conf 中。仅修改 /etc/default/grub 不会更新已有的 BLS 条目！
#    必须用 grubby 删除所有已有内核条目中的 swap 相关参数：
grubby --update-kernel=ALL --remove-args="resume rd.lvm.lv=rl/swap"
# 注意：rd.lvm.lv=rl/swap 中的 rl 替换为实际 VG 名（通过 vgs 查看）

# 验证 BLS 条目已更新（确认 resume 和 swap 参数已消失）
grubby --info=ALL | grep -E "resume|swap" || echo "BLS 条目已清理干净"

# 5. 清除 dracut 的 resume 模块配置
#    dracut 有独立的 resume 配置，仅清理引导参数不够！
#    dracut -f 时 resume 模块仍会自动将已删除的 swap 设备写入 initramfs，
#    导致启动时 initramfs 尝试激活不存在的 swap LV 用于休眠恢复，系统卡住。
rm -f /etc/dracut.conf.d/resume.conf
# 显式禁用 resume 模块，防止 dracut 自动探测残留的 swap 引用
echo 'omit_dracutmodules+=" resume "' > /etc/dracut.conf.d/noresume.conf
# 如果存在 systemd-hibernate-resume 服务也一并禁用
systemctl disable systemd-hibernate-resume@*.service 2>/dev/null || true

# 6. 重新生成 initramfs（此步不可跳过，必须在上一步清除 resume 配置之后执行）
dracut -f

# 7. 重新生成 grub 配置（无论 UEFI 还是 Legacy BIOS 统一用此命令）
grub2-mkconfig -o /boot/grub2/grub.cfg

# 8. 重启验证
init 6

如果有扩盘需求，关机后在VMWare中扩盘，进系统扩容LVM：

sh
#lsblk看到nvme0n1盘容量已经扩容了，现在新建分区，扩容到LVM
fdisk /dev/nvme0n1
#新建一个nvme0n1p3分区，type为8e
#w保存
partprobe /dev/nvme0n1
pvcreate /dev/nvme0n1p3
vgextend rl /dev/nvme0n1p3
lvextend -L +40G /dev/rl/root
xfs_growfs /dev/rl/root

修改内核参数

#将br_netfilter模块添加到 Linux 内核中。这是Linux网桥的核心模块，提供网络包转发和过滤。Kubernetes需要网络插件（如 Flannel、Calico）实现容器间通信，而这些网络插件通常会用Linux网桥来进行数据包转发和过滤。
modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/profile
#开启包过滤
tee /etc/sysctl.d/k8s.conf <<'EOF'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl -p /etc/sysctl.d/k8s.conf

关闭防火墙和dnsmasp

systemctl stop firewalld && systemctl disable firewalld
systemctl disable --now dnsmasq

配置时间同步

yum -y install chrony
systemctl enable chronyd --now 

tee -a /etc/chrony.conf <<'EOF'  
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp1.tencent.com iburst
server ntp2.tencent.com iburst
EOF
#Iburst: 如果设置该选项，前四轮询之间的间隔将是2秒，而不是minpoll。 这对于在chronyd启动后快速获得时钟的第一次更新非常有用。即：重启chronyd后，2秒后，直接同步一次时间。
crontab -e
* * * * * /usr/bin/systemctl restart chronyd
systemctl restart crond

启用ipvs

cat >> /etc/modules-load.d/ipvs.conf << EOF
br_netfilter
ip_conntrack
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

# 安装依赖
dnf install ipvsadm ipset sysstat conntrack libseccomp -y
# 重启服务
systemctl restart systemd-modules-load.service
# 查看模块内容
lsmod | grep -e ip_vs -e nf_conntrack

句柄数设置最大

# 设置为最大
ulimit -SHn 65535

# soft nofile 655360 软限制：单个进程最多同时打开 655,360 个文件
# soft nproc 655350 软限制：单个用户最多创建 655,350 个进程
# soft memlock unlimited 软限制：可锁定到内存的数据量（无限制）
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 655360
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

# 查看修改结果
ulimit -a

【可选】系统优化（针对8C8G的虚拟机，其余配置可以相应更改）

cat > /etc/sysctl.d/k8s_better.conf << EOF
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=16384
fs.inotify.max_user_watches=2097152 
fs.file-max=104857600                    
fs.nr_open=104857600
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=4621440
EOF

modprobe br_netfilter
lsmod |grep conntrack
modprobe ip_conntrack
sysctl -p /etc/sysctl.d/k8s_better.conf

内核参数说明

vm.swappiness=0 # 禁用交换分区，Kubernetes 要求禁用 swap
vm.overcommit_memory=1  # 允许内存过量分配，提高容器密度
vm.panic_on_oom=0 # OOM 时不让系统崩溃，而是杀死进程

# 增加文件系统监控的限制，Kubernetes 组件需要监控大量文件变化（如配置文件、日志）。防止 "too many open files" 错误
fs.inotify.max_user_instances=16384
fs.inotify.max_user_watches=2097152 

# 大幅增加系统可打开的文件描述符数量
fs.file-max=104857600                    
fs.nr_open=104857600

# 禁用IPv6
net.ipv6.conf.all.disable_ipv6=1

#增加网络连接跟踪表的大小
net.netfilter.nf_conntrack_max=4621440

配置aliyun docker repo源

#配置国内安装docker和containerd的阿里云在线源
yum install yum-utils -y
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum clean all && yum makecache

组件安装

docker

#docker也要安装，docker跟containerd不冲突，安装docker是为了能基于dockerfile构建镜像
#安装24.0.6的docker，是为了不与containerd版本冲突。否则docker版本过高，containerd也跟着变成高版本了。
yum install docker-ce-24.0.6 -y
systemctl enable docker --now
#配置docker镜像加速器，k8s所有节点均按照以下配置
tee /etc/docker/daemon.json <<'EOF'
{
 "registry-mirrors":["https://docker.m.daocloud.io"]
} 
EOF
#重启docker：
systemctl daemon-reload && systemctl restart docker && systemctl status docker

containerd

当前k8s版本推荐1.6.22版本的containerd，比较稳定。

yum install containerd.io-1.6.22* -y
cd /etc/containerd
rm -rf *

cat <<EOF | tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter
sysctl --system

如果有harbor环境，修改config.toml为以下。（将\替换为安装harbor机器的IP），除此之外，还修改了以下内容：
SystemdCgroup = true
sandbox_image修改成了aliyun镜像库
可以用sed来替换：

sudo mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
sed -i 's#k8s.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g'  /etc/containerd/config.toml
sed -i 's#registry.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g'  /etc/containerd/config.toml
sed -i 's#registry.k8s.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g'  /etc/containerd/config.toml

完整配置文件如下：

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = true

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.hanxux.local".tls]
            insecure_skip_verify = true
            ca_file = ""
            cert_file = ""
            key_file = ""
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.hanxux.local".auth]
            username = "admin"
            password = "Harbor12345"

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
         [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.hanxux.local"]
            endpoint = ["http://harbor.hanxux.local"]
          [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
             endpoint = ["https://docker.m.daocloud.io"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = ""
    insecure = false
    protocol = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

systemctl daemon-reload && systemctl start containerd && systemctl enable containerd

# 修改/etc/crictl.yaml文件，指定k8s的运行时和拉取镜像都使用containerd
# 这个文件是 crictl 工具的配置文件，用于指定与 containerd 交互的相关设置：
# 指定unix:///run/containerd/containerd.sock 是为了告诉 crictl 使用 Unix 域套接字的方式来连接 containerd 的 API。containerd 提供了一个 socket 文件 /run/containerd/containerd.sock，crictl 通过连接该 socket 文件，可以与 containerd 进行通信，管理容器和镜像等操作。
# runtime-endpoint：指定 containerd 的运行时接口地址：crictl 可以与 containerd 通信来管理容器生命周期和资源隔离。
# image-endpoint：指定 containerd 的镜像接口地址：以便 crictl 可以与 containerd 通信来管理镜像的拉取和推送。
# timeout：指定 crictl 等待 containerd 响应的最大时间，避免出现无响应的情况。
# debug：开启或关闭 crictl 的调试模式，方便排查问题。
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

systemctl restart containerd
# 验证配置
containerd --version
# [可选]
crictl info

crictl是一个命令行工具，用来验证 CRI 兼容性。若报错没有安装 crictl，可以执行：

VERSION="1.25.0"
curl -L -o /usr/local/bin/crictl "https://github.com/kubernetes-sigs/cri-tools/releases/download/v${VERSION}/crictl-v${VERSION}-linux-amd64.tar.gz"
tar zxvf /usr/local/bin/crictl -C /usr/local/bin
chmod +x /usr/local/bin/crictl

检查containerd插件情况

ctr plugin ls
# 下面这两项都要是ok才行
io.containerd.snapshotter.v1          overlayfs                linux/amd64    ok
io.containerd.grpc.v1                 cri                      linux/amd64    ok

k8s-1.33.0

#配置安装k8s组件需要的阿里云的repo源 
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.33/rpm/repodata/repomd.xml.key
EOF
#其中baseurl是指定仓库的URL地址。这个仓库是Kubernetes针对CentOS 7 x86_64架构的yum仓库，yum会从这个指定的URL地址获取Kubernetes的安装包和依赖包，从而进行安装。使用阿里云镜像站可以提高安装速度和稳定性。
yum clean all && yum makecache
yum install -y kubelet-1.33.0 kubeadm-1.33.0 kubectl-1.33.0
# 也可以用 yum install-y kubelet-1.33.0 kubeadm-1.33.0 kubectl-1.33.0 --disableexcludes=kubernetes
# 上面命令确保安装时优先使用我们配置的 Kubernetes 仓库，而不是其他默认仓库。
# 先把kubelet开起来
systemctl enable kubelet --now

如果安装报找不到指定的源，可以按照下面的网址，下载指定版本的rpm文件，传到linux机器：

https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.33

https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.33/rpm/x86_64/?spm=a2c6h.25603864.0.0.7ced7af3l8FcI0

基于上面下载到linux机器的rpm文件，如下载到我linux的指定目录k8s-1.33，手动安装：

rpm -ivh ./*

kubeadm初始化k8s集群

kubedam和二进制安装k8s

kubeadm是官方提供的开源工具，是一个开源项目，用于快速搭建kubernetes集群，目前是比较方便和推荐使用的。kubeadm init 以及 kubeadm join 这两个命令可以快速创建 kubernetes 集群。Kubeadm初始化k8s，所有的组件都是以pod形式运行的，具备故障自恢复能力。kubeadm适合需要经常部署k8s，或者对自动化要求比较高的场景下使用。
二进制：在官网下载相关组件的二进制包，如果手动安装，对kubernetes理解也会更全面。
Kubeadm和二进制都适合生产环境，在生产环境运行都很稳定，具体如何选择，可以根据实际项目进行评估。

初始化集群

在一个master节点操作

kubeadm config print init-defaults > kubeadm.yaml

tee kubeadm.yaml <<'EOF'
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.40.180 #控制节点的ip
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock  #指定containerd容器运行时
  imagePullPolicy: IfNotPresent
  name:  rm1 #控制节点主机名
  taints: null
---
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # 指定国内镜像源
kind: ClusterConfiguration
kubernetesVersion: 1.33.0 # 这里指定K8s版本
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16 
  serviceSubnet: 10.96.0.0/12 
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

# 【可选】提前下载镜像，版本要改成具体安装的版本
kubeadm config images pull \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --kubernetes-version 1.33.0

# 初始化master节点。init命令仅在第一个master上执行。其余master和node节点全部是join进来
kubeadm init --config=kubeadm.yaml --ignore-preflight-errors=SystemVerification

kubeproxy模式采用ipvs：

如果不指定ipvs，会默认使用iptables。开启IPVS可以提高Kubernetes集群的性能和可靠性。IPVS是一个高性能的负载均衡器，与Kubernetes Service一起使用，可以在Pod之间分配负载。相比于Kubernetes自带的iptables模式，IPVS模式在处理大量流量时具有更好的性能和可靠性。此外，IPVS还支持四层和七层协议的负载均衡，能够满足更多的应用场景需求，所以我们生产环境建议开启ipvs。

配置kubeconfig

# admin.conf是集群的钥匙，安全起见最好是仅在这一台master上执行
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 1. 设置kubectl的alias为k：
whereis kubectl ## kubectl: /usr/bin/kubectl
echo "alias k=/usr/bin/kubectl">>/etc/profile
source /etc/profile

# 2. 设置alias的自动补全：
source <(kubectl completion bash | sed 's/kubectl/k/g')

# 3. 解决每次启动k都会失效，要重新刷新环境变量（source /etc/profile）的问题：在~/.bashrc文件中添加以下代码：source /etc/profile
echo "source /etc/profile" >> ~/.bashrc

# 4. 给alias k也配置补全
tee -a ~/.bashrc <<'EOF'
alias k='kubectl'
complete -o default -F __start_kubectl k
source <(kubectl completion bash)
EOF

# 使命令生效
source ~/.bashrc

扩容工作节点

# master上
kubeadm token create --print-join-command
# 把rnode1加入k8s集群，加上参数：--ignore-preflight-errors=SystemVerification
# 打工作节点label
kubectl label nodes rn1 node-role.kubernetes.io/work=worker

安装calico

从官网找到最新版calico的yaml链接（在manifest一栏中找到Download the Calico networking manifest for the Kubernetes API datastore.）：

https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico

下载yaml文件部署

# 修改calico.yaml中网卡名称
# - name: CLUSTER_TYPE
#   value: "k8s,bgp"
# 上面这一段的下面添加：（指定calico从ens33跨节点通信，不指定的话，会走lo的IP，没办法跨节点通信，也就没办法给pod划分IP）
- name: IP_AUTODETECTION_METHOD
  value: "interface=ens33"

kubectl apply -f calico.yaml

#测试网络访问
kubectl run busybox --image busybox:1.28  --image-pull-policy=IfNotPresent --restart=Never --rm -it busybox -- sh
# ping www.baidu.com
# nslookup kubernetes.default.svc.cluster.local

【可选】安装Addon

# 杜宽老师的Addon仓库
git clone https://gitee.com/dukuan/k8s-ha-install.git
cd k8s-ha-install/
# 查看最新版本的分支
git branch -a
git checkout manual-installation-v1.33.x # 切换到最新的版本分支 
cd single/
# 里面有calico、dashboard、krm（杜宽老师自己开发的多集群管理工具）
# krm默认用户名密码是admin/admin，进去之后在集群管理里面添加集群
kubectl apply -f comp.yaml  dashboard-user.yaml  dashboard.yaml  krm.yaml

延长k8s证书

查看证书过期时间

#在master节点上执行
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text | grep -i Not
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -i Not

延长证书有效期的脚本

文件命名为：update-kubeadm-cert.sh

#!/bin/bash
set -o errexit
set -o pipefail
# set -o xtrace

log::err() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%N%z')]: \033[31mERROR: \033[0m$@\n"
}

log::info() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%N%z')]: \033[32mINFO: \033[0m$@\n"
}

log::warning() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%N%z')]: \033[33mWARNING: \033[0m$@\n"
}

check_file() {
  if [[ ! -r  ${1} ]]; then
    log::err "can not find ${1}"
    exit 1
  fi
}

# get x509v3 subject alternative name from the old certificate
cert::get_subject_alt_name() {
  local cert=${1}.crt
  check_file "${cert}"
  local alt_name=$(openssl x509 -text -noout -in ${cert} | grep -A1 'Alternative' | tail -n1 | sed 's/[[:space:]]*Address//g')
  printf "${alt_name}\n"
}

# get subject from the old certificate
cert::get_subj() {
  local cert=${1}.crt
  check_file "${cert}"
  local subj=$(openssl x509 -text -noout -in ${cert}  | grep "Subject:" | sed 's/Subject:/\//g;s/\,/\//;s/[[:space:]]//g')
  printf "${subj}\n"
}

cert::backup_file() {
  local file=${1}
  if [[ ! -e ${file}.old-$(date +%Y%m%d) ]]; then
    cp -rp ${file} ${file}.old-$(date +%Y%m%d)
    log::info "backup ${file} to ${file}.old-$(date +%Y%m%d)"
  else
    log::warning "does not backup, ${file}.old-$(date +%Y%m%d) already exists"
  fi
}

# generate certificate whit client, server or peer
# Args:
#   $1 (the name of certificate)
#   $2 (the type of certificate, must be one of client, server, peer)
#   $3 (the subject of certificates)
#   $4 (the validity of certificates) (days)
#   $5 (the x509v3 subject alternative name of certificate when the type of certificate is server or peer)
cert::gen_cert() {
  local cert_name=${1}
  local cert_type=${2}
  local subj=${3}
  local cert_days=${4}
  local alt_name=${5}
  local cert=${cert_name}.crt
  local key=${cert_name}.key
  local csr=${cert_name}.csr
  local csr_conf="distinguished_name = dn\n[dn]\n[v3_ext]\nkeyUsage = critical, digitalSignature, keyEncipherment\n"

  check_file "${key}"
  check_file "${cert}"

  # backup certificate when certificate not in ${kubeconf_arr[@]}
  # kubeconf_arr=("controller-manager.crt" "scheduler.crt" "admin.crt" "kubelet.crt")
  # if [[ ! "${kubeconf_arr[@]}" =~ "${cert##*/}" ]]; then
  #   cert::backup_file "${cert}"
  # fi

  case "${cert_type}" in
    client)
      openssl req -new  -key ${key} -subj "${subj}" -reqexts v3_ext \
        -config <(printf "${csr_conf} extendedKeyUsage = clientAuth\n") -out ${csr}
      openssl x509 -in ${csr} -req -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -extensions v3_ext \
        -extfile <(printf "${csr_conf} extendedKeyUsage = clientAuth\n") -days ${cert_days} -out ${cert}
      log::info "generated ${cert}"
    ;;
    server)
      openssl req -new  -key ${key} -subj "${subj}" -reqexts v3_ext \
        -config <(printf "${csr_conf} extendedKeyUsage = serverAuth\nsubjectAltName = ${alt_name}\n") -out ${csr}
      openssl x509 -in ${csr} -req -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -extensions v3_ext \
        -extfile <(printf "${csr_conf} extendedKeyUsage = serverAuth\nsubjectAltName = ${alt_name}\n") -days ${cert_days} -out ${cert}
      log::info "generated ${cert}"
    ;;
    peer)
      openssl req -new  -key ${key} -subj "${subj}" -reqexts v3_ext \
        -config <(printf "${csr_conf} extendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = ${alt_name}\n") -out ${csr}
      openssl x509 -in ${csr} -req -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -extensions v3_ext \
        -extfile <(printf "${csr_conf} extendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = ${alt_name}\n") -days ${cert_days} -out ${cert}
      log::info "generated ${cert}"
    ;;
    *)
      log::err "unknow, unsupported etcd certs type: ${cert_type}, supported type: client, server, peer"
      exit 1
  esac

  rm -f ${csr}
}

cert::update_kubeconf() {
  local cert_name=${1}
  local kubeconf_file=${cert_name}.conf
  local cert=${cert_name}.crt
  local key=${cert_name}.key

  # generate  certificate
  check_file ${kubeconf_file}
  # get the key from the old kubeconf
  grep "client-key-data" ${kubeconf_file} | awk {'print$2'} | base64 -d > ${key}
  # get the old certificate from the old kubeconf
  grep "client-certificate-data" ${kubeconf_file} | awk {'print$2'} | base64 -d > ${cert}
  # get subject from the old certificate
  local subj=$(cert::get_subj ${cert_name})
  cert::gen_cert "${cert_name}" "client" "${subj}" "${CAER_DAYS}"
  # get certificate base64 code
  local cert_base64=$(base64 -w 0 ${cert})

  # backup kubeconf
  # cert::backup_file "${kubeconf_file}"

  # set certificate base64 code to kubeconf
  sed -i 's/client-certificate-data:.*/client-certificate-data: '${cert_base64}'/g' ${kubeconf_file}

  log::info "generated new ${kubeconf_file}"
  rm -f ${cert}
  rm -f ${key}

  # set config for kubectl
  if [[ ${cert_name##*/} == "admin" ]]; then
    mkdir -p ~/.kube
    cp -fp ${kubeconf_file} ~/.kube/config
    log::info "copy the admin.conf to ~/.kube/config for kubectl"
  fi
}

cert::update_etcd_cert() {
  PKI_PATH=${KUBE_PATH}/pki/etcd
  CA_CERT=${PKI_PATH}/ca.crt
  CA_KEY=${PKI_PATH}/ca.key

  check_file "${CA_CERT}"
  check_file "${CA_KEY}"

  # generate etcd server certificate
  # /etc/kubernetes/pki/etcd/server
  CART_NAME=${PKI_PATH}/server
  subject_alt_name=$(cert::get_subject_alt_name ${CART_NAME})
  cert::gen_cert "${CART_NAME}" "peer" "/CN=etcd-server" "${CAER_DAYS}" "${subject_alt_name}"

  # generate etcd peer certificate
  # /etc/kubernetes/pki/etcd/peer
  CART_NAME=${PKI_PATH}/peer
  subject_alt_name=$(cert::get_subject_alt_name ${CART_NAME})
  cert::gen_cert "${CART_NAME}" "peer" "/CN=etcd-peer" "${CAER_DAYS}" "${subject_alt_name}"

  # generate etcd healthcheck-client certificate
  # /etc/kubernetes/pki/etcd/healthcheck-client
  CART_NAME=${PKI_PATH}/healthcheck-client
  cert::gen_cert "${CART_NAME}" "client" "/O=system:masters/CN=kube-etcd-healthcheck-client" "${CAER_DAYS}"

  # generate apiserver-etcd-client certificate
  # /etc/kubernetes/pki/apiserver-etcd-client
  check_file "${CA_CERT}"
  check_file "${CA_KEY}"
  PKI_PATH=${KUBE_PATH}/pki
  CART_NAME=${PKI_PATH}/apiserver-etcd-client
  cert::gen_cert "${CART_NAME}" "client" "/O=system:masters/CN=kube-apiserver-etcd-client" "${CAER_DAYS}"

  # restart etcd
  docker ps | awk '/k8s_etcd/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted etcd"
}

cert::update_master_cert() {
  PKI_PATH=${KUBE_PATH}/pki
  CA_CERT=${PKI_PATH}/ca.crt
  CA_KEY=${PKI_PATH}/ca.key

  check_file "${CA_CERT}"
  check_file "${CA_KEY}"

  # generate apiserver server certificate
  # /etc/kubernetes/pki/apiserver
  CART_NAME=${PKI_PATH}/apiserver
  subject_alt_name=$(cert::get_subject_alt_name ${CART_NAME})
  cert::gen_cert "${CART_NAME}" "server" "/CN=kube-apiserver" "${CAER_DAYS}" "${subject_alt_name}"

  # generate apiserver-kubelet-client certificate
  # /etc/kubernetes/pki/apiserver-kubelet-client
  CART_NAME=${PKI_PATH}/apiserver-kubelet-client
  cert::gen_cert "${CART_NAME}" "client" "/O=system:masters/CN=kube-apiserver-kubelet-client" "${CAER_DAYS}"

  # generate kubeconf for controller-manager,scheduler,kubectl and kubelet
  # /etc/kubernetes/controller-manager,scheduler,admin,kubelet.conf
  cert::update_kubeconf "${KUBE_PATH}/controller-manager"
  cert::update_kubeconf "${KUBE_PATH}/scheduler"
  cert::update_kubeconf "${KUBE_PATH}/admin"
  set +e
  grep kubelet-client-current.pem /etc/kubernetes/kubelet.conf > /dev/null 2>&1
  kubelet_cert_auto_update=$?
  set -e
  if [[ "$kubelet_cert_auto_update" == "0" ]]; then
    log::warning "does not need to update kubelet.conf"
  else
    cert::update_kubeconf "${KUBE_PATH}/kubelet"
  fi

  # generate front-proxy-client certificate
  # use front-proxy-client ca
  CA_CERT=${PKI_PATH}/front-proxy-ca.crt
  CA_KEY=${PKI_PATH}/front-proxy-ca.key
  check_file "${CA_CERT}"
  check_file "${CA_KEY}"
  CART_NAME=${PKI_PATH}/front-proxy-client
  cert::gen_cert "${CART_NAME}" "client" "/CN=front-proxy-client" "${CAER_DAYS}"

  # restart apiserve, controller-manager, scheduler and kubelet
  docker ps | awk '/k8s_kube-apiserver/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted kube-apiserver"
  docker ps | awk '/k8s_kube-controller-manager/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted kube-controller-manager"
  docker ps | awk '/k8s_kube-scheduler/{print$1}' | xargs -r -I '{}' docker restart {} || true
  log::info "restarted kube-scheduler"
  systemctl restart kubelet
  log::info "restarted kubelet"
}

main() {
  local node_tpye=$1

  KUBE_PATH=/etc/kubernetes
  CAER_DAYS=36500

  # backup $KUBE_PATH to $KUBE_PATH.old-$(date +%Y%m%d)
  cert::backup_file "${KUBE_PATH}"

  case ${node_tpye} in
    etcd)
      # update etcd certificates
      cert::update_etcd_cert
    ;;
    master)
      # update master certificates and kubeconf
      cert::update_master_cert
    ;;
    all)
      # update etcd certificates
      cert::update_etcd_cert
      # update master certificates and kubeconf
      cert::update_master_cert
    ;;
    *)
      log::err "unknow, unsupported certs type: ${cert_type}, supported type: all, etcd, master"
      printf "Documentation:
  example:
    '\033[32m./update-kubeadm-cert.sh all\033[0m' update all etcd certificates, master certificates and kubeconf
      /etc/kubernetes
      ├── admin.conf
      ├── controller-manager.conf
      ├── scheduler.conf
      ├── kubelet.conf
      └── pki
          ├── apiserver.crt
          ├── apiserver-etcd-client.crt
          ├── apiserver-kubelet-client.crt
          ├── front-proxy-client.crt
          └── etcd
              ├── healthcheck-client.crt
              ├── peer.crt
              └── server.crt

    '\033[32m./update-kubeadm-cert.sh etcd\033[0m' update only etcd certificates
      /etc/kubernetes
      └── pki
          ├── apiserver-etcd-client.crt
          └── etcd
              ├── healthcheck-client.crt
              ├── peer.crt
              └── server.crt

    '\033[32m./update-kubeadm-cert.sh master\033[0m' update only master certificates and kubeconf
      /etc/kubernetes
      ├── admin.conf
      ├── controller-manager.conf
      ├── scheduler.conf
      ├── kubelet.conf
      └── pki
          ├── apiserver.crt
          ├── apiserver-kubelet-client.crt
          └── front-proxy-client.crt
"
      exit 1
    esac
}

main "$@"

运行脚本

chmod +x update-kubeadm-cert.sh
./update-kubeadm-cert.sh all

#检查kube-system pod
kubectl get pods -n kube-system

#检查证书有效期
#在master节点上
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text | grep -i Not
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -i Not

集群卸载

kubeadm reset -f
rm -rf ~/.kube/
rm -rf /etc/kubernetes/
rm -rf /etc/systemd/system/kubelet.service.d
rm -rf /etc/systemd/system/kubelet.service
rm -rf /usr/bin/kube* 
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/etcd
rm -rf /var/etcd

Windows下安装kubectl

推荐使用scoop安装：

安装scoop参考scoop安装helm部分
安装kubectl：scoop install kubectl kubelogin

制作k8s安装需要的离线yum源

要全内网环境安装docker、k8s和相关依赖，需要在内部提供安装k8s、docker需要的yum源。首先需要在联网机器上把镜像拉下来。

制作安装docker需要的离线yum源

1、添加docker在线源

yum-config-manager --add-repo https://download.docker.com/linux/centos/dockerce.repo

2、通过如下命令download远程yum源文件，建立本地docker repo库

yum install --downloadonly --downloaddir=/mnt/docker-ce docker-ce
createrepo -d /mnt/docker-ce

3、把/mnt/docker-c下自动下载的rpm打包，传到内网机器，用过如下方法安装：

rpm -Uvh *.rpm --nodeps --force #这是强制安装当前文件夹中所有的rpm包，忽略依赖去安装

制作安装k8s命令行工具需要的离线yum源

1、添加k8s在线源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2、通过如下命令download远程yum源文件，建立本地k8s repo库

yum install --downloadonly --resolve kubeadm kubelet kubectl --destdir /mnt/k8s
createrepo -d /mnt/k8s

3、把/mnt/k8s下自动下载的rpm打包，传到内网机器，用过如下方法安装：

rpm -Uvh *.rpm --nodeps --force #这是强制安装当前文件夹中所有的rpm包，忽略依赖去安装

制作不同版本k8s集群需要的离线镜像

#如果用docker做容器，按照下面方面制作离线镜像
kubeadm config print init-defaults > kubeadm.yaml
#修改kubeadm.yaml配置文件如下：
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
#上述配置表示，安装k8s需要的镜像要从阿里云镜像仓库拉取

#通过如下命令下载镜像：
kubeadm config images pull --config kubeadm.yaml
#然后把下载好的镜像打包镜像：
docker save -o a.tar.gz registry.aliyuncs.com/google_containers/pause:3.7 jenkins/jenkins:latest ....
#传到内网k8s节点，通过如下命令导出镜像：
ctr -n=k8s.io images import a.tar.gz
docker load -i a.tar.gz

#如果用containerd做容器，按照下面方面制作离线镜像
kubeadm config print init-defaults > kubeadm.yaml
#修改kubeadm.yaml配置文件如下：
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
#上述配置表示，安装k8s需要的镜像要从阿里云镜像仓库拉取
#通过如下命令下载镜像：
kubeadm config images pull --config kubeadm.yaml

ctr -n=k8s.io images ls
#ls出来的的所有镜像做到离线文件里
ctr -n=k8s.io images export k8s1.28.0.tar.gz
#导入到内网环境解压
ctr -n=k8s.io images import k8s1.28.0.tar.gz

安装k8s-1.33-基于rockylinux-最新步骤

K8s架构与核心组件

master组件

Worker Node组件

环境准备

实验环境规划

安装前准备

组件安装

docker

containerd

k8s-1.33.0

kubeadm初始化k8s集群

初始化集群

配置kubeconfig

扩容工作节点

安装calico

【可选】安装Addon

延长k8s证书

查看证书过期时间

延长证书有效期的脚本

运行脚本

集群卸载

Windows下安装kubectl

制作k8s安装需要的离线yum源

制作安装docker需要的离线yum源

制作安装k8s命令行工具需要的离线yum源

制作不同版本k8s集群需要的离线镜像