1. 为什么选择二进制安装Kubernetes在云原生技术栈中Kubernetes的部署方式主要分为两大类使用kubeadm等工具自动化安装和手动二进制安装。前者适合快速搭建测试环境而后者则是生产级部署的首选方案。二进制安装意味着我们需要手动下载各个组件的可执行文件并逐一配置启动参数这种方式虽然步骤繁琐但能带来三个关键优势组件版本可控性每个组件如kube-apiserver、etcd等都可以独立选择特定版本避免工具自动安装时产生的版本绑定问题。比如在v1.34.0中我们可以单独为kubelet选择补丁版本配置透明度所有启动参数显式声明在systemd单元文件中排障时能快速定位问题。例如网络插件Calico的MTU设置可以直接在kubelet配置中调整架构理解深度通过手动配置证书、服务发现等环节能真正掌握Kubernetes各组件间的协作机制。你会清楚地知道kube-controller-manager如何通过6443端口与apiserver通信重要提示生产环境强烈建议使用二进制安装方式。虽然kubeadm简化了安装流程但遇到证书过期、配置变更等场景时二进制方式的手动维护优势就会凸显。2. 环境准备与组件规划2.1 硬件资源配置建议根据官方文档和实际生产经验建议的最低配置如下角色CPU内存磁盘网络Master节点4核8GB100GB千兆双网卡绑定Worker节点8核16GB200GB万兆网络特别提醒etcd集群需要低延迟存储建议使用SSD并单独挂载数据目录如果启用审计日志需要为/var/log/kubernetes预留50GB以上空间交换分区必须禁用否则kubelet会报错2.2 操作系统优化在CentOS 7/8或Ubuntu 20.04上需执行以下优化以root身份操作# 关闭防火墙后续由网络插件管理 systemctl stop firewalld systemctl disable firewalld # 禁用SELinux setenforce 0 sed -i s/^SELINUXenforcing/SELINUXdisabled/ /etc/selinux/config # 优化内核参数 cat /etc/sysctl.d/k8s.conf EOF net.ipv4.ip_forward 1 net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 vm.swappiness 0 EOF sysctl --system # 设置时区并同步时间 timedatectl set-timezone Asia/Shanghai yum install -y chrony || apt install -y chrony systemctl enable chronyd systemctl start chronyd2.3 证书体系设计二进制安装最复杂的环节之一是证书管理。我们需要为以下组件创建TLS证书CA根证书整个集群的信任锚点apiserver证书包含所有Master节点IP和Service网段kubelet客户端证书每个Node单独签发front-proxy证书用于聚合API使用cfssl工具生成证书的示例# 安装cfssl工具集 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson chmod x /usr/local/bin/cfssl* # 生成CA配置 cat ca-config.json EOF { signing: { default: { expiry: 87600h }, profiles: { kubernetes: { usages: [signing, key encipherment, server auth, client auth], expiry: 87600h } } } } EOF3. 核心组件安装与配置3.1 组件版本选择对于v1.34.0版本需要严格匹配的依赖组件版本组件推荐版本下载地址kube-apiserverv1.34.0https://dl.k8s.io/v1.34.0/kubernetes-server-linux-amd64.tar.gzetcdv3.5.10https://github.com/etcd-io/etcd/releases/download/v3.5.10/etcd-v3.5.10-linux-amd64.tar.gzCNI插件v1.3.0https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgzcontainerdv1.7.12https://github.com/containerd/containerd/releases/download/v1.7.12/containerd-1.7.12-linux-amd64.tar.gz安装目录结构建议/opt/kubernetes/ ├── bin # 二进制文件 ├── cfg # 配置文件 ├── ssl # 证书文件 └── logs # 组件日志3.2 etcd集群部署三节点etcd集群配置示例以node1为例# 解压安装 tar zxvf etcd-v3.5.10-linux-amd64.tar.gz mv etcd-v3.5.10-linux-amd64/etcd* /opt/kubernetes/bin/ # 创建systemd服务 cat /usr/lib/systemd/system/etcd.service EOF [Unit] DescriptionEtcd Server Afternetwork.target [Service] Typenotify ExecStart/opt/kubernetes/bin/etcd \\ --nameetcd01 \\ --data-dir/var/lib/etcd \\ --initial-advertise-peer-urlshttps://192.168.1.101:2380 \\ --listen-peer-urlshttps://192.168.1.101:2380 \\ --listen-client-urlshttps://192.168.1.101:2379,https://127.0.0.1:2379 \\ --advertise-client-urlshttps://192.168.1.101:2379 \\ --initial-clusteretcd01https://192.168.1.101:2380,etcd02https://192.168.1.102:2380,etcd03https://192.168.1.103:2380 \\ --client-cert-auth \\ --trusted-ca-file/opt/kubernetes/ssl/ca.pem \\ --cert-file/opt/kubernetes/ssl/etcd.pem \\ --key-file/opt/kubernetes/ssl/etcd-key.pem \\ --peer-client-cert-auth \\ --peer-trusted-ca-file/opt/kubernetes/ssl/ca.pem \\ --peer-cert-file/opt/kubernetes/ssl/etcd.pem \\ --peer-key-file/opt/kubernetes/ssl/etcd-key.pem Restarton-failure LimitNOFILE65536 [Install] WantedBymulti-user.target EOF关键参数说明--data-dir建议单独挂载高性能SSD--snapshot-count默认10万次操作触发快照生产环境可调大--heartbeat-interval节点间心跳间隔默认100ms--election-timeout选举超时时间默认1000ms启动后验证集群状态ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.101:2379 \ --cacert/opt/kubernetes/ssl/ca.pem \ --cert/opt/kubernetes/ssl/etcd.pem \ --key/opt/kubernetes/ssl/etcd-key.pem \ endpoint health3.3 控制平面组件配置kube-apiserver的systemd单元文件关键配置ExecStart/opt/kubernetes/bin/kube-apiserver \ --advertise-address192.168.1.100 \ --allow-privilegedtrue \ --authorization-modeNode,RBAC \ --client-ca-file/opt/kubernetes/ssl/ca.pem \ --enable-admission-pluginsNodeRestriction \ --enable-bootstrap-token-authtrue \ --etcd-cafile/opt/kubernetes/ssl/ca.pem \ --etcd-certfile/opt/kubernetes/ssl/etcd.pem \ --etcd-keyfile/opt/kubernetes/ssl/etcd-key.pem \ --etcd-servershttps://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \ --kubelet-client-certificate/opt/kubernetes/ssl/apiserver.pem \ --kubelet-client-key/opt/kubernetes/ssl/apiserver-key.pem \ --service-account-key-file/opt/kubernetes/ssl/sa.pub \ --service-account-signing-key-file/opt/kubernetes/ssl/sa.key \ --service-account-issuerhttps://kubernetes.default.svc.cluster.local \ --service-cluster-ip-range10.96.0.0/16 \ --tls-cert-file/opt/kubernetes/ssl/apiserver.pem \ --tls-private-key-file/opt/kubernetes/ssl/apiserver-key.pem \ --audit-log-path/var/log/kubernetes/audit.log \ --audit-log-maxage30 \ --audit-log-maxbackup3 \ --audit-log-maxsize100kube-controller-manager需要特别注意的启动参数--cluster-cidr10.244.0.0/16 \ # 必须与Pod网络插件匹配 --node-cidr-mask-size24 \ # 每个节点的Pod IP段大小 --allocate-node-cidrstrue \ # 开启节点CIDR分配 --kubeconfig/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \ --authentication-kubeconfig/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \ --authorization-kubeconfig/opt/kubernetes/cfg/kube-controller-manager.kubeconfig4. 工作节点部署要点4.1 容器运行时安装containerd的推荐配置/etc/containerd/config.tomlversion 2 [plugins.io.containerd.grpc.v1.cri] sandbox_image registry.k8s.io/pause:3.9 [plugins.io.containerd.grpc.v1.cri.containerd] snapshotter overlayfs [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc] runtime_type io.containerd.runc.v2 [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options] SystemdCgroup true关键配置说明SystemdCgrouptrue与kubelet的cgroup驱动保持一致sandbox_image建议提前拉取避免启动延迟snapshotter生产环境建议使用stargz等高级快照器4.2 kubelet服务配置kubelet的启动参数需要特别注意--container-runtimeremote \ --container-runtime-endpointunix:///run/containerd/containerd.sock \ --pod-infra-container-imageregistry.k8s.io/pause:3.9 \ --kubeconfig/opt/kubernetes/cfg/kubelet.kubeconfig \ --cert-dir/opt/kubernetes/ssl \ --network-plugincni \ --cni-conf-dir/etc/cni/net.d \ --cni-bin-dir/opt/cni/bin \ --cluster-dns10.96.0.10 \ --cluster-domaincluster.local \ --hairpin-modehairpin-veth \ --node-ip${NODE_IP} \ # 必须显式指定节点IP --rotate-certificatestrue \ # 开启证书自动轮换 --register-nodetrue常见问题如果节点状态显示NotReady通常检查以下位置/var/log/messages中的kubelet日志journalctl -u kubelet查看服务日志确认CNI插件二进制已正确安装到/opt/cni/bin5. 网络插件选型与部署5.1 Calico网络方案推荐使用Calico v3.26.1版本部署步骤下载清单文件curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml -O关键配置修改# 修改IP池配置 - name: CALICO_IPV4POOL_CIDR value: 10.244.0.0/16 # 必须与kube-controller-manager的--cluster-cidr一致 # 启用IPIP模式适合跨子网场景 - name: CALICO_IPV4POOL_IPIP value: Always # 修改MTU根据实际网络环境调整 - name: FELIX_IPINIPMTU value: 1440应用配置kubectl apply -f calico.yaml5.2 网络策略验证创建测试策略验证网络隔离apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: test-policy spec: podSelector: matchLabels: role: db policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 6379验证命令# 在frontend pod中测试连接 kubectl exec -it frontend-pod -- nc -zv db-pod 6379 # 在其他pod中测试应被拒绝 kubectl exec -it other-pod -- nc -zv db-pod 63796. 集群验证与排错指南6.1 核心健康检查执行以下命令验证集群状态# 检查节点状态 kubectl get nodes -o wide # 检查核心组件状态 kubectl get cs # 检查Pod网络连通性 kubectl run busybox --imagebusybox --restartNever -- sleep 3600 kubectl exec busybox -- ping 其他Pod IP # 检查DNS解析 kubectl exec busybox -- nslookup kubernetes.default6.2 常见问题排查问题1kube-apiserver无法连接etcd排查步骤检查etcd集群健康状态验证证书有效期openssl x509 -in /opt/kubernetes/ssl/etcd.pem -noout -dates检查防火墙规则iptables -L -n -v | grep 2379问题2Pod无法分配IP可能原因CNI插件未正确安装IP池耗尽查看Calico IP池kubectl get ippool -o yaml节点路由表异常检查route -n问题3镜像拉取失败解决方案配置镜像仓库认证kubectl create secret docker-registry regcred \ --docker-serverregistry.example.com \ --docker-usernameuser \ --docker-passwordpass在PodSpec中引用imagePullSecrets: - name: regcred7. 生产环境加固建议7.1 安全基线配置RBAC最小权限# 创建只读用户 kubectl create clusterrole read-only --verbget,list,watch --resource* kubectl create clusterrolebinding read-only-binding --clusterroleread-only --userread-only-userPod安全策略apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - projected - secret - downwardAPI - persistentVolumeClaim hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot seLinux: rule: RunAsAny supplementalGroups: rule: MustRunAs ranges: - min: 1 max: 65535 fsGroup: rule: MustRunAs ranges: - min: 1 max: 655357.2 监控与日志推荐部署方案Prometheus Operator监控集群指标Loki Grafana集中日志收集Falco运行时安全监控基础监控部署命令# 安装kube-prometheus-stack helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace8. 版本升级策略从v1.34.0升级到新版本的步骤前置检查kubectl get pods --all-namespaces -o wide kubectl drain node-name --ignore-daemonsets控制平面升级# 逐个Master节点升级 systemctl stop kube-apiserver kube-controller-manager kube-scheduler cp /opt/kubernetes/bin/kube{apiserver,controller-manager,scheduler} /opt/kubernetes/bin/backup/ wget https://dl.k8s.io/v1.35.0/bin/linux/amd64/kube-apiserver systemctl daemon-reload systemctl start kube-apiserver kube-controller-manager kube-schedulerWorker节点升级# 滚动升级kubelet kubectl cordon node-name systemctl stop kubelet mv /opt/kubernetes/bin/kubelet /opt/kubernetes/bin/backup/ wget https://dl.k8s.io/v1.35.0/bin/linux/amd64/kubelet systemctl daemon-reload systemctl start kubelet kubectl uncordon node-name升级后必做验证kubectl version --short确认组件版本kubectl get cs检查组件健康状态核心业务Pod功能测试9. 性能调优实战9.1 apiserver参数优化在高负载集群中建议调整--max-requests-inflight3000 \ # 默认400 --max-mutating-requests-inflight1000 \ # 默认200 --watch-cache-sizes100 \ # 关键资源watch缓存 --event-ttl168h \ # 延长事件保留时间 --encryption-provider-config/etc/kubernetes/encryption-config.yaml # 启用数据加密9.2 kubelet资源预留配置示例/var/lib/kubelet/config.yamlapiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration systemReserved: cpu: 1 memory: 1Gi ephemeral-storage: 5Gi kubeReserved: cpu: 500m memory: 2Gi ephemeral-storage: 1Gi evictionHard: memory.available: 500Mi nodefs.available: 10%9.3 调度器优化自定义调度策略示例apiVersion: kubescheduler.config.k8s.io/v1beta3 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler plugins: preScore: enabled: - name: InterPodAffinity - name: NodeResourcesFit score: enabled: - name: NodeResourcesBalancedAllocation weight: 2 - name: ImageLocality weight: 110. 备份与灾难恢复10.1 etcd数据备份定时备份方案# 每日全量备份 ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.101:2379 \ --cacert/opt/kubernetes/ssl/ca.pem \ --cert/opt/kubernetes/ssl/etcd.pem \ --key/opt/kubernetes/ssl/etcd-key.pem \ snapshot save /backup/etcd-$(date %Y%m%d).db # 备份保留策略 find /backup/ -name etcd-*.db -mtime 7 -exec rm -f {} \;10.2 关键资源备份使用velero实现应用级备份# 安装velero客户端 wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.0/velero-v1.11.0-linux-amd64.tar.gz tar -zxvf velero-v1.11.0-linux-amd64.tar.gz mv velero-v1.11.0-linux-amd64/velero /usr/local/bin/ # 配置S3存储备份 velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.7.0 \ --bucket my-k8s-backups \ --secret-file ./credentials-velero \ --use-volume-snapshotsfalse \ --backup-location-config regionminio,s3ForcePathStyletrue,s3Urlhttp://minio.example.com:900010.3 恢复演练etcd数据恢复步骤# 停止所有apiserver systemctl stop kube-apiserver # 恢复快照 ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.101:2379 \ --cacert/opt/kubernetes/ssl/ca.pem \ --cert/opt/kubernetes/ssl/etcd.pem \ --key/opt/kubernetes/ssl/etcd-key.pem \ snapshot restore /backup/etcd-20230801.db \ --data-dir /var/lib/etcd-restore # 修改etcd服务文件指向新数据目录 sed -i s|/var/lib/etcd|/var/lib/etcd-restore| /usr/lib/systemd/system/etcd.service systemctl daemon-reload systemctl start etcd # 验证数据 kubectl get ns