Skip to main content

K8S 常见问题

headless service

  • 不需要 clusterIP 的 service
  • 不用于路由,而是用于 dns 查询 - 返回 endpoint 地址
spec:
clusterIP: None

operator vs controller

  • operator 包含一个或多个 controller
  • controller 负责 CRD 处理

cni-plugins 不再包含 flannel

资源限制

K3S 常见问题

  • #396 - Initializing eviction metric for zone

Pod 时区

  1. TZ 环境变量
  • 要求 语言/runtime 支持
  • 需要 tzdata
  1. 映射 tz 信息
  • 可以与 host 保持一致
  • 可以不需要安装 tzdata
  • 要求 host 安装了 tzdata
  • 要求调整 Pod
  1. 代码内控制
  • bundle tzdata,运行时加载
  • golang import _ "time/tzdata"
    • 增加约 450 KB
apiVersion: v1
kind: Pod
metadata:
name: busybox-sleep
spec:
containers:
- name: busybox
image: busybox
args:
- sleep
- '1000000'
env:
- name: TZ
value: Asia/Shanghai
volumeMounts:
- name: tz-config
mountPath: /etc/localtime
volumes:
- name: tz-config
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
type: File

taint

  • node.kubernetes.io/disk-pressure:NoSchedule
  • 为 pod 设置 priorityClassName 确保调度,避免驱逐
    • system-node-critical,system-cluster-critical
df -h
df -Hi

# remove DiskPressure
kubectl taint nodes <nodename> node.kubernetes.io/disk-pressure-

集群限制

集群 Master 性能推荐

NodesGCE TypeSpec
1-5n1-standard-11 核 4G
6-10n1-standard-22 核 8G
11-100n1-standard-44 核 16G
101-250n1-standard-88 核 32G
251-500n1-standard-1616 核 64G
500+n1-standard-3232 核 128G

操作系统要求 / Linux Kernel Requirement

./check-config.sh kernel.config

invalid capacity 0 on image filesystem

  • 确保存在内核参数 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
    • syslinux
      • /etc/update-extlinux.conf
      • update-extlinux && reboot
    • grub
      • /etc/default/grub
cat /proc/cmdline | grep cgroup_enable

导出资源忽略状态字段

  • 常见字段
    • status
    • metadata.managedFields
    • metadata
      • annotations
        • kubectl.kubernetes.io/last-applied-configuration
        • deployment.kubernetes.io/revision
      • resourceVersion
      • selfLink
      • uid
      • creationTimestamp
      • generation
kubectl get -o=yaml deploy whoami | yq d - status | yq d - 'metadata.managedFields'

删除卡在 Terminating 状态的资源

  • 删除 Pod 可尝试强制
  • 资源删不掉尝试移除 finalizers。
# 强制删除 Pod
kubectl delete pods <pod> --grace-period=0 --force
# 尝试移除 Pod 的 finalizers
kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}'

# 查看所有的 Terminating Namespace
kubectl get ns --field-selector status.phase=Terminating

# 所有 NS
kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{..metadata.name}'

# 批量
# kubectl patch ns -p '{"metadata":{"finalizers": null}}' $NS
# kubectl patch ns -p '{"metadata":{"finalizers":[]}}' --type=merge $NS
kubectl patch ns -p '{"metadata":{"finalizers":[]}}' --type=merge $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{..metadata.name}')

删除 rancher 项目空间

  • 主要难点在于 get all 不会返回所有资源
  • 部分资源需要先 patch 才能删除
for ns in local p-66lfd ; do
for error in app.project.cattle.io/cluster-alerting app.project.cattle.io/cluster-monitoring app.project.cattle.io/monitoring-operator app.project.cattle.io/project-monitoring clusteralertgroup.management.cattle.io/cluster-scan-alert clusteralertgroup.management.cattle.io/etcd-alert clusteralertgroup.management.cattle.io/event-alert clusteralertgroup.management.cattle.io/kube-components-alert clusteralertgroup.management.cattle.io/node-alert clusterroletemplatebinding.management.cattle.io/creator-cluster-owner clusterroletemplatebinding.management.cattle.io/u-b4qkhsnliz-admin node.management.cattle.io/machine-9sssc node.management.cattle.io/machine-ks6z6 node.management.cattle.io/machine-v4v89 project.management.cattle.io/p-cnj28 project.management.cattle.io/p-mbvfd projectalertgroup.management.cattle.io/projectalert-workload-alert projectalertrule.management.cattle.io/less-than-half-workload-available projectalertrule.management.cattle.io/memory-close-to-resource-limited projectroletemplatebinding.management.cattle.io/app-jdnmz projectroletemplatebinding.management.cattle.io/creator-project-owner projectroletemplatebinding.management.cattle.io/prtb-s6fhc projectroletemplatebinding.management.cattle.io/u-2gacgc4nfu-member projectroletemplatebinding.management.cattle.io/u-efxo6n6ndd-member ; do
for resource in `kubectl get -n $ns $error -o name` ; do
kubectl patch -n $ns $resource -p '{"metadata": {"finalizers": []}}' --type='merge'
done
done
done

# 全局资源
for res in $(kubectl api-resources --namespaced=false --api-group management.cattle.io | cut -d ' ' -f 1); do
echo "=== $res.management.cattle.io ==="
kubectl get $res.management.cattle.io
done


# namespaced
groups="management.cattle.io catalog.cattle.io project.cattle.io"
for grp in $groups; do
for res in $(kubectl api-resources --namespaced=true --api-group $grp -o name); do
echo "=== $res ==="
kubectl get --all-namespaces $res
done
done


# 清除资源
cleargroup(){
kubectl patch $1 -p '{"metadata":{"finalizers":[]}}' --type=merge $(kubectl get $1 -o jsonpath='{..metadata.name}')
kubectl delete --all $1
}

cleargroup globalroles.management.cattle.io

Unable to connect to the server: x509: certificate is valid for 10.10.1.2, 10.43.0.1, 127.0.0.1, 192.168.1.10, not 123.123.123.123

签发的证书不包含 IP,初始化的时候可以添加,建议新建子账号管理。

# 原先的位于
# /etc/kubernetes/pki/apiserver.*
kubeadm init --apiserver-cert-extra-sans=123.123.123.123

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "cron-1594372200-lmkcb": operation timeout: context deadline exceeded

测试是否正常

apiVersion: batch/v1
kind: Job
metadata:
name: alpine-pwd
spec:
template:
spec:
containers:
- name: alpine
image: wener/base
command: ['pwd']
restartPolicy: Never
backoffLimit: 4

一个主节点异常,重启解决。

error: specifying a root certificates file with the insecure flag is not allowed

insecure-skip-tls-verify 不能用于根证书,新建子账号。

node_lifecycle_controller.go:1433 Initializing eviction metric for zone

running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaims

error: unable to retrieve the complete list of server APIs: write: broken pipe

  • 网络不稳定也可能导致
    • 例如 在使用 sshuttle 的时候偶尔就会出现
    • 可以考虑使用 ssh 转发 - ssh -vNL 6443:10.10.1.1:6443 [email protected] -o ExitOnForwardFailure=yes
  • an error on the server ("") has prevented the request from succeeding
# 概率失败
kubectl api-resources

# 确保 metric 正常
kubectl get apiservices
# 确保 系统服务 正常
kubectl get pods -n kube-system

dns 不通

  • 现象 - 所有 服务 504,网关超时
  • 排查
    • 验证 kube-dns 53 能否解析
    • 所有节点都有问题还是单个节点
      • 所有节点都有问题则可能是服务的问题
      • 单个节点则可能是环境问题
    • ping 后端 endpoint
    • ping 不通则说明可能是 flannel 插件之类异常或者使用的底层 interface 异常
  • 解决
    • 尝试重启 k3s
    • 尝试重启 网络
    • 尝试重启 系统
# 验证 DNS 能否解析
# k3s 默认使用 10.43.0.10
# kube-dns.kube-system.svc
nslookup wener.me 10.43.0.10
# 或
dig @10.43.0.10 wener.me

MountVolume.SetUp failed for volume "config-volume" : failed to sync secret cache: timed out waiting for the condition

  • 条件未满足,无法继续执行,且等待超时
  • 解决办法
    • 等待 或 删除 Pod

查看条件

status:
phase: Pending
conditions:
- type: Initialized
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-09-04T10:09:51Z'
- type: Ready
status: 'False'
lastProbeTime: null
lastTransitionTime: '2020-09-04T10:09:51Z'
reason: ContainersNotReady
message: 'containers with unready status: [alertmanager config-reloader]'
# 这一步未能成功
- type: ContainersReady
status: 'False'
lastProbeTime: null
lastTransitionTime: '2020-09-04T10:09:51Z'
reason: ContainersNotReady
message: 'containers with unready status: [alertmanager config-reloader]'
- type: PodScheduled
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-09-04T10:09:51Z'

kubernetes swap

不建议使用 swap

didn't have free ports for the requested pod ports

deploy 如果只有一个节点,使用 Recreate, RollingUpdate 会失败。

scale to zero

使用 keda 或者 knative

  • keda 简单 - 负责 autoscaler
  • knative 复杂 - 整套开发逻辑

The StorageClass is invalid: provisioner: Forbidden: updates to provisioner are forbidden.

provisioner 名字不匹配,还 sc 名字或删了重建。

预留资源

--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]
--system-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]
--reserved-cpus=0-3
# 开始驱逐的阀值
--eviction-hard=[memory.available<500Mi]
--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]

# 指向现有的 cgroup
--kube-reserved-cgroup=
--system-reserved-cgroup=

node(s) had volume node affinity conflict

  • 如果是 pv 无法迁移,切影响不大,可以考虑删除了从建

可能 pvc 和 pod zone 冲突,可以针对 zone 创建 sc

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: region1storageclass
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
- key: failure-domain.beta.kubernetes.io/zone
values:
- eu-west-2b

如何保持负载均衡

  1. 配置反亲和
  2. kill pod 触发重新调度
  3. 尝试 rollout 从新调度
  4. 使用类似 kubernetes-sigs/descheduler 这样的工具自动维护均衡

pause 是什么进程

1 node(s) had volume node affinity conflict

  • 检查 StorageClass 的亲和策略
  • 检查 PersistentVolume 的亲和策略
    • Pod 需要匹配 PV 的亲和策略

failed to run Kubelet: mountpoint for cpu not found

应该是没有挂载 cgroups

service cgroups start

orphaned pod found, but error not a directory occurred when trying to remove the volumes dir

pods=$(tail -n 1000 /var/log/k0s.log|grep 'orphaned pod'|sed -r 's/.*pod[^"]+"([^\\"]+).*/\1/'|uniq)
for i in $pods
do
echo "Deleting Orphaned pod id: $i";
rm -rf /var/lib/kubelet/pods/$i;
done

1 node(s) didn't have free ports for the requested pod ports.

  • 检查 hostPort

资源

  • cpu
    • 1.0 -> 1vcpu -> 1000m
  • memory
    • base 1000 - E,T,P,G,M,k
    • 2^n - Ei,Pi,Ti,Gi,Mi,Ki
  • hugepages-<size>
  • ephemeral-storage
  • storage

failed to reserve container name "": name "" is reserved for

parent snapshot does not exist: not found