Skip to main content

K8S 常见问题

headless service​

  • 不需要 clusterIP 的 service
  • 不用于路由，而是用于 dns 查询 - 返回 endpoint 地址
spec:
  clusterIP: None

接口 CURL​

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -X GET https://kubernetes.default.svc/api
# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api

注入环境变量​

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
# command 和 args 会展开环境变量
command: ['/bin/echo']
args: ['$(POD_IP)']

limits​

  • 110 pods/node
  • 1 MB/object - etcd 限制 - #19781
    • 大型 CRD 定义可能达到这个限制 - embed 外部定义 - #82292
    • ConfigMap 如果过大，建议拆分
  • 所有 annotation KV 加起来不超过 256Kb - TotalAnnotationSizeLimitB
  • RFC-1123 - DNS LabelName 最长 63 字符 - K8S 中很多地方使用该验证
    • 所有支持 DNS 的对象
      • pod,service,namespace
    • label - #1830
  • gRPC 通常限制为 4MB

QoS​

status:
  qosClass: Guaranteed

operator vs controller​

  • operator 包含一个或多个 controller
  • controller 负责 CRD 处理

cni-plugins 不再包含 flannel​

资源限制​

K3S 常见问题​

  • #396 - Initializing eviction metric for zone

Pod 时区​

  1. TZ 环境变量
  • 要求 语言/runtime 支持
  • 需要 tzdata
  1. 映射 tz 信息
  • 可以与 host 保持一致
  • 可以不需要安装 tzdata
  • 要求 host 安装了 tzdata
  • 要求调整 Pod
  1. 代码内控制
  • bundle tzdata，运行时加载
  • golang import _ "time/tzdata"
    • 增加约 450 KB
apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  containers:
    - name: busybox
      image: busybox
      args:
        - sleep
        - '1000000'
      env:
        - name: TZ
          value: Asia/Shanghai
      volumeMounts:
        - name: tz-config
          mountPath: /etc/localtime
  volumes:
    - name: tz-config
      hostPath:
        path: /usr/share/zoneinfo/Asia/Shanghai
        type: File

taint​

  • node.kubernetes.io/disk-pressure:NoSchedule
  • 为 pod 设置 priorityClassName 确保调度，避免驱逐
    • system-node-critical,system-cluster-critical
df -h
df -Hi

# remove DiskPressure
kubectl taint nodes < nodename > node.kubernetes.io/disk-pressure-

集群限制​

集群 Master 性能推荐

NodesGCE TypeSpec
1-5n1-standard-11 核 4G
6-10n1-standard-22 核 8G
11-100n1-standard-44 核 16G
101-250n1-standard-88 核 32G
251-500n1-standard-1616 核 64G
500+n1-standard-3232 核 128G

操作系统要求 / Linux Kernel Requirement​

./check-config.sh kernel.config

invalid capacity 0 on image filesystem​

  • 确保存在内核参数 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
    • syslinux
      • /etc/update-extlinux.conf
      • update-extlinux && reboot
    • grub
      • /etc/default/grub
cat /proc/cmdline | grep cgroup_enable

导出资源忽略状态字段​

  • 常见字段
    • status
    • metadata.managedFields
    • metadata
      • annotations
        • kubectl.kubernetes.io/last-applied-configuration
        • deployment.kubernetes.io/revision
      • resourceVersion
      • selfLink
      • uid
      • creationTimestamp
      • generation
kubectl get -o=yaml deploy whoami | yq d - status | yq d - 'metadata.managedFields'

删除卡在 Terminating 状态的资源​

  • 删除 Pod 可尝试强制
  • 资源删不掉尝试移除 finalizers。
# 强制删除 Pod
kubectl delete pods --force < pod > --grace-period=0
# 尝试移除 Pod 的 finalizers
kubectl patch pod '{"metadata":{"finalizers":null}}' < pod > -p

# 查看所有的 Terminating Namespace
kubectl get ns --field-selector status.phase=Terminating

# 所有 NS
kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{..metadata.name}'

# 批量
# kubectl patch ns -p '{"metadata":{"finalizers": null}}' $NS
# kubectl patch ns -p '{"metadata":{"finalizers":[]}}' --type=merge $NS
kubectl patch ns -p '{"metadata":{"finalizers":[]}}' --type=merge $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{..metadata.name}')

删除 rancher 项目空间​

  • 主要难点在于 get all 不会返回所有资源
  • 部分资源需要先 patch 才能删除
for ns in local p-66lfd; do
  for error in app.project.cattle.io/cluster-alerting app.project.cattle.io/cluster-monitoring app.project.cattle.io/monitoring-operator app.project.cattle.io/project-monitoring clusteralertgroup.management.cattle.io/cluster-scan-alert clusteralertgroup.management.cattle.io/etcd-alert clusteralertgroup.management.cattle.io/event-alert clusteralertgroup.management.cattle.io/kube-components-alert clusteralertgroup.management.cattle.io/node-alert clusterroletemplatebinding.management.cattle.io/creator-cluster-owner clusterroletemplatebinding.management.cattle.io/u-b4qkhsnliz-admin node.management.cattle.io/machine-9sssc node.management.cattle.io/machine-ks6z6 node.management.cattle.io/machine-v4v89 project.management.cattle.io/p-cnj28 project.management.cattle.io/p-mbvfd projectalertgroup.management.cattle.io/projectalert-workload-alert projectalertrule.management.cattle.io/less-than-half-workload-available projectalertrule.management.cattle.io/memory-close-to-resource-limited projectroletemplatebinding.management.cattle.io/app-jdnmz projectroletemplatebinding.management.cattle.io/creator-project-owner projectroletemplatebinding.management.cattle.io/prtb-s6fhc projectroletemplatebinding.management.cattle.io/u-2gacgc4nfu-member projectroletemplatebinding.management.cattle.io/u-efxo6n6ndd-member; do
    for resource in $(kubectl get -n $ns $error -o name); do
      kubectl patch -n $ns $resource -p '{"metadata": {"finalizers": []}}' --type='merge'
    done
  done
done

# 全局资源
for res in $(kubectl api-resources --namespaced=false --api-group management.cattle.io | cut -d ' ' -f 1); do
  echo "=== $res.management.cattle.io ==="
  kubectl get $res.management.cattle.io
done

# namespaced
groups="management.cattle.io catalog.cattle.io project.cattle.io"
for grp in $groups; do
  for res in $(kubectl api-resources --namespaced=true --api-group $grp -o name); do
    echo "=== $res ==="
    kubectl get --all-namespaces $res
  done
done

# 清除资源
cleargroup() {
  kubectl patch $1 -p '{"metadata":{"finalizers":[]}}' --type=merge $(kubectl get $1 -o jsonpath='{..metadata.name}')
  kubectl delete --all $1
}

cleargroup globalroles.management.cattle.io

Unable to connect to the server: x509: certificate is valid for 10.10.1.2, 10.43.0.1, 127.0.0.1, 192.168.1.10, not 123.123.123.123​

签发的证书不包含 IP，初始化的时候可以添加，建议新建子账号管理。

# 原先的位于
# /etc/kubernetes/pki/apiserver.*
kubeadm init --apiserver-cert-extra-sans=123.123.123.123

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "cron-1594372200-lmkcb": operation timeout: context deadline exceeded​

测试是否正常

apiVersion: batch/v1
kind: Job
metadata:
  name: alpine-pwd
spec:
  template:
    spec:
      containers:
        - name: alpine
          image: wener/base
          command: ['pwd']
      restartPolicy: Never
  backoffLimit: 4

一个主节点异常，重启解决。

error: specifying a root certificates file with the insecure flag is not allowed​

insecure-skip-tls-verify 不能用于根证书，新建子账号。

node_lifecycle_controller.go:1433 Initializing eviction metric for zone​

running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaims​

error: unable to retrieve the complete list of server APIs: write: broken pipe​

  • 网络不稳定也可能导致
    • 例如 在使用 sshuttle 的时候偶尔就会出现
    • 可以考虑使用 ssh 转发 - ssh -vNL 6443:10.10.1.1:6443 [email protected] -o ExitOnForwardFailure=yes
  • an error on the server ("") has prevented the request from succeeding
# 概率失败
kubectl api-resources

# 确保 metric 正常
kubectl get apiservices
# 确保 系统服务 正常
kubectl get pods -n kube-system

dns 不通​

  • 现象 - 所有 服务 504，网关超时
  • 排查
    • 验证 kube-dns 53 能否解析
    • 所有节点都有问题还是单个节点
      • 所有节点都有问题则可能是服务的问题
      • 单个节点则可能是环境问题
    • ping 后端 endpoint
    • ping 不通则说明可能是 flannel 插件之类异常或者使用的底层 interface 异常
  • 解决
    • 尝试重启 k3s
    • 尝试重启 网络
    • 尝试重启 系统
# 验证 DNS 能否解析
# k3s 默认使用 10.43.0.10
# kube-dns.kube-system.svc
nslookup wener.me 10.43.0.10
# 或
dig @10.43.0.10 wener.me

MountVolume.SetUp failed for volume "config-volume" : failed to sync secret cache: timed out waiting for the condition​

  • 条件未满足，无法继续执行，且等待超时
  • 解决办法
    • 等待 或 删除 Pod

查看条件

status:
  phase: Pending
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-09-04T10:09:51Z'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2020-09-04T10:09:51Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [alertmanager config-reloader]'
    # 这一步未能成功
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2020-09-04T10:09:51Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [alertmanager config-reloader]'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-09-04T10:09:51Z'

kubernetes swap​

不建议使用 swap

didn't have free ports for the requested pod ports​

deploy 如果只有一个节点，使用 Recreate， RollingUpdate 会失败。

scale to zero​

使用 keda 或者 knative

  • keda 简单 - 负责 autoscaler
  • knative 复杂 - 整套开发逻辑

The StorageClass is invalid: provisioner: Forbidden: updates to provisioner are forbidden.​

provisioner 名字不匹配，还 sc 名字或删了重建。

预留资源​

--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]
--system-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]
--reserved-cpus=0-3
# 开始驱逐的阀值
--eviction-hard=[memory.available<500Mi]
--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]

# 指向现有的 cgroup
--kube-reserved-cgroup=
--system-reserved-cgroup=

node(s) had volume node affinity conflict​

  • 如果是 pv 无法迁移，切影响不大，可以考虑删除了从建

可能 pvc 和 pod zone 冲突，可以针对 zone 创建 sc

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: region1storageclass
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: failure-domain.beta.kubernetes.io/zone
        values:
          - eu-west-2b

如何保持负载均衡​

  1. 配置反亲和
  2. kill pod 触发重新调度
  3. 尝试 rollout 从新调度
  4. 使用类似 kubernetes-sigs/descheduler 这样的工具自动维护均衡

pause 是什么进程​

1 node(s) had volume node affinity conflict​

  • 检查 StorageClass 的亲和策略
  • 检查 PersistentVolume 的亲和策略
    • Pod 需要匹配 PV 的亲和策略

failed to run Kubelet: mountpoint for cpu not found​

应该是没有挂载 cgroups

service cgroups start

orphaned pod found, but error not a directory occurred when trying to remove the volumes dir​

pods=$(tail -n 1000 /var/log/k0s.log | grep 'orphaned pod' | sed -r 's/.*pod[^"]+"([^\\"]+).*/\1/' | uniq)
for i in $pods; do
  echo "Deleting Orphaned pod id: $i"
  rm -rf /var/lib/kubelet/pods/$i
done

1 node(s) didn't have free ports for the requested pod ports.​

  • 检查 hostPort

资源​

  • cpu
    • 1.0 -> 1vcpu -> 1000m
  • memory
    • base 1000 - E,T,P,G,M,k
    • 2^n - Ei,Pi,Ti,Gi,Mi,Ki
  • hugepages-<size>
  • ephemeral-storage
  • storage

failed to reserve container name "": name "" is reserved for​

# reserved containerd
ctr -n k8s.io c ls

unable to find data in memory cache​

parent snapshot does not exist: not found​

invalid bearer token, Token has been invalidated​

Node Shell​

可通过一个 Pod 进入

apiVersion: v1
kind: Pod
metadata:
  name: node-shell
  namespace: kube-system
spec:
  containers:
    - name: shell
      image: docker.io/alpine:3.13
      command:
        - nsenter
      args:
        - '-t'
        - '1'
        - '-m'
        - '-u'
        - '-i'
        - '-n'
        - sleep
        - '14000'
      resources: {}
      volumeMounts:
        - name: kube-api-access
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
  volumes:
    - name: kube-api-access
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  # 希望进入的 Node
  nodeName: master-1
  hostNetwork: true
  hostPID: true
  hostIPC: true
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - operator: Exists
  priorityClassName: system-node-critical
  priority: 2000001000
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

证书轮换​

eStargz​

  • k3s/k0s 使用 containerd 的能力支持 stargz
k3s server --snapshotter=stargz
/etc/containerd/config.toml
# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
# The specified address below is the default which stargz snapshotter listen to.
[proxy_plugins]

[proxy_plugins.stargz]
type = "snapshot"
address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

# Use stargz snapshotter through CRI
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "stargz"

could not load server certificate file : Permission denied​

Usage of EmptyDir volume exceeds the limit 1m​

  • emptyDir 使用 tmpfs 挂载
    • medium: Memory

常用 fieldRef​

  • fieldRef
  • resourceFieldRef
env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: POD_SERVICE_ACCOUNT
    valueFrom:
      fieldRef:
        fieldPath: spec.serviceAccountName

镜像地址​

  • k8s.gcr.io - 旧的
  • registry.k8s.io - 新的

failed to execute portforward in network namespace​

高可用和复杂均衡​

  • HA
    • 集群外 Kubelet API 复杂均衡
    • 用外部的 HAProxy/Nginx/VIP/Tinc 实现
    • 最终的目的是用一个不变的地址访问 Kubelet API
  • LLB
    • 集群内 LoadBalance 服务
    • 分配 ClutserIP
    • 不使用云服务可以用 MetalLB 这类的组件+Overlay/LAN 实现 ClusterIP 能力
      • 例如: flannel+host+tinc switch
    • 使用云服务可以用云服务的 LB 组件 - 可直接提供外网 IP