跳到主要内容

Kubernetes 版本

versionrelease
Kubernetes 1.292023-12-13
Kubernetes 1.282023-08-15
Kubernetes 1.272023-04-11
Kubernetes 1.252022-08-23
Kubernetes 1.242022-05-03
Kubernetes 1.232021-12-07
Kubernetes 1.222021-08-04
Kubernetes 1.212021-04-08
Kubernetes 1.202020-12-08
Kubernetes 1.19
APIfromto
networking.k8s.io/v1beta1v1.22
networking.k8s.io/v1v1.19

特性分组说明

feature groupsdesc
api-machinery接口
apps执行
auth授权
cli命令行 - kubectl
cluster-lifecycle集群生命周期
cluster-provider集群供应商
instrumentationkubelet 自身相关
network网络
node节点
scheduling调度 - Pod
storage存储
windowsWindows 相关
特性阶段默认
  • Stable 一般默认开启,
  • Beta 可能开启
  • Alpha 一般为关闭
版本关注点
  • breaking-change
    • 移除 depracated - 例如 Ingress 升级过程
  • 默认启用的特性 - Stable, Beta
  • distro 开启的特性 - K3S, K0S
  • 对使用影响大的特性

Kubernetes 1.23​

  • Alpha
    • nftables backend for kube-proxy

Kubernetes 1.28​

  • CP 组件版本 <-> 节点组件版本 从 n-2 -> n-3
    • CP 可升级更快
    • CP 组件 - kube-apiserver, kube-scheduler, kube-controller-manager, cloud-controller-manager
    • 节点组件 - kubelet, kube-proxy
  • Alpha
    • init container restartPolicy

Kubernetes 1.27​

  • k8s.gcr.io -> registry.k8s.io
  • Beta
    • ReadWriteOncePod for PV & PVC

Kubernetes 1.26​

  • Stable
    • KubeletCredentialProviders
    • CPUManager
    • ServiceInternalTrafficPolicy
      • Service.spec.trafficPolicy=Cluster - Topology,PreferLocal,Local
    • MixedProtocolLBService
  • Beta
    • NodeOutOfServiceVolumeDetach
      • Non-graceful node shutdown
    • ProxyTerminatingEndpoints
      • 避免滚动升级时流量丢失
  • Alpha
    • MinimizeIPTablesRestore=false
      • iptables-restore 只传递修改内容
  • https://sysdig.com/blog/kubernetes-1-26-whats-new/

Kubernetes 1.25​

  • PodSecurityPolicy
  • Stable
    • Pod Security Admission
    • Ephemeral Containers
    • cgroups v2
  • k8s.gcr.io -> registry.k8s.io
  • KMS v2

Kubernetes 1.24​

groupstagefeaturedefaultalphabeta
appsAlphaCronJobTimeZone
appsStableIndexedJob
appsStableJobReadyPods
authStableCSRDuration
cloud-providerStableServiceLoadBalancerClass
networkAlphaNetworkPolicyStatus
networkAlphaServiceIPStaticSubrange
networkBetaMixedProtocolLBService
networkBetaServiceInternalTrafficPolicy
nodeBetaDisableKubeletCloudCredentialProviders
nodeBetaGracefulNodeShutdownBasedOnPodPriority
nodeBetaGRPCContainerProbe
nodeStablePodOverhead
schedulingStableNonPreemptingPriority
schedulingStablePodAffinityNamespaceSelector
storageAlphaCSIVolumeHealth
storageAlphaNonGracefulFailover
storageBetaAnyVolumeDataSource
storageBetaHonorPVReclaimPolicy
storageStableCSIStorageCapacity
storageStableExpandCSIVolumes1.81.11
storageStableExpandInUsePersistentVolumes1.81.11
storageStableExpandPersistentVolumes1.81.11
windowsBetaIdentifyPodOS

Kubernetes 1.23​

提示
  • Kubernetes 会逐步把 CSI 移出核心
stagegroupfeaturedefaultalphabetastable
Alphaapi-machineryCustomResourceValidationExpressions
Alphaapi-machineryOpenAPIEnum
Alphaapi-machineryOpenApiv3
Alphaapi-machineryServerSideFieldValidation
AlphaappsStatefulSetAutoDeletePVC
Alphacluster-lifecycleUnversionedKubeletConfigMap
AlphanodeCPUManagerPolicyExperimentalOptions
AlphanodeGracefulNodeShutdownBasedOnPodPriority
AlphanodeGRPCContainerProbe
AlphanodePodAndContainerStatsFromCRI
AlphastorageCSIMigrationPortworx
AlphastorageCSIMigrationRBD
AlphastorageDelegateFSGroupToCSIDriver
AlphastorageHonorPVReclaimPolicy
AlphastorageRecoverVolumeExpansionFailure
AlphawindowsIdentifyPodOS
AlphawindowsWindowsHostProcessContainerstrue
BetaauthPodSecuritytrue
BetanodeCPUManagerPolicyOptionsfalse
BetanodeEphemeralContainerstrue
BetanodeKubeletPodResourcesGetAllocatabletrue
BetaschedulingJobMutableNodeSchedulingDirectivestrue
StableappsJobReadyPods
StableappsJobTrackingWithFinalizers
StableappsStatefulSetMinReadySeconds
StableappsTTLAfterFinished
StablenetworkIngressClassNamespacedParams
StablenetworkIPv6DualStack
StablenetworkTopologyAwareHints
StablestorageConfigurableFSGroupPolicy
StablestorageCSIMigrationAWSfalse
StablestorageCSIVolumeFSGroupPolicy
StablestorageGenericEphemeralVolume

Kubernetes 1.22​

警告
  • ingress 移除了 networking.k8s.io/v1beta1 , 建议等下一个大版本再升级
    • 因为很多周边配套服务还没有升级使用 networking.k8s.io/v1
  • 新的 PodSecurity admission 控制器 作为 PSP 替代
  • Rootless 模式容器
    • KubeletInUserNamespace
  • Seccomp
    • SeccompDefault - 支持默认开启
  • 支持 swap
    • NodeMemorySwap
    • --fail-on-swap
    • kubelet MemorySwap.SwapBehavior=UnlimitedSwap
  • cgroupsv2
    • Memory QoS - min,max,low,high
  • 稳定 API
    • Server-side Apply - kubectl apply 功能迁移到服务端 - 1.14+
    • 废弃接口告警机制 - 1.19+
    • namespace 不可变 label - kubernetes.io/metadata.name - 1.21+
    • CronJobs - 1.4+
    • PodDisruptionBudget - 1.4+
    • EndpointSlice - 解决 pod 多 IP 问题
      • 1.22 会为超过 1000 的 Endpoints 添加 endpoints.kubernetes.io/over-capacity: truncated
        • 之前是 endpoints.kubernetes.io/over-capacity: warning
    • AppProtocol - Services 和 Endpoints 自定义协议
    • HugePageStorageMediumSize - 1.18+
    • Pod hostnameFQDN: true - hotsname 设置为 FQDN - 1.19+
    • CSIServiceAccountToken
    • Windows CSI 插件 - 1.16+
  • Beta API
    • APIPriorityAndFairness
    • Job .spec.suspend
    • CRS 有效期 - CertificateSigningRequestSpec.ExpirationSeconds
    • Service LoadBalancer Class - 支持多 LB
    • .spec.egress.ports.endPort - 配置端口段
    • 临时容器 - kubectl debug
    • emptyDir.sizeLimit - 限制 tmpfs 内存使用
    • 存储 volume DataSource - 支持预先包含数据
  • Alpha
    • MaxDNSSearchPathsExpanded - 扩展 DNS 长度
      • MaxDNSSearchPaths 6 -> 32
      • MaxDNSSearchListChars 256 -> 2048
    • 存储 ReadWriteOncePod - 单几点 RWM，只允许一个节点
  • 移除废弃 Beta API
    • Ingress
    • CustomResourceDefinition
    • ValidatingWebhookConfiguration
    • MutatingWebhookConfiguration
    • CertificateSigningRequest
  • 功能废弃
    • StreamingProxyRedirects
    • ServiceTopology - 使用 Topology Aware Hints / 1.17+
    • DynamicKubeletConfig
  • 参考
stagefeaturedefaultalphabetastable
AlphaAPIServerTracing1.22
AlphaCPUManagerPolicyOptions1.22
AlphaDelegateFSGroupToCSIDriver1.22
AlphaDisableCloudProviders1.22
AlphaExpandedDNSConfig1.22
AlphaJobTrackingWithFinalizers1.22
AlphaKubeletInUserNamespace1.22
AlphaMemoryQoS1.22
AlphaNodeSwap1.22
AlphaPodSecurity1.22
AlphaProxyTerminatingEndpoints1.22
AlphaReadWriteOncePod1.22
AlphaSeccompDefault1.22
AlphaStatefulSetMinReadySeconds1.22
AlphaWindowsHostProcessContainers1.22
AlphaWindowsHostProcessContainers1.22
BetaControllerManagerLeaderMigrationtrue1.211.22
BetaCSRDurationtrue1.22
BetaDaemonSetUpdateSurgetrue1.211.22
BetaEndpointSliceTerminatingConditiontrue1.201.22
BetaIndexedJobtrue1.211.22
BetaIngressClassNamespacedParamstrue1.211.22
BetaLogarithmicScaleDowntrue1.211.22
BetaMemoryManagertrue1.211.22
BetaNetworkPolicyEndPorttrue1.211.22
BetaPodAffinityNamespaceSelectortrue1.211.22
BetaPodDeletionCosttrue1.211.22
BetaPreferNominatedNodetrue1.211.22
BetaProbeTerminationGracePeriodfalse1.211.22
BetaServiceInternalTrafficPolicytrue1.211.22
BetaServiceLBNodePortControltrue1.201.22
BetaServiceLoadBalancerClasstrue1.211.22
BetaSizeMemoryBackedVolumestrue1.201.22
BetaSuspendJobtrue1.211.22
DeprecatedBalanceAttachedNodeVolumesfalse1.11
DeprecatedCSIMigrationvSphereComplete1.19
DeprecatedDynamicKubeletConfigfalse1.41.11
DeprecatedStreamingProxyRedirectsfalse1.5
DeprecatedValidateProxyRedirectstrue1.121.14
StableBoundServiceAccountTokenVolume1.131.211.22
StableCronJobControllerV21.201.211.22
StableCSIServiceAccountToken1.201.211.22
StableEndpointSliceProxying1.181.191.22
StableHugePageStorageMediumSize1.181.191.22
StableHugePageStorageMediumSize1.181.191.22
StableNamespaceDefaultLabelName1.211.22
StableServerSideApply1.141.161.22
StableSetHostnameAsFQDN1.191.201.22
StableWarningHeaders1.191.22
StableWindowsEndpointSliceProxying1.191.211.22
# 当前使用的 ingress 版本
kubectl get ing --all-namespaces -o $'go-template={{range $k,$v := .items}}{{.apiVersion}}\n{{end}}' | sort -n -u
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      defaults: # Defaults applied when a mode label is not set.
        enforce: <default enforce policy level>
        enforce-version: <default enforce policy version>
        audit: <default audit policy level>
        audit-version: <default audit policy version>
        warn: <default warn policy level>
        warn-version: <default warn policy version>
      exemptions:
        usernames: [<array of authenticated usernames to exempt>]
        runtimeClassNames: [<array of runtime class names to exempt>]
        namespaces: [<array of namespaces to exempt>]

Kubernetes 1.21​

  • 13 stable, 16 beta, 20 alpha, 2 废弃
  • Kustomize 从 v2.0.3 升级到 v4.0.5 - 之前因为依赖问题被 block
  • kubelet 支持结构化日志
  • TokenRequest, TokenRequestProjection - 默认开启
  • Stable/GA
    • CronJobs - 1.8 ScheduledJobs beta
    • 不可变的 Secrets, ConfigMaps
      • kubelet 不 watch 变化，减少 apiserver 负载
    • ServiceNodeExclusion, NodeDisruptionExclusion, LegacyNodeRoleBehavior
    • KEP-85 policy/v1/PodDisruptionBudget - 1.5 beta
    • KEP-23 支持 sysctl - 自 1.4
  • Beta
    • IPv4/IPv6 dual-stack
    • CSI Service Account Token
    • Generic ephemeral volumes - 1.19 alpha
    • Storage Capacity - 1.19 alpha
      • 之前调度不会关心存储空间问题
      • 用于后续支持基于节点存储空间调度功能
    • Graceful node shutdown
      • 1.21 默认开启
        • ShutdownGracePeriod, ShutdownGracePeriodCriticalPods 默认为 0
        • 设置为非 0 激活节点优雅关机
      • 依赖 systemd
    • KEP-592 Job ttlSecondsAfterFinished
      • Job 完成后，在 TTL 时间过后被删除
  • Alpha
    • CSI Health Monitoring
    • Topology Aware Hints
      • 在 EndpointSlice 和 Endpoints 添加元信息 - hints
      • KEP-2433
    • kEP-2255 Pod label controller.kubernetes.io/pod-deletion-cost 标识 删除成本
      • 默认 0
      • 影响 ReplicaSet 缩容调度 逻辑
    • KEP-2214 Job 索引
    • KEP-2232 停止 Job
      • .spec.suspend=true 停止，设置为 false 恢复调度
    • KEP_2227 Pod 默认容器标签 kubectl.kubernetes.io/default-container
      • 设置后 kubectl exec 可以不在指定容器 - 使用上会方便很多
  • 废弃
  • 参考
stagefeaturedefaultalphabetastable
AlphaBalanceAttachedNodeVolumes1.11
AlphaControllerManagerLeaderMigration1.211.22
AlphaControllerManagerLeaderMigration1.211.22
AlphaCSIVolumeHealth1.21
AlphaDaemonSetUpdateSurge1.211.22
AlphaEndpointSliceTerminatingCondition1.201.22
AlphaIndexedJob1.211.22
AlphaIngressClassNamespacedParams1.211.22
AlphaInTreePluginAWSUnregister1.21
AlphaInTreePluginAzureDiskUnregister1.21
AlphaInTreePluginAzureFileUnregister1.21
AlphaInTreePluginGCEUnregister1.21
AlphaInTreePluginOpenStackUnregister1.21
AlphaInTreePluginvSphereUnregister1.21
AlphaKubeletPodResourcesGetAllocatable1.21
AlphaLogarithmicScaleDown1.211.22
AlphaMemoryManager1.211.22
AlphaNetworkPolicyEndPort1.211.22
AlphaPodAffinityNamespaceSelector1.211.22
AlphaPodDeletionCost1.211.22
AlphaPreferNominatedNode1.211.22
AlphaProbeTerminationGracePeriod1.211.22
AlphaServiceInternalTrafficPolicy1.211.22
AlphaServiceLBNodePortControl1.201.22
AlphaServiceLoadBalancerClass1.211.22
AlphaSizeMemoryBackedVolumes1.201.22
AlphaSuspendJob1.211.22
AlphaTopologyAwareHints1.21
AlphaVolumeCapacityPriority1.21
BetaBoundServiceAccountTokenVolumetrue1.131.211.22
BetaCronJobControllerV2true1.201.211.22
BetaCSIMigrationAzureFilefalse1.151.21
BetaCSIMigrationvSphereCompletefalse1.19
BetaCSIServiceAccountTokentrue1.201.211.22
BetaCSIStorageCapacitytrue1.191.21
BetaDownwardAPIHugePagesfalse1.201.21
BetaDynamicKubeletConfigtrue1.41.11
BetaEfficientWatchResumptiontrue1.201.21
BetaEndpointSliceProxyingtrue1.181.191.22
BetaGenericEphemeralVolumetrue1.191.21
BetaGracefulNodeShutdowntrue1.201.21
BetaHugePageStorageMediumSizetrue1.181.191.22
BetaHugePageStorageMediumSizetrue1.181.191.22
BetaIPv6DualStacktrue1.151.21
BetaNamespaceDefaultLabelNametrue1.211.22
BetaServerSideApplytrue1.141.161.22
BetaSetHostnameAsFQDNtrue1.191.201.22
BetaTTLAfterFinishedtrue1.121.21
BetaValidateProxyRedirectstrue1.121.14
BetaWarningHeaderstrue1.191.22
BetaWindowsEndpointSliceProxyingtrue1.191.211.22
DeprecatedCSIMigrationAWSComplete1.17
DeprecatedCSIMigrationAzureDiskComplete1.17
DeprecatedCSIMigrationAzureFileComplete1.17
DeprecatedCSIMigrationGCEComplete1.17
DeprecatedCSIMigrationOpenStackComplete1.17
DeprecatedStreamingProxyRedirectstrue1.5
StableCRIContainerLogRotation1.101.111.21
StableEndpointSlice1.161.171.21
StableEndpointSliceNodeName1.201.21
StableImmutableEphemeralVolumes1.181.191.21
StableLegacyNodeRoleBehaviorfalse1.161.191.21
StableNodeDisruptionExclusion1.161.191.21
StablePodDisruptionBudget1.31.51.21
StableRootCAConfigMap1.131.201.21
StableRunAsGroup1.141.21
StableServiceAccountIssuerDiscovery1.181.201.21
StableServiceNodeExclusion1.81.191.21
StableSysctls1.111.21

POD

apiVersion: v1
kind: Pod
metadata:
  name: sysctl-example
  annotations:
    # 旧的 sysctl 控制
    security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1
    # 定义 kubectl 的默认 容器
    kubectl.kubernetes.io/default-container: 'test'
    # 删除成本 Alpha
    # 影响调度
    controller.kubernetes.io/pod-deletion-cost: '10'
spec:
  securityContext:
    # 新的 sysctl 控制
    sysctls:
      - name: kernel.shm_rmid_forced
        value: 1
  # 优雅停止 - Alpha
  # Pod 级别
  terminationGracePeriodSeconds: 3600
  containers:
    containers:
      - name: test
        image: alpine
        annotations:
        livenessProbe:
          # 容器覆盖 Pod 级别配置
          terminationGracePeriodSeconds: 60

JOB

apiVersion: batch/v1
 kind: Job
 metadata:
   name: 'indexed-job'
 spec:
  # 停止的 Job
  suspend: true

  completions: 5
  parallelism: 3
  # 索引 Job
  # 每个 Pod 的索引为 0 到 completions-1
  # 默认 NonIndexed
  completionMode: Indexed

  # TTLAfterFinished
  ttlSecondsAfterFinished: 100

Kubernetes 1.20​

  • 11 stable, 15 beta, 16 alpha
  • 废弃 dockershim
    • 只是废弃 dockershim - 暂无移除计划
    • 使用 CRI 标准接口 - 目前 docker 不支持, containerd 默认实现
    • docker 底层使用 containerd, docker 依然能正常使用, 只是移除了特殊中间层支持: docker -> shim -> containerd
    • 非 Linux 平台 containerd 无支持，需要 docker
    • Mirantis 会通过 Mirantis/cri-dockerd 添加 docker 的 cri 支持
    • 结论
      • 维持现状
  • go 1.15.5
  • 不再默认包含云平台控制器 - 由云平台自行提供
  • Stable/GA
    • CSI 卷快照
    • PID 限制 - SupportNodePidsLimit
    • TokenRequest/TokenRequestProjection - 目前还需要打开特性，1.21 会默认打开
    • node.k8s.io API 进入 v1 废弃 v1beta1
  • Beta
    • API 优先级处理 - 对 API 进行优先级分层，优先处理重要 API 请求
    • 非递归 卷所归属关系/fsgroup - 可理解为 chown $GROUP:$USER $VOLUME
    • FSGroup CSIDriver 策略
    • RootCAConfigMap - 所有 NS 可访问，用于校验 kube-apiserver 链接
    • kubectl debug
    • SetHostnameAsFQDN
  • Alpha
    • CronJob v2
    • 从新实现 IPv4/IPv6 双栈
    • CSI 安全增强 - CSIServiceAccountToken - 使用其他 SA 而非当前
    • 优雅的节点停止 - GracefulNodeShutdown
    • 移除日志敏感信息 - --experimental-logging-sanitization
      • 移除字段 passwords, keys, tokens
      • 支持组件 - 不支持用户 Pod 维度相关日志
        • kube-controller-manager
        • kube-apiserver
        • kube-scheduler
        • kubelet
    • 所有 Pod 资源指标 /metrics/resources - --show-hidden-metrics-for-version=1.20
  • 可能遇到的问题
    • exec probe 不会超时，现在 ExecProbeTimeout 会生效，没有配置的默认 1s
  • CHANGELOG-1.20
# IPv4/IPv6 双栈
apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    app: MyApp
spec:
  # 可集群维度配置
  # ipFamilyPolicy: SingleStack # 单栈
  ipFamilyPolicy: PreferDualStack
  # 可选
  ipFamilies:
    - IPv6
    - IPv4
  selector:
    app: MyApp
  ports:
    - protocol: TCP
      port: 80

Kubernetes 1.19​

  • EndpointSlices 默认启用
  • Ingress GA
  • seccomp GA
  • KubeSchedulerConfiguration Beta
  • 存储容量跟踪 - Alpha
    • 之前调度时都不会考虑节点存储容量
  • 通用临时存储卷
  • 不可变的 Secrets 和 ConfigMaps - Beta
  • kubernetes/dashboard v2
  • 通用 ephemeral volumes
    • 生命周期与 pod 绑定
    • 支持使用动态 provisioning 存储作为临时卷 - ALPHA
  • CHANGELOG-1.19

Script​

extract-features.js
const version = '1.21';
function build(ver = version) {
  const all = Array.prototype.map.call($$('table tbody tr'), ($tr) =>
    Array.prototype.map.call($tr.querySelectorAll('td'), ($td) => $td.innerText),
  );
  const byName = all.reduce((o, [name, _, stage, s]) => {
    const v = (o[name] ||= {});
    v[stage] ??= s;
    return o;
  }, {});

  let rows = all
    .filter((v) => v[3] === ver || v[4] === ver)
    .map(([name, def, stage, s, u]) => {
      const o = byName[name];
      stage = stage.replace('GA', 'Stable');
      def = def.replace('-', '');
      switch (stage) {
        case 'Alpha':
          def = def.replace('false', '');
          break;
        case 'Stable':
          def = def.replace('true', '');
          break;
      }
      return [stage, name, def, o.Alpha, o.Beta, o.GA];
    });

  copy(rows.map((v) => v.join('|')).join('\n'));
}