從 v1.3.2 升級到 v1.4.0

一般資訊

每當有可供升級的新 Harvester 版本時,Dashboard 畫面上就會出現 Upgrade 按鈕。如需更多資訊,請參閱開始升級。

對於離線環境,請參閱準備離線升級。

升級期間防止虛擬機映像檔損壞

警告

在升級到 Harvester v1.4.0 之前,請確保 BackingImage CRD 已事先更新到 Longhorn v1.7.2 版本。

如果跳過此步驟,可能會導致 backing image 損壞,如此已知 Longhorn 問題中所述。

要防止問題發生,您可以在升級 Harvester 之前手動更新 BackingImage CRD。

1. 修補 Harvester managedchart 以避免相關錯誤和警告。

kubectl patch managedchart harvester \
-n fleet-local \
--type='json' \
-p='[
  {
    "op":"add",
    "path":"/spec/diff/comparePatches/-",
    "value": {
      "apiVersion":"apiextensions.k8s.io/v1",
      "jsonPointers":["/spec","/metadata/annotations", "/metadata/labels", "/status"],
      "kind":"CustomResourceDefinition",
      "name":"backingimages.longhorn.io"
    }
  }
]'

1. 套用 Longhorn v1.7.2 BackingImage CRD。

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.15.0
  labels:
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/version: v1.7.2
    longhorn-manager: ""
  name: backingimages.longhorn.io
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: longhorn-conversion-webhook
          namespace: longhorn-system
          path: /v1/webhook/conversion
          port: 9501
      conversionReviewVersions:
      - v1beta2
      - v1beta1
  group: longhorn.io
  names:
    kind: BackingImage
    listKind: BackingImageList
    plural: backingimages
    shortNames:
    - lhbi
    singular: backingimage
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: The backing image name
      jsonPath: .spec.image
      name: Image
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: BackingImage is where Longhorn stores backing image object.
        properties:
          apiVersion:
            description: |-
              APIVersion defines the versioned schema of this representation of an object.
              Servers should convert recognized schemas to the latest internal value, and
              may reject unrecognized values.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            type: string
          kind:
            description: |-
              Kind is a string value representing the REST resource this object represents.
              Servers may infer this from the endpoint the client submits requests to.
              Cannot be updated.
              In CamelCase.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            type: string
          metadata:
            type: object
          spec:
            x-kubernetes-preserve-unknown-fields: true
          status:
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: false
    subresources:
      status: {}
  - additionalPrinterColumns:
    - description: The system generated UUID
      jsonPath: .status.uuid
      name: UUID
      type: string
    - description: The source of the backing image file data
      jsonPath: .spec.sourceType
      name: SourceType
      type: string
    - description: The backing image file size in each disk
      jsonPath: .status.size
      name: Size
      type: string
    - description: The virtual size of the image (may be larger than file size)
      jsonPath: .status.virtualSize
      name: VirtualSize
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1beta2
    schema:
      openAPIV3Schema:
        description: BackingImage is where Longhorn stores backing image object.
        properties:
          apiVersion:
            description: |-
              APIVersion defines the versioned schema of this representation of an object.
              Servers should convert recognized schemas to the latest internal value, and
              may reject unrecognized values.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            type: string
          kind:
            description: |-
              Kind is a string value representing the REST resource this object represents.
              Servers may infer this from the endpoint the client submits requests to.
              Cannot be updated.
              In CamelCase.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            type: string
          metadata:
            type: object
          spec:
            description: BackingImageSpec defines the desired state of the Longhorn
              backing image
            properties:
              checksum:
                type: string
              diskFileSpecMap:
                additionalProperties:
                  properties:
                    evictionRequested:
                      type: boolean
                  type: object
                type: object
              diskSelector:
                items:
                  type: string
                type: array
              disks:
                additionalProperties:
                  type: string
                description: Deprecated. We are now using DiskFileSpecMap to assign
                  different spec to the file on different disks.
                type: object
              minNumberOfCopies:
                type: integer
              nodeSelector:
                items:
                  type: string
                type: array
              secret:
                type: string
              secretNamespace:
                type: string
              sourceParameters:
                additionalProperties:
                  type: string
                type: object
              sourceType:
                enum:
                - download
                - upload
                - export-from-volume
                - restore
                - clone
                type: string
            type: object
          status:
            description: BackingImageStatus defines the observed state of the Longhorn
              backing image status
            properties:
              checksum:
                type: string
              diskFileStatusMap:
                additionalProperties:
                  properties:
                    lastStateTransitionTime:
                      type: string
                    message:
                      type: string
                    progress:
                      type: integer
                    state:
                      type: string
                  type: object
                nullable: true
                type: object
              diskLastRefAtMap:
                additionalProperties:
                  type: string
                nullable: true
                type: object
              ownerID:
                type: string
              size:
                format: int64
                type: integer
              uuid:
                type: string
              virtualSize:
                description: Virtual size of image, which may be larger than physical
                  size. Will be zero until known (e.g. while a backing image is uploading)
                format: int64
                type: integer
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

3. 開始升級過程。

---

已知問題

1. 帶有容器磁碟的虛擬機無法遷移,導致升級卡在預清空狀態

提示

手動停止虛擬機以繼續升級過程。

當從 v1.3.2 升級到 v1.4.0 時,如果帶有容器磁碟的虛擬機無法遷移,升級過程可能會卡住。即時遷移存在一些限制。

相關 issue:#7005

2. 升級卡在等待 Harvester Bundle

當從 v1.3.2 升級到 v1.4.0 時,升級過程可能會卡在等待 Harvester bundle 變為就緒狀態。此問題是由 Fleet 代理程式(fleet-agent)重新部署時的競爭條件引起的。

以下錯誤訊息表示問題存在。

> kubectl get bundles -n fleet-local
NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
mcc-harvester                                 0/1                       ErrApplied(1) [Cluster fleet-local/local: encountered 2 deletion errors. First is: admission webhook "validator.harvesterhci.io" denied the request: Internal error occurred: no route match found for DELETE /v1, Kind=Secret harvester-system/sh.helm.release.v1.harvester.v2]
mcc-harvester-crd                             0/1                       ErrApplied(1) [Cluster fleet-local/local: admission webhook "validator.harvesterhci.io" denied the request: Internal error occurred: no route match found for DELETE /v1, Kind=Secret harvester-system/sh.helm.release.v1.harvester-crd.v1]

您可以執行以下腳本來修復問題。

#!/bin/bash

patch_fleet_bundle() {
  local bundleName=$1
  local generation=$(kubectl get -n fleet-local bundle ${bundleName} -o jsonpath='{.spec.forceSyncGeneration}')
  local new_generation=$((generation+1))
  patch_manifest="$(mktemp)"
  cat > "$patch_manifest" <<EOF
{
  "spec": {
    "forceSyncGeneration": $new_generation
  }
}
EOF
  echo "patch bundle to new generation: $new_generation"
  kubectl patch -n fleet-local bundle ${bundleName}  --type=merge --patch-file $patch_manifest
  rm -f $patch_manifest
}

echo "removing harvester validating webhook"
kubectl delete validatingwebhookconfiguration harvester-validator

for bundle in mcc-harvester-crd mcc-harvester
do
  patch_fleet_bundle ${bundle}
done

echo "removing longhorn services"
kubectl delete svc longhorn-engine-manager -n longhorn-system --ignore-not-found=true
kubectl delete svc longhorn-replica-manager -n longhorn-system --ignore-not-found=true

3. 升級卡在等待 Fleet

當從 v1.3.2 升級到 v1.4.0 時,升級過程可能會卡在等待 Fleet 變為就緒狀態。此問題是由 Rancher 重新部署時的競爭條件引起的。

檢查 Harvester 日誌和 Fleet 歷史記錄以查找以下指標:

• manifest Pod 卡在 deployed 狀態。
• 升級待處理,但 chart 版本已部署。

範例:

> kubectl logs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest
wait helm release cattle-fleet-system fleet fleet-104.0.2+up0.10.2 0.10.2 deployed

> helm history -n cattle-fleet-system fleet
REVISION	UPDATED                 	STATUS         	CHART                	APP VERSION	DESCRIPTION
26      	Tue Dec 10 03:09:13 2024	superseded     	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
27      	Sun Dec 15 09:26:54 2024	superseded     	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
28      	Sun Dec 15 09:27:03 2024	superseded     	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
29      	Mon Dec 16 05:57:03 2024	deployed       	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
30      	Mon Dec 16 05:57:13 2024	pending-upgrade	fleet-103.1.5+up0.9.5	0.9.5      	Preparing upgrade

您可以執行以下命令來修復問題。

helm rollback fleet -n cattle-fleet-system <last-deployed-revision>

4. 點擊「Dismiss it」按鈕後升級意外重新啟動

當您使用 Rancher 升級 Harvester 時,Rancher UI 會顯示一個帶有標記為「Dismiss it」按鈕的對話框。點擊此按鈕可能會導致以下問題:

• harvesterhci.io/v1beta1/upgrade CR 的 status 部分被清除,導致丟失所有關於升級的重要資訊。
• 升級過程意外重新啟動。

此問題影響使用 v1.0.2、v1.0.3 和 v1.0.4 的 Harvester UI Extension 的 Rancher v2.10.x。所有 Harvester UI 版本不受影響。此問題將在 Harvester UI Extension v1.0.5 和 v1.5.0 中修復。

要避免此問題,請執行以下任一操作:

• 使用 Harvester UI 升級 Harvester。在 Harvester UI 上點擊「Dismiss it」按鈕不會導致意外行為。
• 不要點擊 Rancher UI 上的按鈕,而是對叢集執行以下命令:

kubectl -n harvester-system label upgrades -l harvesterhci.io/latestUpgrade=true harvesterhci.io/read-message=true

相關 issue:#7791

5. 使用可遷移 RWX 卷的虛擬機意外重新啟動

Longhorn 問題導致使用可遷移 RWX 卷的虛擬機在 CSI 外掛 Pod 重新啟動時意外重新啟動。此問題影響 Harvester v1.4.x、v1.5.0 和 v1.5.1。

解決方法是在開始升級之前在 Longhorn UI 上停用設定 Automatically Delete Workload Pod When The Volume Is Detached Unexpectedly。升級完成後,您必須再次啟用該設定。

此問題將在 Longhorn v1.8.3、v1.9.1 及更高版本中修復。Harvester v1.6.0 將包含 Longhorn v1.9.1。

相關 issue:#8534 和 #11158