從 v1.3.1 升級到 v1.3.2
一般資訊
每當有可供升級的新 Harvester 版本時,Dashboard 畫面上就會出現 Upgrade 按鈕。如需更多資訊,請參閱開始升級。
對於離線環境,請參閱準備離線升級。
已知問題
1. 雙節點叢集升級在第一個節點預清空後卡住
important
在升級雙節點叢集之前關閉所有工作負載虛擬機以防止資料遺失。
當在管理節點上升級 RKE2 時,工作節點可能會錯誤地轉換為未就緒狀態。因此,工作節點上的現有 Pod 被驅逐,新 Pod 無法在任何節點上排程。這些最終導致整個叢集的連鎖失敗並阻止升級過程完成。
當發生以下情況時,請檢查叢集狀態:
- 升級過程卡住一段時間。
- 您無法存取 Harvester UI 並收到 HTTP 503 錯誤。
-
檢查最新
Upgrade自訂資源的條件和節點狀態。如果滿足以下條件,請繼續下一步:
SystemServicesUpgraded設定為True,表示系統服務升級已完成。- 在
nodeStatuses中,管理節點的狀態為Pre-drained或Waiting Reboot。 - 在
nodeStatuses中,工作節點的狀態為Images preloaded。
範例:
# Find out the latest Upgrade custom resource
$ kubectl -n harvester-system get upgrades.harvesterhci -l harvesterhci.io/latestUpgrade=true
NAME AGE
hvst-upgrade-szlg8 48m
# Check the conditions and node statuses
$ kubectl -n harvester-system get upgrades hvst-upgrade-szlg8 -o yaml
apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
...
labels:
harvesterhci.io/latestUpgrade: "true"
harvesterhci.io/upgradeState: UpgradingNodes
name: hvst-upgrade-szlg8
namespace: harvester-system
...
spec:
image: ""
logEnabled: false
version: v1.3.2-rc2
status:
conditions:
- status: Unknown
type: Completed
- lastUpdateTime: "2024-09-02T11:57:04Z"
message: Upgrade observability is administratively disabled
reason: Disabled
status: "False"
type: LogReady
- lastUpdateTime: "2024-09-02T11:58:01Z"
status: "True"
type: ImageReady
- lastUpdateTime: "2024-09-02T12:02:31Z"
status: "True"
type: RepoReady
- lastUpdateTime: "2024-09-02T12:18:44Z"
status: "True"
type: NodesPrepared
- lastUpdateTime: "2024-09-02T12:31:25Z"
status: "True"
type: SystemServicesUpgraded
- status: Unknown
type: NodesUpgraded
imageID: harvester-system/hvst-upgrade-szlg8
nodeStatuses:
harvester-c6phd:
state: Pre-drained
harvester-jkqhq:
state: Images preloaded
previousVersion: v1.3.1
... -
檢查節點狀態。
如果滿足以下條件,請繼續下一步:
- 工作節點的狀態為
NotReady。 - 管理節點的狀態為
Ready,SchedulingDisabled。
範例:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
harvester-c6phd Ready,SchedulingDisabled control-plane,etcd,master 174m v1.28.12+rke2r1
harvester-jkqhq NotReady <none> 166m v1.27.13+rke2r1 - 工作節點的狀態為
-
檢查工作節點上的 Pod。
如果大多數 Pod 的狀態為
Terminating,則叢集中存在此問題。範例:
# Assume harvester-jkqhq is the worker node
$ kubectl get pods -A --field-selector spec.nodeName=harvester-jkqhq
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-fleet-local-system fleet-agent-6779fb5dd9-dkpjz 1/1 Terminating 0 18m
cattle-fleet-system fleet-agent-86db8d9954-qgcpq 1/1 Terminating 2 (18m ago) 61m
cattle-fleet-system fleet-controller-696d4b8878-ddctd 1/1 Terminating 1 (19m ago) 29m
cattle-fleet-system gitjob-694dd97686-s4z68 1/1 Terminating 1 (19m ago) 29m
cattle-provisioning-capi-system capi-controller-manager-6f497d5574-wkrnf 1/1 Terminating 0 20m
cattle-system cattle-cluster-agent-76db9cf9fc-5hhsx 1/1 Terminating 0 20m
cattle-system cattle-cluster-agent-76db9cf9fc-dnr6m 1/1 Terminating 0 20m
cattle-system harvester-cluster-repo-7458c7c69d-p982g 1/1 Terminating 0 27m
cattle-system rancher-7d65df9bd4-77n7w 1/1 Terminating 0 31m
cattle-system rancher-webhook-cfc66d5d7-fd6gm 1/1 Terminating 0 28m
harvester-system harvester-85ff674986-wxkl4 1/1 Terminating 0 26m
harvester-system harvester-load-balancer-54cd9754dc-cwtxg 1/1 Terminating 0 20m
harvester-system harvester-load-balancer-webhook-c8699b786-x6clw 1/1 Terminating 0 20m
harvester-system harvester-network-controller-manager-b69bf6b69-9f99x 1/1 Terminating 0 178m
harvester-system harvester-network-controller-vs4jg 1/1 Running 0 178m
harvester-system harvester-network-webhook-7b98f8cd98-gjl8b 1/1 Terminating 0 20m
harvester-system harvester-node-disk-manager-tbh4b 1/1 Running 0 26m
harvester-system harvester-node-manager-7pqcp 1/1 Running 0 178m
harvester-system harvester-node-manager-webhook-9cfccc84c-68tgp 1/1 Running 0 20m
harvester-system harvester-node-manager-webhook-9cfccc84c-6bbvg 1/1 Running 0 20m
harvester-system harvester-webhook-565dc698b6-np89r 1/1 Terminating 0 26m
harvester-system hvst-upgrade-szlg8-apply-manifests-4rmjw 0/1 Completed 0 33m
harvester-system virt-api-6fb7d97b68-cbc5m 1/1 Terminating 0 20m
harvester-system virt-api-6fb7d97b68-gqg5c 1/1 Terminating 0 23m
harvester-system virt-controller-67d8b4c75c-5qz9x 1/1 Terminating 0 24m
harvester-system virt-controller-67d8b4c75c-bdf8w 1/1 Terminating 2 (18m ago) 23m
harvester-system virt-handler-xw98h 1/1 Running 0 24m
harvester-system virt-operator-6c98db546-brgnx 1/1 Terminating 2 (18m ago) 26m
kube-system harvester-snapshot-validation-webhook-b75f94bcb-95zlb 1/1 Terminating 0 20m
kube-system harvester-snapshot-validation-webhook-b75f94bcb-xfrmf 1/1 Terminating 0 20m
kube-system harvester-whereabouts-tdr5g 1/1 Running 1 (178m ago) 178m
kube-system helm-install-rke2-ingress-nginx-4wt4j 0/1 Terminating 0 15m
kube-system helm-install-rke2-metrics-server-jn58m 0/1 Terminating 0 15m
kube-system kube-proxy-harvester-jkqhq 1/1 Running 0 178m
kube-system rke2-canal-wfpch 2/2 Running 0 178m
kube-system rke2-coredns-rke2-coredns-864fbd7785-t7k6t 1/1 Terminating 0 178m
kube-system rke2-coredns-rke2-coredns-autoscaler-6c87968579-rg6g4 1/1 Terminating 0 20m
kube-system rke2-ingress-nginx-controller-d4h25 1/1 Running 0 178m
kube-system rke2-metrics-server-7f745dbddf-2mp5j 1/1 Terminating 0 20m
kube-system rke2-multus-fsp94 1/1 Running 0 178m
kube-system snapshot-controller-65d5f465d9-5b2sb 1/1 Terminating 0 20m
kube-system snapshot-controller-65d5f465d9-c264r 1/1 Terminating 0 20m
longhorn-system backing-image-manager-c16a-7c90 1/1 Terminating 0 54m
longhorn-system csi-attacher-5fbd66cf8-674vc 1/1 Terminating 0 20m
longhorn-system csi-attacher-5fbd66cf8-725mn 1/1 Terminating 0 20m
longhorn-system csi-attacher-5fbd66cf8-85k5d 1/1 Terminating 0 20m
longhorn-system csi-provisioner-5b6ff8f4d4-97wsf 1/1 Terminating 0 20m
longhorn-system csi-provisioner-5b6ff8f4d4-cbpm9 1/1 Terminating 0 20m
longhorn-system csi-provisioner-5b6ff8f4d4-q7z58 1/1 Terminating 0 19m
longhorn-system csi-resizer-74c5555748-6rmbf 1/1 Terminating 0 20m
longhorn-system csi-resizer-74c5555748-fw2cw 1/1 Terminating 0 20m
longhorn-system csi-resizer-74c5555748-p4nph 1/1 Terminating 0 20m
longhorn-system csi-snapshotter-6bc4bcf4c5-6858b 1/1 Terminating 0 20m
longhorn-system csi-snapshotter-6bc4bcf4c5-cqkbw 1/1 Terminating 0 20m
longhorn-system csi-snapshotter-6bc4bcf4c5-mkqtg 1/1 Terminating 0 20m
longhorn-system engine-image-ei-b0369a5d-2t4k4 1/1 Running 0 178m
longhorn-system instance-manager-a5bd20597b82bcf3ba9d314620b7e670 1/1 Terminating 0 178m
longhorn-system longhorn-csi-plugin-x6bdg 3/3 Running 0 178m
longhorn-system longhorn-driver-deployer-85cf4b4849-5lc52 1/1 Terminating 0 20m
longhorn-system longhorn-loop-device-cleaner-hhvgv 1/1 Running 0 178m
longhorn-system longhorn-manager-5h2zw 1/1 Running 0 178m
longhorn-system longhorn-ui-6b677889f8-hrg8j 1/1 Terminating 0 20m
longhorn-system longhorn-ui-6b677889f8-w5hng 1/1 Terminating 0 20m
要解決此問題,您必須重新啟動工作節點上的 rke2-agent 服務。
# On the worker node
sudo systemctl restart rke2-agent.service
rke2-agent 服務完全重新啟動後,升級應該恢復。
備註
此問題的發生是因為工作節點上的代理程式負載平衡器在 rke2-server 服務重新啟動後無法連線到管理節點上的 API 伺服器。由於升級節點時 rke2-server 服務可能會重新啟動多次,因此升級過程可能會再次卡住。您可能需要多次重新啟動 rke2-agent 服務。
要確定代理程式負載平衡器是否正常運作,請執行以下命令:
# On the management node, check if the `rke2-server` service is running.
sudo systemctl status rke2-server.service
# On the worker node, check if the agent load balancer is functioning.
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig get nodes
如果 kubectl 命令沒有返回回應,則 kubelet 無法透過代理程式負載平衡器存取 API 伺服器。您必須重新啟動 rke2-agent 服務。
如需更多資訊,請參閱 Issue #6432。
2. 自動映像檔清理無法運作
由於發布的 Harvester ISO 包含不完整的映像檔列表,因此在從 v1.3.1 升級到 v1.3.2 期間無法執行自動映像檔清理。此問題不會阻止升級,您可以在升級完成後使用此腳本手動清理容器映像檔。如需更多資訊,請參閱 issue #6620。
3. 升級過程卡在「Pre-draining」狀態
由於即時遷移功能的限制,帶有容器磁碟的虛擬機無法遷移。這會導致升級過程卡在「Pre-draining」狀態。
提示
手動停止虛擬機以繼續升級過程。
如需更多資訊,請參閱 Issue #7005。