更換故障的控制器節點
在我們的 CubeCOS 叢集中，其中一個控制器節點發生硬體故障，導致其無法繼續運作。 由於控制器節點對雲端管理至關重要，我們必須迅速替換它以確保系統持續穩定運行。
我們已將故障節點從叢集移除，並準備了一台具有相同主機名稱和基礎設定的新伺服器。 在重新安裝作業系統後，我們遵循逐步流程恢復 CubeCOS 服務，並將節點重新加入叢集。
本指南適用於以下角色：
- 控制
- 管理融合節點
- 邊緣運算
- 仲裁節點
它說明如何：
- 重建節點
- 重新配置 OpenStack 服務
- 將節點重新加入高可用性叢集
- 驗證系統是否如預期運作
從儲存池移除 OSD
1. Connect to any healthy node in the cluster.
2. Run the remove_osd command to remove each failed OSD.
3. Repeat the process until all missing OSDs have been removed.
4. Use status to verify that all offline OSDs are gone.
cc2:storage> remove_osd
Enter osd id to be removed:
1: osd.0 (hdd)
2: osd.1 (hdd)
3: osd.2 (hdd)
4: osd.3 (hdd)
5: osd.4 (hdd)
6: osd.5 (hdd)
7: osd.6 (hdd)
8: osd.7 (hdd)
Enter index: 1
Enter 'YES' to confirm: YES
Remove osd.0 successfully.
替換節點
選擇 進階 選項
First Time Setup Options:
1: Wizard
2: Advanced
Enter index: 2
必須設定控制節點角色以重新加入叢集
Welcome to the Cube Appliance
Enter "help" for a list of available commands
unconfigured> first
unconfigured:first> control_rejoin
Set or clear control rejoin flag?
1: set
2: clear
Enter index: 1
Control rejoin markers set
從儲存媒體取得系統快照
unconfigured> snapshot
unconfigured:snapshot> pull
Select a media:
1: usb
2: nfs
Enter index: 1
Insert a USB drive into the USB port on the appliance.
Enter 'YES' to confirm: YES
1: CUBE_2.4.3_20241208-130432.765318_cc1.snapshot
Enter index: 1
Copying...
Automatically generated on 2024-12-08 13:04:32
Copy complete. It is safe to remove the USB drive.
套用設定
unconfigured:snapshot> apply
1: CUBE_2.4.3_20241208-054308.133092_unconfigured.snapshot
2: CUBE_2.4.3_20241208-130432.765318_cc1.snapshot
Enter index: 2
Automatically generated on 2024-12-08 13:04:32
Date/Time is important for applying changes to an unconfigured box.
Please confirm the current time is good.
* Local Time: 12/08/2024 00:46:58 EST
Enter 'YES' to confirm: YES
Policy snapshot file applied
已管理者身份重新登入
unconfigured:snapshot> exit
cc1 Login: admin
Password:
Welcome to the Cube Appliance
Enter "help" for a list of available commands
Notice: your license will expire in 29 days.
Please contact system administrator to renew the license.
cc1>
切換至手動開機
cc1:boot_mode> manual
Switch to manual bootstrap mode:
1: one-time manual
2: always manual
Enter index: 1
Enter 'YES' to confirm: YES
Exit current CLI and log in again as admin to see changes in effect
退出管理員命令列（CLI）介面並重新登入
cc1:boot_mode> exit
# su admin
Welcome to the Cube Appliance
Enter "help" for a list of available commands
License (type: trial) is valid for 102 days
cc1>
叢集同步
cc1> boot cluster_sync
(1/9) processing: ceph
(2/9) processing: keystone
(3/9) processing: neutron
(4/9) processing: nova
(5/9) processing: cinder
(6/9) processing: manila
(7/9) processing: octavia
(8/9) processing: haproxy
(9/9) processing: telegraf
cluster_sync successfully
系統修復服務
cc1> cluster check_repair
Service Status Report
ClusterLink ok [ link(v) clock(v) dns(v) ]
ClusterSys ok [ bootstrap(v) license(v) ]
ClusterSettings ok [ etcd(v) ]
HaCluster FIXING [ hacluster(3) ]
ok [ hacluster(f) ]
MsgQueue ok [ rabbitmq(v) ]
IaasDb ok [ mysql(v) ]
VirtualIp ok [ vip(v) haproxy_ha(v) ]
Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ]
ApiService ok [ haproxy(v) httpd(v) lmi(v) memcache(v) ]
SingleSignOn ok [ keycloak(v) ]
Compute FIXING [ nova(8) ]
ok [ nova(f) ]
Baremetal ok [ ironic(v) ]
Network FIXING [ neutron(3) ]
ok [ neutron(f) ]
Image ok [ glance(v) ]
BlockStor ok [ cinder(v) ]
FileStor ok [ manila(v) ]
ObjectStor ok [ swift(v) ]
Orchestration ok [ heat(v) ]
LBaaS ok [ octavia(v) ]
DNSaaS ok [ designate(v) ]
K8SaaS ok [ k3s(v) rancher(v) ]
InstanceHa ok [ masakari(v) ]
BusinessLogic ok [ senlin(v) watcher(v) ]
DataPipe ok [ zookeeper(v) kafka(v) ]
Metrics ok [ monasca(v) telegraf(v) grafana(v) ]
LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) es(v) kibana(v) ]
Notifications ok [ influxdb(v) kapacitor(v) ]
切換至 root shell並刪除該檔案
rm -f /etc/appliance/state/boot_mode