移除故障硬碟
情境：硬碟故障時，您需要將其從叢集移除，並恢復儲存池的正常運作狀態。
Connect to the Console
Warning: Permanently added '192.168.1.x' (ECDSA) to the list of known hosts.
Password:
檢查儲存狀態
首先，我們需要先檢查儲存叢集的狀態。
如以下資訊，受影響節點主機名稱 (s3) 由於硬碟故障導致兩個OSD編號為56與58失效。
指令 status 處於 storage 模式。
s1:storage> status
cluster:
id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85
health: HEALTH_WARN
2 osds down
Degraded data redundancy: 1611/44204 objects degraded (3.644%), 106 pgs degraded
services:
mon: 3 daemons, quorum s1,s2,s3
mgr: s1(active), standbys: s2, s3
mds: cephfs-1/1/1 up {0=s3=up:active}, 2 up:standby
osd: 60 osds: 58 up, 60 in
rgw: 3 daemons active
data:
pools: 22 pools, 5488 pgs
objects: 21.59k objects, 82.8GiB
usage: 232GiB used, 92.3TiB / 92.6TiB avail
pgs: 1611/44204 objects degraded (3.644%)
4873 active+clean
509 active+undersized
106 active+undersized+degraded
io:
client: 0B/s rd, 1.88MiB/s wr, 2.74kop/s rd, 64op/s wr
recovery: 5B/s, 0objects/s
cache: 0op/s promote
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | s1 | 1597M | 445G | 0 | 0 | 0 | 4096 | exists,up |
| 1 | s1 | 1543M | 445G | 0 | 819 | 0 | 1638 | exists,up |
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
| 54 | s3 | 4562M | 1858G | 2 | 64.0k | 8 | 32.8k | exists,up |
| 55 | s2 | 3784M | 1859G | 6 | 119k | 429 | 1717k | exists,up |
| 56 | s3 | 3552M | 1859G | 0 | 0 | 0 | 0 | exists |
| 57 | s2 | 5285M | 1857G | 3 | 49.6k | 12 | 76.8k | exists,up |
| 58 | s3 | 4921M | 1858G | 0 | 0 | 0 | 0 | exists |
| 59 | s2 | 3865M | 1859G | 1 | 17.6k | 2 | 9011 | exists,up |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
移除硬碟
指令 remove_disk 在 storage 分類下。
移除硬碟有兩種模式：安全模式與強制模式。
- 在安全模式下，Ceph 會嘗試將硬碟上的所有 PG 進行遷移，藉此將現有資料遷移至其他儲存裝置，從而平穩移除該硬碟。
- 強制模式下，將不執行任何資料傳輸，指定的磁碟機將被強制移除。 此操作極可能導致 Ceph 叢集損毀。
輸入指令後，會顯示 /dev/sdj 於索引 10 處與 ID 56 及 ID 58 關聯。
我們需要將 /dev/sdj 從 Ceph 儲存池中移除。
s3> storage
s3:storage> remove_disk
index name size storage ids
--
1 /dev/sda 894.3G 21 23
2 /dev/sdb 894.3G 25 27
3 /dev/sdc 3.7T 29 31
4 /dev/sdd 3.7T 33 35
5 /dev/sde 3.7T 36 38
6 /dev/sdf 3.7T 41 42
7 /dev/sdg 3.7T 44 46
8 /dev/sdh 3.7T 48 50
9 /dev/sdi 3.7T 52 54
10 /dev/sdj 3.7T 56 58
--
Enter the index of disk to be removed: 10
Disk removal mode:
1: safe
2: force
Enter index: 1
safe mode takes longer by attempting to migrate data on disk(s).
Enter 'YES' to confirm: YES
Remove disk /dev/sdj successfully.
移除問題硬碟後，需要檢查儲存池的狀態。 Ceph 會自動恢復資料。
s3:storage> status
cluster:
id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85
health: HEALTH_WARN
276/44228 objects misplaced (0.624%)
Degraded data redundancy: 908/44228 objects degraded (2.053%), 68 pgs degraded
services:
mon: 3 daemons, quorum s1,s2,s3
mgr: s1(active), standbys: s3, s2
mds: cephfs-1/1/1 up {0=s3=up:active}, 2 up:standby
osd: 58 osds: 58 up, 58 in; 7 remapped pgs
rgw: 3 daemons active
data:
pools: 22 pools, 5488 pgs
objects: 21.60k objects, 82.9GiB
usage: 227GiB used, 88.7TiB / 88.9TiB avail
pgs: 908/44228 objects degraded (2.053%)
276/44228 objects misplaced (0.624%)
5409 active+clean
61 active+recovery_wait+degraded
7 active+recovering+degraded
4 active+recovering
4 active+remapped+backfill_wait
3 active+undersized+remapped+backfill_wait
io:
client: 273KiB/s rd, 34.7KiB/s wr, 318op/s rd, 4op/s wr
recovery: 135MiB/s, 0keys/s, 37objects/s
結果
稍後，我們便能透過在 storage 模式下使用 status 指令來檢查狀態。
若我們已成功移除故障硬碟，其健康狀態應為正常。
s1:storage> status
cluster:
id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85
health: HEALTH_OK
services:
mon: 3 daemons, quorum s1,s2,s3
mgr: s1(active), standbys: s2, s3
mds: cephfs-1/1/1 up {0=s3=up:active}, 2 up:standby
osd: 58 osds: 58 up, 58 in
rgw: 3 daemons active
data:
pools: 22 pools, 5488 pgs
objects: 21.59k objects, 82.8GiB
usage: 229GiB used, 88.7TiB / 88.9TiB avail
pgs: 5488 active+clean
io:
client: 132KiB/s rd, 5.44KiB/s wr, 159op/s rd, 0op/s wr
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | s1 | 1594M | 445G | 0 | 0 | 0 | 0 | exists,up |
| 1 | s1 | 1536M | 445G | 0 | 0 | 0 | 0 | exists,up |
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
| 54 | s3 | 4665M | 1858G | 3 | 29.6k | 0 | 0 | exists,up |
| 55 | s2 | 3769M | 1859G | 0 | 0 | 0 | 0 | exists,up |
| 57 | s2 | 5366M | 1857G | 0 | 819 | 0 | 0 | exists,up |
| 59 | s2 | 3851M | 1859G | 0 | 0 | 0 | 0 | exists,up |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
欲了解更多資訊，請查看狀態。
移除節點上所有現有 OSD
我們會在命令列（CLI）介面中使用 storage 模式下的 add_avail 指令。
cc1:storage> remove_exist
index name size osd serial
--
1 /dev/sda 745.2G 1 0 BTWA632602ZU800HGN
2 /dev/sdb 745.2G 3 2 BTWA632601X9800HGN
3 /dev/sdc 745.2G 5 4 BTWA632601RW800HGN
4 /dev/sdd 745.2G 7 6 BTWA6326038Z800HGN
5 /dev/sde 745.2G 9 8 BTWA632605GP800HGN
6 /dev/sdg 745.2G 11 10 BTWA632601U9800HGN
7 /dev/sdh 745.2G 13 12 BTWA632604RJ800HGN
8 /dev/sdi 745.2G 15 14 BTWA632602Q3800HGN
9 /dev/sdj 745.2G 16 17 BTWA63260373800HGN
10 /dev/sdk 745.2G 19 18 BTWA632605EV800HGN
11 /dev/sdl 745.2G 21 20 BTWA63250476800HGN
12 /dev/sdm 745.2G 22 23 BTWA6326047A800HGN
13 /dev/sdn 745.2G 24 25 BTWA632602Q0800HGN
14 /dev/sdo 744.6G 27 26 BTWA632605EV800HGN
--
Disk removal mode:
1: safe
2: force
Enter index: 1
safe mode takes longer by attempting to migrate data on disk(s).
Enter 'YES' to confirm: YES
--
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 ssd 0.36349 1.00000 372 GiB 4.9 GiB 4.9 GiB 2 KiB 21 MiB 367 GiB 1.32 1.00 61 up
TOTAL 372 GiB 4.9 GiB 4.9 GiB 2.7 KiB 21 MiB 367 GiB 1.32
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
OSD(s) 0 have 40 pgs currently mapped to them.
OSD(s) 0 have 36 pgs currently mapped to them.
OSD(s) 0 have 26 pgs currently mapped to them.
OSD(s) 0 have 14 pgs currently mapped to them.
marked down osd.0.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 ssd 0.36349 1.00000 372 GiB 11 GiB 9.9 GiB 2 KiB 1.3 GiB 361 GiB 3.01 1.00 64 up
TOTAL 372 GiB 11 GiB 9.9 GiB 2.7 KiB 1.3 GiB 361 GiB 3.01
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
OSD(s) 1 have 33 pgs currently mapped to them.
OSD(s) 1 have 3 pgs currently mapped to them.
OSD(s) 1 have 2 pgs currently mapped to them.
marked down osd.1.
Removed disk /dev/sda.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
2 ssd 0.36349 1.00000 372 GiB 15 GiB 15 GiB 4 KiB 94 MiB 358 GiB 3.95 1.00 76 up
TOTAL 372 GiB 15 GiB 15 GiB 4.8 KiB 94 MiB 358 GiB 3.95
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
OSD(s) 2 have 53 pgs currently mapped to them.
OSD(s) 2 have 46 pgs currently mapped to them.
OSD(s) 2 have 36 pgs currently mapped to them.
OSD(s) 2 have 24 pgs currently mapped to them.
OSD(s) 2 have 8 pgs currently mapped to them.
OSD(s) 2 have 4 pgs currently mapped to them.
Restored /dev/sdb: osd.2 as pgs could not be moved likely due to too few disks or little space in the failure domain.
Failed to remove disk /dev/sdb with safe mode for storage cannot become healthy
without it.
--
Processed 1 disk out of 14.