Add a Hard Disk
Scenario: We are having a new hard disk to add to our storage pool.
Connect to the Console
First, we need to connect to the console of the node having a new hard disk.
ssh [email protected]
Warning: Permanently added '192.168.1.x' (ECDSA) to the list of known hosts.
Password:
Add the New Disk
The CLI is under storage mode. The command to add the new disk is add_disk.
This example adds the disk with the raw mode instead of the encryption mode.
s3> storage
s3:storage> add_disk
index name size on_hours error serial
--
1 /dev/sdj 3.7T 66090 0 BTWA632602ZU800HGN
--
Found 1 available disk
Enter the index to add this disk into the pool: 1
Disk protection mode:
1: raw
2: encrypt
Enter index: 1
No disk encryption (default mode).
Enter 'YES' to confirm: YES
Add disk /dev/sdj successfully.
Wait for Data Recovery
This action is performed automatically. The OSD size is changed from 58 to 60.
s3:storage> status
cluster:
id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85
health: HEALTH_WARN
90/44225 objects misplaced (0.204%)
Degraded data redundancy: 1631/44225 objects degraded (3.688%), 114 pgs degraded
services:
mon: 3 daemons, quorum s1,s2,s3
mgr: s1(active), standbys: s3, s2
mds: cephfs-1/1/1 up {0=s3=up:active}, 2 up:standby
osd: 60 osds: 60 up, 60 in; 10 remapped pgs
rgw: 3 daemons active
data:
volumes: 1/1 healthy
pools: 22 pools, 5488 pgs
objects: 21.59k objects, 82.9GiB
usage: 232GiB used, 92.3TiB / 92.6TiB avail
pgs: 1631/44225 objects degraded (3.688%)
90/44225 objects misplaced (0.204%)
5362 active+clean
113 active+recovery_wait+degraded
9 active+remapped+backfill_wait
2 active+recovering
1 active+remapped+backfilling
1 active+recovering+degraded
io:
client: 0B/s rd, 78.1KiB/s wr, 15op/s rd, 15op/s wr
recovery: 40.4MiB/s, 12objects/s
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | s1 | 1587M | 445G | 0 | 0 | 0 | 0 | exists,up |
| 1 | s1 | 1535M | 445G | 0 | 0 | 0 | 0 | exists,up |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 55 | s2 | 3769M | 1859G | 0 | 0 | 0 | 0 | exists,up |
| 56 | s3 | 3525M | 1859G | 0 | 0 | 0 | 0 | exists,up |
| 57 | s2 | 5262M | 1857G | 0 | 0 | 0 | 0 | exists,up |
| 58 | s3 | 4895M | 1858G | 0 | 0 | 0 | 0 | exists,up |
| 59 | s2 | 3851M | 1859G | 0 | 0 | 0 | 0 | exists,up |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
For more information, check status.
Add the New Disk with Encryption
Similar to adding a new disk with raw mode, we also use the command add_disk under storage mode in CLI.
info
Disks added with raw mode would each have 2 OSDs. On the other hand, disks added with encryption mode would each have only 1 OSD.
Disks added with raw mode would have only 1 OSD in the coming version.
s3> storage
s3:storage> add_disk
index name size
--
1 /dev/sdj 3.7T
--
Found 1 available disk
Enter the index to add this disk into the pool: 1
Disk protection mode:
1: raw
2: encrypt
Enter index: 2
Encrypt disk(s) to protect physical disk loss (beware of performance impacts).
Enter 'YES' to confirm: YES
Added disk(encrypt) /dev/sdj.
Add All Available Disks on a Node
We would use command add_avail under storage mode in CLI.
There are also two modes to add disks, raw mode and encryption mode.
cc1:storage> add_avail
index name size on_hours error serial
--
1 /dev/sda 745.2G 66091 0 BTWA632602ZU800HGN
--
Disk protection mode:
1: raw
2: encrypt
Enter index: 2
Encrypt disk(s) to protect physical disk loss (beware of performance impacts).
Enter 'YES' to confirm: YES
--
====== Device report /dev/sda ======
path /dev/sda
ceph device False
lsm data {}
available False
rejected reasons locked, Has GPT headers, Has partitions
device id ATA_INTEL_SSDSC2BB800G6_BTWA632602ZU800HGN
removable 0
ro 0
vendor ATA
model INTEL SSDSC2BB80
sas address
rotational 0
scheduler mode mq-deadline
human readable size 745.21 GB
Added disk(encrypt) /dev/sda.
--
Processed 1 disk out of 1.
Check Status
CLI: storage > list_osd
List all OSDs in the Ceph storage cluster.
cc1:storage> list_osd
OSD STATE HOST DEV SERIAL POWER_ON USE REMARK
0 ok cc1 sda BTWA632602ZU800HGN 2753 days 0% LUKS2-encrypted
2 ok cc1 sdb3 BTWA632601X9800HGN 2753 days 3%
3 ok cc1 sdb4 BTWA632601X9800HGN 2753 days 2%
4 ok cc1 sdc3 BTWA632601RW800HGN 2753 days 2%
5 ok cc1 sdc4 BTWA632601RW800HGN 2753 days 3%
6 ok cc1 sdd3 BTWA6326038Z800HGN 2753 days 2%
7 ok cc1 sdd4 BTWA6326038Z800HGN 2753 days 6%
8 ok cc1 sde3 BTWA632605GP800HGN 2753 days 3%
9 ok cc1 sde4 BTWA632605GP800HGN 2753 days 2%
10 ok cc1 sdg3 BTWA632601U9800HGN 2753 days 3%
11 ok cc1 sdg4 BTWA632601U9800HGN 2753 days 2%
12 ok cc1 sdh3 BTWA632604RJ800HGN 2753 days 3%
13 ok cc1 sdh4 BTWA632604RJ800HGN 2753 days 2%
14 ok cc1 sdi3 BTWA632602Q3800HGN 2753 days 1%
15 ok cc1 sdi4 BTWA632602Q3800HGN 2753 days 2%
16 ok cc1 sdj3 BTWA63260373800HGN 2753 days 1%
17 ok cc1 sdj4 BTWA63260373800HGN 2753 days 3%
18 ok cc1 sdk3 BTWA632605EV800HGN 2753 days 1%
19 ok cc1 sdk4 BTWA632605EV800HGN 2753 days 1%
20 ok cc1 sdl3 BTWA63250476800HGN 2753 days 1%
21 ok cc1 sdl4 BTWA63250476800HGN 2753 days 2%
22 ok cc1 sdm3 BTWA6326047A800HGN 2753 days 3%
23 ok cc1 sdm4 BTWA6326047A800HGN 2753 days 3%
24 ok cc1 sdn3 BTWA632602Q0800HGN 2753 days 2%
25 ok cc1 sdn4 BTWA632602Q0800HGN 2753 days 3%
26 ok cc1 sdo3 BTWA632605EV800HGN 2753 days 1%
27 ok cc1 sdo4 BTWA632605EV800HGN 2753 days 4%
Column REMARK would show whether the disk is encrypted or not.
CLI: storage > list_osd [OSD ID]
Get more information about an OSD.
cc1:storage> list_osd 0
overall-health self-assessment test result: PASSED
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 66089
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 581
170 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
174 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 466
175 Power_Loss_Cap_Test 0x0033 100 100 010 Pre-fail Always - 6910 (366 2358)
183 SATA_Downshift_Count 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
190 Temperature_Case 0x0022 073 071 000 Old_age Always - 27 (Min/Max 25/29)
192 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 466
194 Temperature_Internal 0x0022 100 100 000 Old_age Always - 27
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 1924300
226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 7127
227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 23
228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 3965167
232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 093 093 000 Old_age Always - 0
234 Thermal_Throttle 0x0032 100 100 000 Old_age Always - 0/0
241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 1924300
242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 599478
243 NAND_Writes_32MiB 0x0032 100 100 000 Old_age Always - 5771690
HDD Device 0: /dev/sda
HDD Model ID : INTEL SSDSC2BB800G6
HDD Serial No: BTWA632602ZU800HGN
HDD Revision : G2010150
HDD Size : 763098 MB
Interface : SAT Standard USB/ATA
Temperature : 27 °C
Highest Temp.: 29 °C
Health : 93 %
Performance : 100 %
Power on time: 2753 days, 17 hours
Est. lifetime: 93 days
Total written: 58.72 TB
The status of the solid state disk is PERFECT. Problematic or weak sectors were not found.
The health is determined by SSD specific S.M.A.R.T. attribute(s): #233 Media Wearout Indicator
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 ssd 0.36349 1.00000 372 GiB 4.9 GiB 4.9 GiB 2 KiB 21 MiB 367 GiB 1.32 1.00 61 up
TOTAL 372 GiB 4.9 GiB 4.9 GiB 2.7 KiB 21 MiB 367 GiB 1.32
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
OSD STATE HOST DEV SERIAL POWER_ON USE REMARK
0 ok cc1 sda3 BTWA632602ZU800HGN 2753 days 1%