Add a Hard disk replacement

Story: We had bought a new hard disk to add in to our storage pool

Connect to console (which add the new hard disk)

$ ssh [email protected]
Warning: Permanently added '192.168.1.x' (ECDSA) to the list of known hosts.
Password:

Adding add new disk to the node with CLI storage> add_disk

s3> storage
s3:storage> add_disk
  index          name      size
--
      1      /dev/sdj      3.7T
--
Found 1 available disk
Enter the index to add this disk into the pool: 1
Enter 'YES' to confirm: YES
Add disk /dev/sdj successfully.

Wait for data recovery

• this action is automatically, osd size is changed from 58 to 60

s3:storage> status
  cluster:
    id:     c6e64c49-09cf-463b-9d1c-b6645b4b3b85
    health: HEALTH_WARN
            90/44225 objects misplaced (0.204%)
            Degraded data redundancy: 1631/44225 objects degraded (3.688%), 114 pgs degraded

  services:
    mon: 3 daemons, quorum s1,s2,s3
    mgr: s1(active), standbys: s3, s2
    mds: cephfs-1/1/1 up  {0=s3=up:active}, 2 up:standby
    osd: 60 osds: 60 up, 60 in; 10 remapped pgs
    rgw: 3 daemons active

  data:
    pools:   22 pools, 5488 pgs
    objects: 21.59k objects, 82.9GiB
    usage:   232GiB used, 92.3TiB / 92.6TiB avail
    pgs:     1631/44225 objects degraded (3.688%)
             90/44225 objects misplaced (0.204%)
             5362 active+clean
             113  active+recovery_wait+degraded
             9    active+remapped+backfill_wait
             2    active+recovering
             1    active+remapped+backfilling
             1    active+recovering+degraded

  io:
    client:   0B/s rd, 78.1KiB/s wr, 15op/s rd, 15op/s wr
    recovery: 40.4MiB/s, 12objects/s
        
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| id | host |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+------+-------+-------+--------+---------+--------+---------+-----------+
| 0  |  s1  | 1587M |  445G |    0   |     0   |    0   |     0   | exists,up |
| 1  |  s1  | 1535M |  445G |    0   |     0   |    0   |     0   | exists,up |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 55 |  s2  | 3769M | 1859G |    0   |     0   |    0   |     0   | exists,up |
| 56 |  s3  | 3525M | 1859G |    0   |     0   |    0   |     0   | exists,up |
| 57 |  s2  | 5262M | 1857G |    0   |     0   |    0   |     0   | exists,up |
| 58 |  s3  | 4895M | 1858G |    0   |     0   |    0   |     0   | exists,up |
| 59 |  s2  | 3851M | 1859G |    0   |     0   |    0   |     0   | exists,up |
+----+------+-------+-------+--------+---------+--------+---------+-----------+