# Storage node Replacement Guide > Make sure your storage data replication either 2 or 3. ## Remove Storage node from cluster {#remove-storage-node-from-cluster} Make sure your storage data replication either 2 or 3. ### Connect to controller {#connect-to-controller} ```bash $ ssh admin@192.168.1.x Warning: Permanently added '192.168.1.x' (ECDSA) to the list of known hosts. Password: ``` ### Check the storage status {#check-the-storage-status} ```bash cc1> storage cc1:storage> status cluster: id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85 health: HEALTH_OK services: mon: 3 daemons, quorum cc1,cc2,cc3 mgr: cc1(active), standbys: cc2, cc3 mds: cephfs-1/1/1 up {0=cc1=up:active}, 2 up:standby osd: 24 osds: 24 up, 24 in rgw: 3 daemons active data: pools: 23 pools, 1837 pgs objects: 10.50k objects, 12.7GiB usage: 31.3GiB used, 3.74TiB / 3.77TiB avail pgs: 1837 active+clean io: client: 15.5KiB/s rd, 0B/s wr, 15op/s rd, 10op/s wr +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | cc1 | 2063M | 117G | 0 | 0 | 2 | 61 | exists,up | | 1 | cc1 | 2020M | 117G | 0 | 0 | 1 | 36 | exists,up | | 2 | cc1 | 1089M | 135G | 0 | 0 | 0 | 0 | exists,up | | 3 | cc1 | 1081M | 135G | 0 | 0 | 0 | 0 | exists,up | | 4 | cc2 | 1656M | 117G | 0 | 0 | 0 | 0 | exists,up | | 5 | cc2 | 2073M | 116G | 0 | 0 | 0 | 0 | exists,up | | 6 | cc2 | 1089M | 135G | 0 | 0 | 0 | 0 | exists,up | | 7 | cc2 | 1089M | 135G | 0 | 0 | 4 | 0 | exists,up | | 8 | cc3 | 1781M | 117G | 0 | 0 | 0 | 0 | exists,up | | 9 | cc3 | 1961M | 117G | 0 | 0 | 7 | 157 | exists,up | | 10 | cc3 | 1089M | 135G | 0 | 0 | 0 | 0 | exists,up | | 11 | cc3 | 1089M | 135G | 0 | 0 | 0 | 0 | exists,up | | 12 | compute01 | 1462M | 56.5G | 0 | 0 | 0 | 0 | exists,up | | 13 | compute01 | 1400M | 56.6G | 0 | 0 | 0 | 0 | exists,up | | 14 | compute01 | 1334M | 56.7G | 0 | 0 | 0 | 6 | exists,up | | 15 | compute01 | 1426M | 56.6G | 0 | 0 | 0 | 0 | exists,up | | 16 | compute01 | 1101M | 464G | 0 | 0 | 0 | 19 | exists,up | | 17 | compute01 | 1089M | 464G | 0 | 0 | 0 | 0 | exists,up | | 18 | storage01 | 1040M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 19 | storage01 | 1040M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 20 | storage01 | 1040M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 21 | storage01 | 1048M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 22 | storage01 | 1081M | 464G | 0 | 0 | 0 | 0 | exists,up | | 23 | storage01 | 1105M | 464G | 0 | 0 | 0 | 0 | exists,up | +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ ``` ### Remove node {#remove-node} Removing storage01 from the cluster ```bash cc1:cluster> remove_node 1: compute01 2: storage01 3: cc1 4: cc3 5: cc2 Enter index: 2 this command is only applicable for compute or storage nodes make sure its running instances have been properly terminated or migrated shutdown the target host before proceeding Enter 'YES' to confirm: YES cc1:cluster> ``` ### Check the storage status {#check-the-storage-status-1} storage01 node has been removed from the cluster ```bash cc1> storage cc1:storage> status cluster: id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85 health: HEALTH_WARN Reduced data availability: 2 pgs inactive Degraded data redundancy: 139/21222 objects degraded (0.655%), 10 pgs degraded services: mon: 3 daemons, quorum cc1,cc2,cc3 mgr: cc1(active), standbys: cc2, cc3 mds: cephfs-1/1/1 up {0=cc1=up:active}, 2 up:standby osd: 18 osds: 18 up, 18 in; 510 remapped pgs rgw: 3 daemons active data: pools: 23 pools, 1837 pgs objects: 10.50k objects, 12.7GiB usage: 25.4GiB used, 2.61TiB / 2.63TiB avail pgs: 10.670% pgs unknown 0.435% pgs not active 139/21222 objects degraded (0.655%) 1406 active+clean 214 active+clean+remapped 196 unknown 9 active+recovery_wait+degraded 5 activating+remapped 3 activating 3 active+undersized 1 active+undersized+degraded+remapped+backfilling io: client: 13.1KiB/s rd, 0B/s wr, 13op/s rd, 9op/s wr recovery: 1.11MiB/s, 4objects/s +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | cc1 | 2060M | 117G | 0 | 0 | 0 | 0 | exists,up | | 1 | cc1 | 2025M | 117G | 0 | 0 | 0 | 0 | exists,up | | 2 | cc1 | 1093M | 135G | 0 | 0 | 0 | 0 | exists,up | | 3 | cc1 | 1086M | 135G | 10 | 0 | 5 | 69 | exists,up | | 4 | cc2 | 1668M | 117G | 0 | 0 | 0 | 0 | exists,up | | 5 | cc2 | 2086M | 116G | 0 | 0 | 0 | 0 | exists,up | | 6 | cc2 | 1093M | 135G | 0 | 0 | 0 | 0 | exists,up | | 7 | cc2 | 1094M | 135G | 0 | 0 | 4 | 0 | exists,up | | 8 | cc3 | 1785M | 117G | 0 | 0 | 0 | 0 | exists,up | | 9 | cc3 | 1957M | 117G | 0 | 0 | 0 | 0 | exists,up | | 10 | cc3 | 1093M | 135G | 0 | 0 | 0 | 0 | exists,up | | 11 | cc3 | 1094M | 135G | 0 | 0 | 0 | 0 | exists,up | | 12 | compute01 | 1463M | 56.5G | 0 | 0 | 0 | 0 | exists,up | | 13 | compute01 | 1402M | 56.6G | 0 | 0 | 0 | 0 | exists,up | | 14 | compute01 | 1336M | 56.7G | 0 | 0 | 0 | 0 | exists,up | | 15 | compute01 | 1427M | 56.6G | 0 | 0 | 0 | 0 | exists,up | | 16 | compute01 | 1106M | 464G | 0 | 0 | 0 | 0 | exists,up | | 17 | compute01 | 1094M | 464G | 0 | 0 | 0 | 0 | exists,up | +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ ``` ### Check repair make sure services are ok {#check-repair-make-sure-services-are-ok} ```bash cc3> cluster cc3:cluster> check_repair Service Status Report ClusterLink ok [ link(v) clock(v) dns(v) ] ClusterSys ok [ bootstrap(v) license(v) ] ClusterSettings ok [ etcd(v) ] HaCluster FIXING [ hacluster(3) ] ok [ hacluster(f) ] MsgQueue ok [ rabbitmq(v) ] IaasDb ok [ mysql(v) ] VirtualIp ok [ vip(v) haproxy_ha(v) ] Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ] ApiService ok [ haproxy(v) httpd(v) lmi(v) memcache(v) ] SingleSignOn ok [ keycloak(v) ] Compute ok [ nova(v) ] Baremetal ok [ ironic(v) ] Network ok [ neutron(v) ] Image ok [ glance(v) ] BlockStor ok [ cinder(v) ] FileStor ok [ manila(v) ] ObjectStor ok [ swift(v) ] Orchestration ok [ heat(v) ] LBaaS ok [ octavia(v) ] DNSaaS ok [ designate(v) ] K8SaaS ok [ k3s(v) rancher(v) ] InstanceHa ok [ masakari(v) ] DisasterRecovery ok [ freezer(v) ] BusinessLogic ok [ mistral(v) murano(v) cloudkitty(v) senlin(v) watcher(v) ] ApiManager ok [ tyk(v) redis(v) mongodb(v) ] DataPipe ok [ zookeeper(v) kafka(v) ] Metrics ok [ monasca(v) telegraf(v) grafana(v) ] LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) es(v) kibana(v) ] Notifications ok [ influxdb(v) kapacitor(v) ] cc3:cluster> ``` ### Backup storage01 policies {#backup-storage01-policies} From your local pc terminal ```bash scp -r root@storage01_:/etc/policies Downloads/storage01_policy ``` ### Shutdown storage01 {#shutdown-storage01} ```bash $ ssh admin@192.168.1.121 Warning: Permanently added '192.168.1.121' (ECDSA) to the list of known hosts. Password: Welcome to the Cube Appliance Enter "help" for a list of available commands storage01> shutdown Enter 'YES' to confirm: YES Connection to 192.168.1.121 closed by remote host. Connection to 192.168.1.121 closed. ``` ## Adding Storage Host {#adding-storage-host} Prepare a new node with CubeCOS installed ### Configuration {#configuration} Reconfigure a new storage01 node by following any options of the list below: - [Storage node installation](../../configuration/node_roles/storage) - Swift Deploy with existing policy (recommended) ### Connect to controller {#connect-to-controller-1} ```bash $ ssh admin@192.168.1.x Warning: Permanently added '192.168.1.x' (ECDSA) to the list of known hosts. Password: ``` ### Check & Repair services {#check--repair-services} ```bash cc1> cluster cc1:cluster> check_repair Service Status Report ClusterLink ok [ link(v) clock(v) dns(v) ] ClusterSys ok [ bootstrap(v) license(v) ] ClusterSettings ok [ etcd(v) ] HaCluster FIXING [ hacluster(3) ] ok [ hacluster(f) ] MsgQueue ok [ rabbitmq(v) ] IaasDb ok [ mysql(v) ] VirtualIp ok [ vip(v) haproxy_ha(v) ] Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ] ApiService ok [ haproxy(v) httpd(v) lmi(v) memcache(v) ] SingleSignOn ok [ keycloak(v) ] Compute ok [ nova(v) ] Baremetal ok [ ironic(v) ] Network ok [ neutron(v) ] Image ok [ glance(v) ] BlockStor ok [ cinder(v) ] FileStor ok [ manila(v) ] ObjectStor ok [ swift(v) ] Orchestration ok [ heat(v) ] LBaaS ok [ octavia(v) ] DNSaaS ok [ designate(v) ] K8SaaS ok [ k3s(v) rancher(v) ] InstanceHa ok [ masakari(v) ] DisasterRecovery ok [ freezer(v) ] BusinessLogic ok [ mistral(v) murano(v) cloudkitty(v) senlin(v) watcher(v) ] ApiManager ok [ tyk(v) redis(v) mongodb(v) ] DataPipe ok [ zookeeper(v) kafka(v) ] Metrics ok [ monasca(v) telegraf(v) grafana(v) ] LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) es(v) kibana(v) ] Notifications ok [ influxdb(v) kapacitor(v) ] cc1:cluster> ``` ### Connect to storage01 {#connect-to-storage01} ```bash $ ssh admin@192.168.1.121 Warning: Permanently added '192.168.1.121' (ECDSA) to the list of known hosts. Password: ``` ### Check the storage status {#check-the-storage-status-2} ```bash storage01> storage storage01:storage> status cluster: id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85 health: HEALTH_OK services: mon: 3 daemons, quorum cc1,cc2,cc3 mgr: cc1(active), standbys: cc2, cc3 mds: cephfs-1/1/1 up {0=cc1=up:active}, 2 up:standby osd: 24 osds: 24 up, 24 in rgw: 3 daemons active data: pools: 23 pools, 1837 pgs objects: 10.50k objects, 12.7GiB usage: 31.6GiB used, 3.74TiB / 3.77TiB avail pgs: 1837 active+clean io: client: 42.3KiB/s rd, 49op/s rd, 0op/s wr +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | cc1 | 2063M | 117G | 0 | 0 | 0 | 0 | exists,up | | 1 | cc1 | 2036M | 117G | 0 | 0 | 0 | 0 | exists,up | | 2 | cc1 | 1088M | 135G | 0 | 0 | 0 | 0 | exists,up | | 3 | cc1 | 1088M | 135G | 0 | 0 | 0 | 0 | exists,up | | 4 | cc2 | 1663M | 117G | 0 | 0 | 0 | 0 | exists,up | | 5 | cc2 | 2080M | 116G | 0 | 0 | 0 | 0 | exists,up | | 6 | cc2 | 1096M | 135G | 0 | 0 | 0 | 0 | exists,up | | 7 | cc2 | 1096M | 135G | 0 | 0 | 4 | 0 | exists,up | | 8 | cc3 | 1788M | 117G | 0 | 0 | 0 | 0 | exists,up | | 9 | cc3 | 1952M | 117G | 0 | 0 | 0 | 0 | exists,up | | 10 | cc3 | 1096M | 135G | 0 | 0 | 0 | 0 | exists,up | | 11 | cc3 | 1096M | 135G | 0 | 0 | 0 | 0 | exists,up | | 12 | compute01 | 1464M | 56.5G | 0 | 0 | 0 | 0 | exists,up | | 13 | compute01 | 1403M | 56.6G | 0 | 0 | 0 | 0 | exists,up | | 14 | compute01 | 1337M | 56.7G | 0 | 0 | 0 | 0 | exists,up | | 15 | compute01 | 1428M | 56.6G | 0 | 0 | 0 | 0 | exists,up | | 16 | compute01 | 1109M | 464G | 0 | 0 | 0 | 0 | exists,up | | 17 | compute01 | 1096M | 464G | 0 | 0 | 0 | 0 | exists,up | | 18 | storage01 | 1042M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 19 | storage01 | 1042M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 20 | storage01 | 1042M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 21 | storage01 | 1042M | 57.0G | 0 | 0 | 0 | 0 | exists,up | | 22 | storage01 | 1096M | 464G | 0 | 0 | 0 | 0 | exists,up | | 23 | storage01 | 1112M | 464G | 0 | 0 | 0 | 0 | exists,up | +----+-----------+-------+-------+--------+---------+--------+---------+-----------+ ```