# Restore CubeCOS from a snapshot > Learn how to restore CubeCOS from a snapshot to recover a failed node. :::info USB storage devices are blocked by default. To enable USB access, set the tuning option `cubesys.probeusb` to `true` using the CLI: `CLI > tuning > configure` ::: Use this procedure to restore the CubeCOS platform to a previously saved state using a system snapshot. ## Back up CubeCOS policies on the existing node {#back-up-cubecos-policies-on-the-existing-node} ### Cluster check {#cluster-check} ```bash cc1> cluster check Service Status Report IaasDb ok [ mysql(v) ] Baremetal ok [ ironic(v) ] K8SaaS ok [ rancher(v) ] Image ok [ glance(v) ] ClusterSys ok [ bootstrap(v) license(v) ] SingleSignOn ok [ k3s(v) keycloak(v) ] FileStor ok [ manila(v) ] ObjectStor ok [ swift(v) ] ClusterLink ok [ link(v) clock(v) dns(v) ] BlockStor ok [ cinder(v) ] Orchestration ok [ heat(v) ] MsgQueue ok [ rabbitmq(v) ] Notifications ok [ influxdb(v) kapacitor(v) ] VirtualIp ok [ vip(v) haproxy_ha(v) ] HaCluster ok [ hacluster(v) ] DataPipe ok [ zookeeper(v) kafka(v) ] InstanceHa ok [ masakari(v) ] LBaaS ok [ octavia(v) ] BusinessLogic ok [ senlin(v) watcher(v) ] Compute ok [ nova(v) cyborg(v) ] Metrics ok [ monasca(v) telegraf(v) grafana(v) ] ClusterSettings ok [ etcd(v) nodelist(v) mongodb(v) ] Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ] LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) opensearch(v) opensearch-dashboards(v) ] Network ok [ neutron(v) ] DNSaaS ok [ designate(v) ] ApiService ok [ haproxy(v) httpd(v) nginx(v) api(v) skyline(v) memcache(v) ] cc1> ``` ### [Optional] Remove disk {#optional-remove-disk} This action is required if only the node is included in the storage. ```bash cc1> storage cc1:storage> cc1:storage> cc1:storage> remove_disk index name size osd serial -- 1 /dev/sda 894.3G 1 0 BTYF91230BVA960CGN 2 /dev/sdb 447.1G 2 3 PHYM809401PZ480BGN 3 /dev/sdd 447.1G 4 5 183222E1E59A -- Enter the index of disk to be removed: 1 Disk removal mode: 1: safe 2: force Enter index: 2 force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS. Enter 'YES' to confirm: YES ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 ssd 0.43629 1.00000 447 GiB 631 MiB 578 MiB 9 KiB 53 MiB 446 GiB 0.14 1.00 221 up TOTAL 447 GiB 631 MiB 578 MiB 9.1 KiB 53 MiB 446 GiB 0.14 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 ssd 0.43629 1.00000 447 GiB 806 MiB 741 MiB 9 KiB 64 MiB 446 GiB 0.18 1.00 310 up TOTAL 447 GiB 806 MiB 741 MiB 9.7 KiB 64 MiB 446 GiB 0.18 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Removed disk /dev/sda. cc1:storage> remove_disk index name size osd serial -- 1 /dev/sdb 447.1G 2 3 PHYM809401PZ480BGN 2 /dev/sdd 447.1G 4 5 183222E1E59A -- Enter the index of disk to be removed: 1 Disk removal mode: 1: safe 2: force Enter index: 2 force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS. Enter 'YES' to confirm: YES ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 ssd 0.21790 1.00000 223 GiB 433 MiB 407 MiB 4 KiB 26 MiB 223 GiB 0.19 1.00 216 up TOTAL 223 GiB 433 MiB 407 MiB 4.9 KiB 26 MiB 223 GiB 0.19 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 3 ssd 0.21790 1.00000 223 GiB 515 MiB 489 MiB 3 KiB 26 MiB 223 GiB 0.23 1.00 295 up TOTAL 223 GiB 515 MiB 489 MiB 3.3 KiB 26 MiB 223 GiB 0.23 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Removed disk /dev/sdb. cc1:storage> remove_disk index name size osd serial -- 1 /dev/sdd 447.1G 4 5 183222E1E59A -- Enter the index of disk to be removed: 1 Disk removal mode: 1: safe 2: force Enter index: 2 force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS. Enter 'YES' to confirm: YES ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 4 ssd 0.21790 1.00000 223 GiB 1.1 GiB 1.0 GiB 5 KiB 40 MiB 222 GiB 0.48 1.00 466 up TOTAL 223 GiB 1.1 GiB 1.0 GiB 5.9 KiB 40 MiB 222 GiB 0.48 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 5 ssd 0.21790 1.00000 223 GiB 1.3 GiB 1.2 GiB 8 KiB 35 MiB 222 GiB 0.57 1.00 820 up TOTAL 223 GiB 1.3 GiB 1.2 GiB 8.3 KiB 35 MiB 222 GiB 0.57 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Removed disk /dev/sdd. cc1:storage> ``` ### [Optional] Check status {#optional-check-status} Ensure all associated OSDs are removed, remove any remaining ones. ```bash cc1:storage> status cluster: id: c6e64c49-09cf-463b-9d1c-b6645b4b3b85 health: HEALTH_WARN Degraded data redundancy: 773/2748 objects degraded (28.130%), 124 pgs degraded, 796 pgs undersized services: mon: 3 daemons, quorum cc1,cc3,cc2 (age 2h) mgr: cc1(active, since 15h), standbys: cc2, cc3 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 12 osds: 12 up (since 103s), 12 in (since 104s); 85 remapped pgs rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 25 pools, 945 pgs objects: 916 objects, 2.2 GiB usage: 5.4 GiB used, 3.5 TiB / 3.5 TiB avail pgs: 773/2748 objects degraded (28.130%) 143/2748 objects misplaced (5.204%) 672 active+undersized 124 active+undersized+degraded 85 active+clean+remapped 64 active+clean io: client: 767 B/s rd, 1 op/s rd, 0 op/s wr ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 6 cc3 507M 222G 0 0 0 0 exists,up 7 cc2 643M 222G 0 0 0 0 exists,up 8 cc3 163M 223G 0 0 0 0 exists,up 9 cc2 404M 222G 0 0 0 0 exists,up 10 cc3 513M 222G 0 0 1 0 exists,up 11 cc2 539M 446G 0 0 1 90 exists,up 12 cc3 468M 222G 0 0 0 0 exists,up 13 cc2 819M 445G 0 0 0 0 exists,up 14 cc3 428M 446G 0 0 1 16 exists,up 15 cc2 229M 222G 0 0 1 0 exists,up 16 cc3 547M 446G 0 0 0 0 exists,up 17 cc2 242M 222G 0 0 0 0 exists,up cc1:storage> ``` ## Replace a new node {#replace-a-new-node} ### Choose `Advanced` option {#choose-advanced-option} ```bash First Time Setup Options: 1: Wizard 2: Advanced Enter index: 2 ``` ### [Optional] Additional options for control node {#optional-additional-options-for-control-node} ```bash Welcome to the Cube Appliance Enter "help" for a list of available commands unconfigured> first unconfigured:first> control_rejoin Set or clear control rejoin flag? 1: set 2: clear Enter index: 1 Control rejoin markers set ``` ### Pull snapshot from external media via NFS {#pull-snapshot-from-external-media-via-nfs} ```bash unconfigured:snapshot> pull Select a media: 1: usb 2: nfs 3: url Enter index: 2 ------------------------------------------------------------ Dev Label DevID/Cfg BusID/Slaves HWAddr Driver Link State Speed(actual/advertised) ------------------------------------------------------------ eth0 IF.1 8086.1533 0000:01:00.0 c4:00:ad:54:b9:68 igb yes UP 1000F/1000F eth1 IF.2 8086.1533 0000:02:00.0 c4:00:ad:54:b9:69 igb no DOWN /1000 eth2 IF.3 8086.1533 0000:05:00.0 c4:00:ad:54:b9:6a igb no DOWN /1000 eth3 IF.4 8086.374d 0000:3e:00.0 c4:00:ad:54:b9:6b i40e yes DOWN 10000F/10000F eth4 IF.5 8086.374d 0000:3e:00.1 c4:00:ad:54:b9:6c i40e no DOWN /10000 eth5 IF.6 8086.374d 0000:3e:00.2 c4:00:ad:54:b9:6d i40e no DOWN /10000 eth6 IF.7 8086.374d 0000:3e:00.3 c4:00:ad:54:b9:6e i40e no DOWN /10000 ------------------------------------------------------------ Select an interface: 1: eth0 2: eth1 3: eth2 4: eth3 5: eth4 6: eth5 7: eth6 Enter index: 1 Setting up network ... Enter NFS location [server:/path]: 10.32.xxx.xxx:volume1/cube-snaps/sqa_test Select a snapshot: 1: (Automatically generated on 2025-09-11 10:50:12) CUBE_3.0.0_20250911-105012.687656_cc1.snapshot 2: (Automatically generated on 2025-09-11 11:17:21) CUBE_3.0.0_20250911-111721.852512_cc3.snapshot 3: (Automatically generated on 2025-09-11 12:24:44) CUBE_3.0.0_20250911-122444.561617_cc1.snapshot Enter index: 1 Copying... CUBE_3.0.0_20250911-105012.687656_cc1.snapshot 626,698 100% 29.81MB/s 0:00:00 (xfr#1, to-chk=0/1) Automatically generated on 2025-09-11 10:50:12 Copy complete. unconfigured:snapshot> ``` ### Apply the Setting {#apply-the-setting} ```bash unconfigured:snapshot> apply 1: CUBE_3.0.0_20250911-072751.480371_unconfigured.snapshot 2: CUBE_3.0.0_20250911-105012.687656_cc1.snapshot Enter index: 2 Automatically generated on 2025-09-11 10:50:12 Date/Time is important for applying changes to an unconfigured box. Please confirm the current time is good. * Local Time: 09/11/2025 04:23:06 EDT Enter 'YES' to confirm: YES (01/22) committing: sys (02/22) committing: appliance cron cubesys debug syslog update (03/22) committing: first (04/22) committing: net time bootstrap fixpack iso password snapshot snapshotmgr support supportbase usb volume-meta (05/22) committing: net_static net_dynamic (06/22) committing: sshd (07/22) committing: standalone (08/22) committing: cube_scan (09/22) committing: corosync mysql cluster libvirt4 dns mongodb kafka ntp opensearch prometheus MongoServerError: no replset config has been received (10/22) committing: pacemaker rabbitmq docker logstash opensearch-dashboards telegraf (11/22) committing: haproxy k3s (12/22) committing: keystone keycloak ceph (13/22) committing: memcache ceph_dashboard_idp keystone_idp rancher (14/22) committing: barbican cinder glance heat horizon neutron nova masakari monasca swift influxdb designate skyline octavia s (15/22) committing: manila cyborg grafana ironic k3s_last kapacitor nginx (16/22) committing: apache2 (17/22) committing: api (18/22) committing: pacemaker_last (19/22) committing: last (20/22) committing: monasca_setup neutron_last (21/22) committing: cube_last (22/22) committing: done Policy snapshot file applied cc1:snapshot> ``` ### Re-Login as Admin and set password {#re-login-as-admin-and-set-password} ```bash cc1:snapshot> exit cc1 Login: admin Password: Welcome to the Cube Appliance Enter "help" for a list of available commands Notice: your license will expire in 29 days. Please contact system administrator to renew the license. cc1> management set_password Enter old password: Enter new password: Confirm new password: Password successfully changed ``` ### Check and Repair Services {#check-and-repair-services} ```bash cc1> cluster check Service Status Report IaasDb ok [ mysql(v) ] Baremetal ok [ ironic(v) ] K8SaaS ok [ rancher(v) ] Image ok [ glance(v) ] ClusterSys ok [ bootstrap(v) license(v) ] SingleSignOn ok [ k3s(v) keycloak(v) ] FileStor ok [ manila(v) ] ObjectStor ok [ swift(v) ] ClusterLink ok [ link(v) clock(v) dns(v) ] BlockStor ok [ cinder(v) ] Orchestration ok [ heat(v) ] MsgQueue ok [ rabbitmq(v) ] Notifications ok [ influxdb(v) kapacitor(v) ] VirtualIp ok [ vip(v) haproxy_ha(v) ] HaCluster ok [ hacluster(v) ] DataPipe ok [ zookeeper(v) kafka(v) ] InstanceHa ok [ masakari(v) ] LBaaS ok [ octavia(v) ] BusinessLogic ok [ senlin(v) watcher(v) ] Compute ok [ nova(v) cyborg(v) ] Metrics ok [ monasca(v) telegraf(v) grafana(v) ] ClusterSettings ok [ etcd(v) nodelist(v) mongodb(v) ] Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ] LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) opensearch(v) opensearch-dashboards(v) ] Network ok [ neutron(v) ] DNSaaS ok [ designate(v) ] ApiService ok [ haproxy(v) httpd(v) nginx(v) api(v) skyline(v) memcache(v) ] cc1> ```