# Restore Control Node from snapshot > :::info :::info USB storage devices are blocked by default. If accessing USB devices is needed, please set the tuning option `cubesys.probeusb` to `true` using `CLI > tuning > configure` after logging in as admin. ::: Use this procedure to recover a CubeCOS control node by applying a previously saved snapshot. ## Targeting Roles {#targeting-roles} This guide apply to roles: - control - control-converged - edge-core - moderator ### [Control-Converged] Remove Disk {#control-converged-remove-disk} This action is required if only the role node is control-converged, we need to remove the OSD from the storage pool when the node is removed. ```bash cc3> storage cc3:storage> remove_disk index name size osd serial -- 1 /dev/sdb 447.1G 8 6 PHYM809401PY480BGN 2 /dev/sdc 894.3G 12 10 BTYF91230FJX960CGN 3 /dev/sdd 447.1G 16 14 PHYM809401LJ480BGN -- Enter the index of disk to be removed: 1 Disk removal mode: 1: safe 2: force Enter index: 2 force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS. Enter 'YES' to confirm: YES ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 6 ssd 0.21790 1.00000 223 GiB 534 MiB 521 MiB 8 KiB 13 MiB 223 GiB 0.23 1.00 110 up TOTAL 223 GiB 534 MiB 521 MiB 8.2 KiB 13 MiB 223 GiB 0.23 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 AID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 8 ssd 0.21790 1.00000 223 GiB 285 MiB 265 MiB 3 KiB 20 MiB 223 GiB 0.12 1.00 126 up TOTAL 223 GiB 285 MiB 265 MiB 3.3 KiB 20 MiB 223 GiB 0.12 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Removed disk /dev/sdb. cc3:storage> remove_disk index name size osd serial -- 1 /dev/sdc 894.3G 12 10 BTYF91230FJX960CGN 2 /dev/sdd 447.1G 16 14 PHYM809401LJ480BGN -- Enter the index of disk to be removed: 1 Disk removal mode: 1: safe 2: force Enter index: 2 force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS. Enter 'YES' to confirm: YES ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 10 ssd 0.43629 1.00000 447 GiB 768 MiB 739 MiB 11 KiB 29 MiB 446 GiB 0.17 1.00 304 up TOTAL 447 GiB 768 MiB 739 MiB 12 KiB 29 MiB 446 GiB 0.17 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 12 ssd 0.43629 1.00000 447 GiB 1.0 GiB 994 MiB 6 KiB 35 MiB 446 GiB 0.22 1.00 411 up TOTAL 447 GiB 1.0 GiB 994 MiB 6.7 KiB 35 MiB 446 GiB 0.22 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Removed disk /dev/sdc. cc3:storage> remove_disk index name size osd serial -- 1 /dev/sdd 447.1G 16 14 PHYM809401LJ480BGN -- Enter the index of disk to be removed: 1 Disk removal mode: 1: safe 2: force Enter index: 2 force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS. Enter 'YES' to confirm: YES ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 14 ssd 0.21790 1.00000 223 GiB 1.3 GiB 1.2 GiB 5 KiB 31 MiB 222 GiB 0.57 1.00 449 up TOTAL 223 GiB 1.3 GiB 1.2 GiB 5.1 KiB 31 MiB 222 GiB 0.57 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21790 1.00000 223 GiB 1.1 GiB 1.1 GiB 5 KiB 25 MiB 222 GiB 0.51 1.00 810 up TOTAL 223 GiB 1.1 GiB 1.1 GiB 5.7 KiB 25 MiB 222 GiB 0.51 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Removed disk /dev/sdd. cc3:storage> ``` ## Replace a New Node {#replace-a-new-node} ### Choose `Advanced` option {#choose-advanced-option} ```bash First Time Setup Options: 1: Wizard 2: Advanced Enter index: 2 ``` ### Additional Options for Control Node {#additional-options-for-control-node} ```bash Welcome to the Cube Appliance Enter "help" for a list of available commands unconfigured> first unconfigured:first> control_rejoin Set or clear control rejoin flag? 1: set 2: clear Enter index: 1 Control rejoin markers set unconfigured:first> ``` ### Pull Snapshot from Media by NFS {#pull-snapshot-from-media-by-nfs} ```bash unconfigured> snapshot unconfigured:snapshot> pull Select a media: 1: usb 2: nfs 3: url Enter index: 2 --------------------------------------------------------------------------------- Dev Label DevID/Cfg BusID/Slaves HWAddr Driver Link State Speed(actual/advertised) --------------------------------------------------------------------------------- eth0 IF.1 8086.1533 0000:01:00.0 c4:00:ad:54:ba:c3 igb yes UP 1000F/1000F eth1 IF.2 8086.1533 0000:02:00.0 c4:00:ad:54:ba:c4 igb no DOWN /1000 eth2 IF.3 8086.1533 0000:05:00.0 c4:00:ad:54:ba:c5 igb no DOWN /1000 eth3 IF.4 8086.374a 0000:3e:00.0 c4:00:ad:54:ba:c6 i40e yes DOWN 10000F/10000F eth4 IF.5 8086.374a 0000:3e:00.1 c4:00:ad:54:ba:c7 i40e no DOWN /10000 eth5 IF.6 8086.374a 0000:3e:00.2 c4:00:ad:54:ba:c8 i40e no DOWN /10000 eth6 IF.7 8086.374a 0000:3e:00.3 c4:00:ad:54:ba:c9 i40e no DOWN /10000 --------------------------------------------------------------------------------- Select an interface: 1: eth0 2: eth1 3: eth2 4: eth3 5: eth4 6: eth5 7: eth6 Enter index: 1 Setting up network ... Enter NFS location [server:/path]: 10.32.xxx.xxx:/volume1/cube-snaps/sqa_test Select a snapshot: 1: (Automatically generated on 2025-09-11 10:50:12) CUBE_3.0.0_20250911-105012.687656_cc1.snapshot 2: (Automatically generated on 2025-09-11 11:17:21) CUBE_3.0.0_20250911-111721.852512_cc3.snapshot Enter index: 2 Copying... CUBE_3.0.0_20250911-111721.852512_cc3.snapshot 626,933 100% 141.66MB/s 0:00:00 (xfr#1, to-chk=0/1) Automatically generated on 2025-09-11 11:17:21 Copy complete. unconfigured:snapshot> ``` ### Apply the Setting {#apply-the-setting} ```bash unconfigured:snapshot> apply 1: CUBE_3.0.0_20250911-043159.452589_unconfigured.snapshot 2: CUBE_3.0.0_20250911-111721.852512_cc3.snapshot Enter index: 2 Automatically generated on 2025-09-11 11:17:21 Date/Time is important for applying changes to an unconfigured box. Please confirm the current time is good. * Local Time: 09/11/2025 00:39:16 EDT Enter 'YES' to confirm: YES (01/22) committing: sys (02/22) committing: appliance cron cubesys debug syslog update (03/22) committing: first (04/22) committing: net time bootstrap fixpack iso password snapshot snapshotmgr support supportbase usb volume-meta (05/22) committing: net_static net_dynamic (06/22) committing: sshd (07/22) committing: standalone (08/22) committing: cube_scan (09/22) committing: corosync mysql cluster libvirt4 dns mongodb kafka ntp opensearch prometheus (10/22) committing: pacemaker rabbitmq docker logstash opensearch-dashboards telegraf (11/22) committing: haproxy k3s (12/22) committing: keystone keycloak ceph (13/22) committing: memcache ceph_dashboard idp keystone_idp rancher (14/22) committing: barbican cinder glance heat horizon neutron magnum manila masakari monasca swift influxdb designate skyline octavia s (15/22) committing: manila cyborg grafana ironic k3s_last kapacitor nginx (16/22) committing: apache2 (17/22) committing: api (18/22) committing: pacemaker_last (19/22) committing: monasca_setup neutron_last (20/22) committing: cube_last (21/22) committing: done Policy snapshot file applied ``` ### Re-Login as Admin and set password {#re-login-as-admin-and-set-password} ```bash cc3:snapshot> exit cc3 Login: admin Password: Welcome to the Cube Appliance Enter "help" for a list of available commands Notice: your license will expire in 29 days. Please contact system administrator to renew the license. cc3> management set_password Enter old password: Enter new password: Confirm new password: Password successfully changed cc3> exit ``` ### Enable Manual Boot {#enable-manual-boot} ```bash cc1:boot_mode> manual Switch to manual bootstrap mode: 1: one-time manual 2: always manual Enter index: 1 Enter 'YES' to confirm: YES Exit current CLI and log in again as admin to see changes in effect ``` ### Cluster Sync {#cluster-sync} ```bash cc3> boot cluster_sync (1/11) processing: keycloak (2/11) processing: libvirt4 (3/11) processing: keystone (4/11) processing: ceph (5/11) processing: nova (6/11) processing: cinder (7/11) processing: manila (8/11) processing: octavia (9/11) processing: haproxy (10/11) processing: k3s_last (11/11) processing: telegraf cluster_sync successfully cc3> ``` ### Change Boot mode by backdoor {#change-boot-mode-by-backdoor} ```bash cc3 login: root Password: Last login: Thu Sep 11 13:18:12 from 10.32.140.1 Welcome to CUBE_3.0.0_20250608-0804_ef9b72f [cc3 ~]# rm -f /etc/appliance/state/boot_mode [cc3 ~]# exit ``` ### Check and Repair Services {#check-and-repair-services} ```bash cc3> cluster check Service Status Report IaasDb ok [ mysql(v) ] Baremetal ok [ ironic(v) ] K8SaaS ok [ rancher(v) ] Image ok [ glance(v) ] ClusterSys ok [ bootstrap(v) license(v) ] SingleSignOn ok [ k3s(v) keycloak(v) ] FileStor ok [ manila(v) ] ObjectStor ok [ swift(v) ] ClusterLink ok [ link(v) clock(v) dns(v) ] BlockStor ok [ cinder(v) ] Orchestration ok [ heat(v) ] MsgQueue ok [ rabbitmq(v) ] Notifications ok [ influxdb(v) kapacitor(v) ] VirtualIp ok [ vip(v) haproxy_ha(v) ] HaCluster ok [ hacluster(v) ] DataPipe ok [ zookeeper(v) kafka(v) ] InstanceHa ok [ masakari(v) ] LBaaS ok [ octavia(v) ] BusinessLogic ok [ senlin(v) watcher(v) ] Compute ok [ nova(v) cyborg(v) ] Metrics ok [ monasca(v) telegraf(v) grafana(v) ] ClusterSettings ok [ etcd(v) nodelist(v) mongodb(v) ] Storage ok [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ] LogAnalytics ok [ filebeat(v) auditbeat(v) logstash(v) opensearch(v) opensearch-dashboards(v) ] Network ok [ neutron(v) ] DNSaaS ok [ designate(v) ] ApiService ok [ haproxy(v) httpd(v) nginx(v) api(v) skyline(v) memcache(v) ] cc3> ```