Version: 3.0

Restore Control Node from snapshot

info

USB storage devices are blocked by default. If accessing USB devices is needed, please set the tuning option cubesys.probeusb to true using CLI > tuning > configure after logging in as admin.

Use this procedure to recover a CubeCOS control node by applying a previously saved snapshot.

Targeting Roles

This guide apply to roles:

control
control-converged
edge-core
moderator

[Control-Converged] Remove Disk

This action is required if only the role node is control-converged, we need to remove the OSD from the storage pool when the node is removed.

cc3> storage
cc3:storage> remove_disk
  index          name      size     osd              serial
--
      1      /dev/sdb    447.1G     8 6  PHYM809401PY480BGN
      2      /dev/sdc    894.3G   12 10  BTYF91230FJX960CGN
      3      /dev/sdd    447.1G   16 14  PHYM809401LJ480BGN
--
Enter the index of disk to be removed: 1
Disk removal mode:
1: safe
2: force
Enter index: 2
force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS.
Enter 'YES' to confirm: YES
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META    AVAIL    %USE  VAR   PGS  STATUS
 6    ssd  0.21790   1.00000  223 GiB  534 MiB  521 MiB    8 KiB  13 MiB  223 GiB  0.23  1.00  110      up
                       TOTAL  223 GiB  534 MiB  521 MiB  8.2 KiB  13 MiB  223 GiB  0.23
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
AID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META    AVAIL    %USE  VAR   PGS  STATUS
 8    ssd  0.21790   1.00000  223 GiB  285 MiB  265 MiB    3 KiB  20 MiB  223 GiB  0.12  1.00  126      up
                       TOTAL  223 GiB  285 MiB  265 MiB  3.3 KiB  20 MiB  223 GiB  0.12
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
Removed disk /dev/sdb.

cc3:storage> remove_disk
  index          name      size     osd              serial
--
      1      /dev/sdc    894.3G   12 10  BTYF91230FJX960CGN
      2      /dev/sdd    447.1G   16 14  PHYM809401LJ480BGN
--
Enter the index of disk to be removed: 1
Disk removal mode:
1: safe
2: force
Enter index: 2
force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS.
Enter 'YES' to confirm: YES
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP    META    AVAIL    %USE  VAR   PGS  STATUS
10    ssd  0.43629   1.00000  447 GiB  768 MiB  739 MiB  11 KiB  29 MiB  446 GiB  0.17  1.00  304      up
                       TOTAL  447 GiB  768 MiB  739 MiB  12 KiB  29 MiB  446 GiB  0.17
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META    AVAIL    %USE  VAR   PGS  STATUS
12    ssd  0.43629   1.00000  447 GiB  1.0 GiB  994 MiB    6 KiB  35 MiB  446 GiB  0.22  1.00  411      up
                       TOTAL  447 GiB  1.0 GiB  994 MiB  6.7 KiB  35 MiB  446 GiB  0.22
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
Removed disk /dev/sdc.

cc3:storage> remove_disk
  index          name      size     osd              serial
--
      1      /dev/sdd    447.1G   16 14  PHYM809401LJ480BGN
--
Enter the index of disk to be removed: 1
Disk removal mode:
1: safe
2: force
Enter index: 2
force mode immediately destroys disk data so USE IT AT YOUR OWN RISKS.
Enter 'YES' to confirm: YES
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META    AVAIL    %USE  VAR   PGS  STATUS
14    ssd  0.21790   1.00000  223 GiB  1.3 GiB  1.2 GiB    5 KiB  31 MiB  222 GiB  0.57  1.00  449      up
                       TOTAL  223 GiB  1.3 GiB  1.2 GiB  5.1 KiB  31 MiB  222 GiB  0.57
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META    AVAIL    %USE  VAR   PGS  STATUS
16    ssd  0.21790   1.00000  223 GiB  1.1 GiB  1.1 GiB    5 KiB  25 MiB  222 GiB  0.51  1.00  810      up
                       TOTAL  223 GiB  1.1 GiB  1.1 GiB  5.7 KiB  25 MiB  222 GiB  0.51
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
Removed disk /dev/sdd.
cc3:storage>

Replace a New Node

Choose `Advanced` option

First Time Setup Options:
1: Wizard
2: Advanced
Enter index: 2

Additional Options for Control Node

Welcome to the Cube Appliance
Enter "help" for a list of available commands
unconfigured> first
unconfigured:first> control_rejoin
Set or clear control rejoin flag?
1: set
2: clear
Enter index: 1
Control rejoin markers set
unconfigured:first>

Pull Snapshot from Media by NFS

unconfigured> snapshot
unconfigured:snapshot> pull
Select a media:
1: usb
2: nfs
3: url
Enter index: 2

---------------------------------------------------------------------------------
 Dev   Label    DevID/Cfg      BusID/Slaves   HWAddr            Driver  Link State  Speed(actual/advertised)
---------------------------------------------------------------------------------
 eth0  IF.1     8086.1533      0000:01:00.0   c4:00:ad:54:ba:c3 igb     yes  UP     1000F/1000F
 eth1  IF.2     8086.1533      0000:02:00.0   c4:00:ad:54:ba:c4 igb     no   DOWN   /1000
 eth2  IF.3     8086.1533      0000:05:00.0   c4:00:ad:54:ba:c5 igb     no   DOWN   /1000
 eth3  IF.4     8086.374a      0000:3e:00.0   c4:00:ad:54:ba:c6 i40e    yes  DOWN   10000F/10000F
 eth4  IF.5     8086.374a      0000:3e:00.1   c4:00:ad:54:ba:c7 i40e    no   DOWN   /10000
 eth5  IF.6     8086.374a      0000:3e:00.2   c4:00:ad:54:ba:c8 i40e    no   DOWN   /10000
 eth6  IF.7     8086.374a      0000:3e:00.3   c4:00:ad:54:ba:c9 i40e    no   DOWN   /10000
---------------------------------------------------------------------------------

Select an interface:
1: eth0
2: eth1
3: eth2
4: eth3
5: eth4
6: eth5
7: eth6
Enter index: 1
Setting up network ...
Enter NFS location [server:/path]: 10.32.xxx.xxx:/volume1/cube-snaps/sqa_test
Select a snapshot:
1: (Automatically generated on 2025-09-11 10:50:12) CUBE_3.0.0_20250911-105012.687656_cc1.snapshot
2: (Automatically generated on 2025-09-11 11:17:21) CUBE_3.0.0_20250911-111721.852512_cc3.snapshot
Enter index: 2
Copying...
CUBE_3.0.0_20250911-111721.852512_cc3.snapshot
    626,933 100% 141.66MB/s 0:00:00 (xfr#1, to-chk=0/1)
Automatically generated on 2025-09-11 11:17:21
Copy complete.
unconfigured:snapshot>

Apply the Setting

unconfigured:snapshot> apply
1: CUBE_3.0.0_20250911-043159.452589_unconfigured.snapshot
2: CUBE_3.0.0_20250911-111721.852512_cc3.snapshot
Enter index: 2
Automatically generated on 2025-09-11 11:17:21
Date/Time is important for applying changes to an unconfigured box.
Please confirm the current time is good.

* Local Time: 09/11/2025 00:39:16 EDT

Enter 'YES' to confirm: YES
(01/22) committing: sys
(02/22) committing: appliance cron cubesys debug syslog update
(03/22) committing: first
(04/22) committing: net time bootstrap fixpack iso password snapshot snapshotmgr support supportbase usb volume-meta
(05/22) committing: net_static net_dynamic
(06/22) committing: sshd
(07/22) committing: standalone
(08/22) committing: cube_scan
(09/22) committing: corosync mysql cluster libvirt4 dns mongodb kafka ntp opensearch prometheus
(10/22) committing: pacemaker rabbitmq docker logstash opensearch-dashboards telegraf
(11/22) committing: haproxy k3s
(12/22) committing: keystone keycloak ceph
(13/22) committing: memcache ceph_dashboard idp keystone_idp rancher
(14/22) committing: barbican cinder glance heat horizon neutron magnum manila masakari monasca swift influxdb designate skyline octavia s
(15/22) committing: manila cyborg grafana ironic k3s_last kapacitor nginx
(16/22) committing: apache2
(17/22) committing: api
(18/22) committing: pacemaker_last
(19/22) committing: monasca_setup neutron_last
(20/22) committing: cube_last
(21/22) committing: done

Policy snapshot file applied

cc3:snapshot> exit
cc3 Login: admin
Password:
Welcome to the Cube Appliance
Enter "help" for a list of available commands
Notice: your license will expire in 29 days.
        Please contact system administrator to renew the license.
cc3> management set_password
Enter old password:
Enter new password:
Confirm new password:
Password successfully changed
cc3> exit

Enable Manual Boot

cc1:boot_mode> manual
Switch to manual bootstrap mode:
1: one-time manual
2: always manual
Enter index: 1
Enter 'YES' to confirm: YES
Exit current CLI and log in again as admin to see changes in effect

Cluster Sync

cc3> boot cluster_sync
(1/11) processing: keycloak
(2/11) processing: libvirt4
(3/11) processing: keystone
(4/11) processing: ceph
(5/11) processing: nova
(6/11) processing: cinder
(7/11) processing: manila
(8/11) processing: octavia
(9/11) processing: haproxy
(10/11) processing: k3s_last
(11/11) processing: telegraf
cluster_sync successfully
cc3>

Change Boot mode by backdoor

cc3 login: root
Password:
Last login: Thu Sep 11 13:18:12 from 10.32.140.1
Welcome to CUBE_3.0.0_20250608-0804_ef9b72f
[cc3 ~]# rm -f /etc/appliance/state/boot_mode
[cc3 ~]# exit

Check and Repair Services

cc3> cluster check
          Service  Status  Report
           IaasDb      ok  [ mysql(v) ]
        Baremetal      ok  [ ironic(v) ]
           K8SaaS      ok  [ rancher(v) ]
            Image      ok  [ glance(v) ]
       ClusterSys      ok  [ bootstrap(v) license(v) ]
     SingleSignOn      ok  [ k3s(v) keycloak(v) ]
         FileStor      ok  [ manila(v) ]
       ObjectStor      ok  [ swift(v) ]
      ClusterLink      ok  [ link(v) clock(v) dns(v) ]
        BlockStor      ok  [ cinder(v) ]
    Orchestration      ok  [ heat(v) ]
         MsgQueue      ok  [ rabbitmq(v) ]
    Notifications      ok  [ influxdb(v) kapacitor(v) ]
        VirtualIp      ok  [ vip(v) haproxy_ha(v) ]
        HaCluster      ok  [ hacluster(v) ]
         DataPipe      ok  [ zookeeper(v) kafka(v) ]
       InstanceHa      ok  [ masakari(v) ]
            LBaaS      ok  [ octavia(v) ]
    BusinessLogic      ok  [ senlin(v) watcher(v) ]
          Compute      ok  [ nova(v) cyborg(v) ]
          Metrics      ok  [ monasca(v) telegraf(v) grafana(v) ]
  ClusterSettings      ok  [ etcd(v) nodelist(v) mongodb(v) ]
          Storage      ok  [ ceph(v) ceph_mon(v) ceph_mgr(v) ceph_mds(v) ceph_osd(v) ceph_rgw(v) rbd_target(v) ]
     LogAnalytics      ok  [ filebeat(v) auditbeat(v) logstash(v) opensearch(v) opensearch-dashboards(v) ]
          Network      ok  [ neutron(v) ]
           DNSaaS      ok  [ designate(v) ]
       ApiService      ok  [ haproxy(v) httpd(v) nginx(v) api(v) skyline(v) memcache(v) ]
cc3>

Targeting Roles​

[Control-Converged] Remove Disk​

Replace a New Node​

Choose Advanced option​

Additional Options for Control Node​

Pull Snapshot from Media by NFS​

Apply the Setting​

Re-Login as Admin and set password​

Enable Manual Boot​

Cluster Sync​

Change Boot mode by backdoor​

Check and Repair Services​