# Perform manual upgrades via the CLI

> Use the CLI to manually upgrade CubeCOS firmware and fix packs. This includes prerequisite checks, step-by-step commands and verification procedures.

Manual upgrade mode lets you control CubeCOS upgrades directly through the command line. With this approach, you log in to nodes remotely and run system commands to manage the update process.

Use manual upgrade only if:

- One-click upgrades have failed
- You need precise control over specific nodes
- You're running CubeCOS version 3.0.0

:::note

If you're running CubeCOS version 3.1.0 or later, try the [one-click upgrade](./introduction) first.

:::

## Prerequisites {#prerequisites}

Before you start, review the [prerequisites](./introduction#prerequisites) and [upgrade workflow](./introduction#upgrade-workflow) sections in the one-click upgrade guide.

<!--

## Update fixpacks {#update-fixpacks}

### Connect to your cluster {#connect-to-your-cluster}

Open a terminal and connect to your CubeCOS cluster via SSH:

```bash
ssh <your-username>@<your-cluster-ip>
```

Parameters:

- `your-username`: The username of your CubeCOS cluster.
- `your-cluster-ip`: The IP address of your CubeCOS cluster.

3. When prompted, enter your SSH password.

### Prepare for the update {#prepare-for-the-update}

Before you update, verify your cluster health and upload the necessary files.

#### Check cluster health {#check-cluster-health}

Before starting the update, check the health status of your cluster to ensure all services are running properly.

1. Enter the cluster management mode.

   ```bash
   cluster
   ```

2. Check cluster health:

   ```bash
   check
   ```

3. Verify the output shows `ok` for all services. Example:

   ```bash
             Service  Status  Report
          IaasDb      ok  [ mysql(v) ]
       Baremetal      ok  [ ironic(v) ]
          K8SaaS      ok  [ rancher(v) ]
      ObjectStor      ok  [ swift(v) ]
   ... # output abbreviated
   ```

#### Verify all nodes are healthy {#verify-all-nodes-are-healthy}

1. Open the CubeCOS UI.
2. Go to **Home** > **Nodes**.
3. Confirm all nodes show a healthy status.

#### Upload fixpack files {#upload-fixpack-files}

1. From your local machine, upload the fixpack and MD5 checksum files to the `/var/update/` directory:

   ```bash
   scp <local-file-path> <your-username>@<your-cluster-ip>:/var/update/
   ```

   Replace:
   - `<local-file-path>`: Local path to the fixpack or checksum file
   - `<your-username>`: Your cluster username
   - `<your-cluster-ip>`: Your cluster IP address

2. When prompted, enter your SSH password.

#### Verify uploaded files {#verify-uploaded-files}

1. Enter update management mode:

   ```bash
   update
   ```

2. List uploaded fixpacks:

   ```bash
   list
   ```

3. When prompted, select **local**.

### Install the fixpack {#install-the-fixpack}

1. Enter fixpack management mode:

   ```bash
   fixpack
   ```

2. Install the fixpack:

   ```bash
   install
   ```

3. When prompted, select **local** to specify where your uploaded files are located.

4. If the fixpack supports rollback, you can revert to the previous version if needed:

   ```bash
   rollback
   ```

5. If the fixpack requires a reboot, run:
   ```bash
   reboot
   ```

### Verify the update {#verify-the-update}

After the upgrade, check the health status of your cluster to ensure all services are running properly.

1. Enter the cluster management mode.

   ```bash
   cluster
   ```

2. Check cluster health again:

   ```bash
   check
   ```

3. Verify all services show `ok` status.
-->

## Upload the firmware files {#upload-the-firmware-files}

Before you upgrade, verify your cluster health and upload the necessary files.

### Check cluster health {#check-cluster-health-1}

Before starting the upgrade, check the health status of your cluster to ensure all services are running properly.

1. Enter the cluster management mode.

   ```bash
   cluster
   ```

2. Check cluster health:

   ```bash
   check
   ```

3. Verify all services show `ok` status.

### Verify all nodes are healthy {#verify-all-nodes-are-healthy-1}

1. Open the CubeCOS UI.
2. Go to **Home** > **Nodes**.
3. Confirm all nodes show a healthy status.

### Upload firmware files {#upload-firmware-files}

1. From your local machine, upload the firmware and MD5 checksum files to the `/var/update/` directory:

   ```bash
   scp <local-file-path> <your-username>@<your-cluster-ip>:/var/update/
   ```

   Replace:
   - `<local-file-path>`: Local path to the firmware or checksum file
   - `<your-username>`: Your cluster username
   - `<your-cluster-ip>`: Your cluster IP address

2. When prompted, enter your SSH password.
3. Repeat the upload operation for all hosts in the cluster.

### Verify uploaded firmware files {#verify-uploaded-firmware-files}

Before starting the upgrade, make sure that all the firmware files have been uploaded to each host in the cluster.

1. Enter the update management mode.

   ```bash
   update
   ```

2. List uploaded firmware images:

   ```bash
   list
   ```

3. When prompted, select `local`. This option specifies the location of your uploaded files.

## Install the firmware {#install-the-firmware}

Install the firmware onto all the hosts in the cluster.

1. Enter the update management mode.

   ```bash
   update
   ```

2. Start the interactive install process.

   ```bash
   update
   ```

3. When prompted, select `local`. This option specifies the location of your uploaded files.
4. Select the image file that you want to install.
5. Enter `YES` to confirm the installation.
6. Wait for the installation process to return complete.

### Migrate VMs before reboot {#migrate-vms-before-reboot}

To ensure service continuity, migrate all virtual machines (VMs) from the node to other nodes before upgrading and rebooting.

:::danger

A cluster upgrade can affect performance, so make sure you perform this task during a maintenance window.

:::

Run the following command on the master node only.

1. Enter the IAAS management mode.

   ```bash
   iaas
   ```

2. Access compute management:

   ```bash
   compute
   ```

3. Evacuate VMs from the node:

   ```bash
   upgrade_host_evacuation
   ```

### Reboot the cluster {#reboot-the-cluster}

Firmware upgrades require a reboot.

:::warning[One time operation]

Only a manual migration is required on the master node of the cluster. Subsequent hosts will automatically migrate VMs and reboot sequentially.

:::

Run this command on the master node:

1. Start rolling reboot operation for the entire cluster by executing the following command on the master node.

   ```bash
   reboot
   ```

2. Once the master node has completed the bootstrap process, the second node automatically reboots.
3. Before the reboot, the VM is automatically migrated to other nodes. After the reboot, auto-bootstrap is triggered automatically.
4. This will be performed for all nodes in the cluster.

### Verify the upgrade {#verify-the-upgrade}

After the upgrade, check the health status of your cluster to ensure all services are running properly.

1. Enter the cluster management mode.

   ```bash
   cluster
   ```

2. Check the health status of the cluster.

   ```bash
   check
   ```

3. Ensure that the health status is `OK`. The following example shows the expected output:

   ```bash
             Service  Status  Report
          IaasDb      ok  [ mysql(v) ]
       Baremetal      ok  [ ironic(v) ]
          K8SaaS      ok  [ rancher(v) ]
      ObjectStor      ok  [ swift(v) ]
   ... # output abbreviated
   ```

#### Check firmware version on all nodes {#check-firmware-version-on-all-nodes}

After the upgrade, check the firmware version of each node matches the version you uploaded.

1. Enter the firmware management mode.

   ```bash
   firmware
   ```

2. Check the version on the active firmware partition.

   ```bash
   list
   ```

3. Ensure that the version of the active firmware matches the version you uploaded. The following example shows the expected output:

   ```bash
   1: CUBE_3.1.0_20260203-2300_be0cb70 [ACTIVE]
   Firmware Version: Cube Appliance 3.1.0
   Installation Date: Feb  5, 2026 11:33:20 AM
   Installation Type: Re-install USB
   Last Boot: Feb  5, 2026 11:46:38 AM
   Comment:
   2: CUBE_3.1.0_20260203-2300_be0cb70
   Firmware Version: Cube Appliance 3.1.0
   Installation Date: Feb  5, 2026 11:35:05 AM
   Installation Type: Re-install USB
   Last Boot: Never
   Comment:
   ```

### Check all nodes after upgrade {#check-all-nodes-after-upgrade}

Check the status of all nodes to ensure they are up and in a healthy state.

1. Open CubeCOS UI.
2. Navigate to **Home** > **Nodes**.
3. Check the status of all nodes from the node list.

### Check the status of all the VMs {#check-the-status-of-all-the-vms}

Check the status of all VMs to ensure they are all active.

1. Open skyline UI.
2. Navigate to **Home** > **Compute** > **Instances**.
3. Check the status of all VMs from the VM list.