# Resolving VM evacuation failures during CubeCOS rolling upgrades

> Learn how to resolve aborted CubeCOS rolling upgrades caused by VM evacuation failures. This is a step-by-step guide to manual migration and node reboot for version 3.1.0 or higher.

## Summary {#summary}

During a rolling upgrade, the first master control node reboots. Then, the rest of the upgrade processes are carried out automatically. However, timing issues or temporary service unavailability may prevent virtual machines (VMs) from being evacuated from their original node. If VMs fail to evacuate a node, the upgrade process aborts and requires manual intervention to resolve the issue.

## Details {#details}

Errors related to VM evacuation failures may occur during a rolling upgrade process due to temporary service unavailability of pertinent components or timing issues on the platform. In this case, the virtual machines must be evacuated manually, and the node must be rebooted before the automatic rolling upgrade process can continue.

### Applicable versions {#applicable-versions}

CubeCOS 3.1.0 and above.

This article applies to environments running CubeCOS version 3.1.0 or higher.

## Resolution {#resolution}

### Evacuate VM manually and reboot node to resume automatic rolling process {#evacuate-vm-manually-and-reboot-node-to-resume-automatic-rolling-process}

1. Open a terminal session.
2. Connect to the cluster VIP via SSH and log in as the `admin` user.

   ```bash
   ssh admin@<your-cluster-vip>
   ```

3. In the admin CLI, enter the following command to resolve the host evacuation failure:

   ```bash
   iaas compute pre_failure_host_evacuation
   ```

4. Enter the index of the failed node when prompted to evacuate the failed host.

   ```bash
   examplecubenode1> iaas compute pre_failure_host_evacuation
   Evacuate which compute node:
   1: examplecubenode1
   2: examplecubenode2
   3: examplecubenode3
   Enter index: 1
   ```

5. Verify the node and VM details, then type "YES" to confirm the evacuation.

   The CLI will then display the status and relevant information for all the VMs on the host that will be evacuated. Verify that the host and VM information are correct before entering "yes" to execute the evacuation.

   ```bash
   +------------------+----------------------+--------+--------------------------------+--------------------------+----------------+
   | ID               | Name                 | Status | Networks                       | Image                    | Flavor         |
   +------------------+----------------------+--------+--------------------------------+--------------------------+----------------+
   | some-id1         | cinder-volumes_nodeID| ACTIVE | \_internal-network=some-ip-info| N/A (booted from volume) | t2.diagnostics |
   +------------------+----------------------+--------+--------------------------------+--------------------------+----------------+
   Enter 'YES' to confirm: YES
   ```

6. To continue the automatic rolling upgrade process, reboot the node after evacuation has been performed.
   1. Connect to the cluster VIP via SSH and log in as the `admin` user: `ssh admin@<mgnt-ip-of-errored-node>`
   2. Enter `reboot` into the CLI to continue the automatic rolling-upgrade process and reboot the node.
7. Navigate to `Maintenance > Update`.
8. Verify that the automatic rolling upgrade continues without error and that the status of the failed node returns `Succeeded`.