Resolving VM evacuation failures during CubeCOS rolling upgrades

Summary

During a rolling upgrade, the first master control node reboots. Then, the rest of the upgrade processes are carried out automatically. However, timing issues or temporary service unavailability may prevent virtual machines (VMs) from being evacuated from their original node. If VMs fail to evacuate a node, the upgrade process aborts and requires manual intervention to resolve the issue.

Details

Errors related to VM evacuation failures may occur during a rolling upgrade process due to temporary service unavailability of pertinent components or timing issues on the platform. In this case, the virtual machines must be evacuated manually, and the node must be rebooted before the automatic rolling upgrade process can continue.

Applicable versions

CubeCOS 3.1.0 and above.

This article applies to environments running CubeCOS version 3.1.0 or higher.

Resolution

Evacuate VM manually and reboot node to resume automatic rolling process

1. Open a terminal session.

2. Connect to the cluster VIP via SSH and log in as the admin user.

   ssh admin@<your-cluster-vip>

3. In the admin CLI, enter the following command to resolve the host evacuation failure:

   iaas compute pre_failure_host_evacuation

4. Enter the index of the failed node when prompted to evacuate the failed host.

   examplecubenode1> iaas compute pre_failure_host_evacuation
   Evacuate which compute node:
   1: examplecubenode1
   2: examplecubenode2
   3: examplecubenode3
   Enter index: 1

5. Verify the node and VM details, then type "YES" to confirm the evacuation.

   The CLI will then display the status and relevant information for all the VMs on the host that will be evacuated. Verify that the host and VM information are correct before entering "yes" to execute the evacuation.

   +------------------+----------------------+--------+--------------------------------+--------------------------+----------------+
   | ID               | Name                 | Status | Networks                       | Image                    | Flavor         |
   +------------------+----------------------+--------+--------------------------------+--------------------------+----------------+
   | some-id1         | cinder-volumes_nodeID| ACTIVE | \_internal-network=some-ip-info| N/A (booted from volume) | t2.diagnostics |
   +------------------+----------------------+--------+--------------------------------+--------------------------+----------------+
   Enter 'YES' to confirm: YES

6. To continue the automatic rolling upgrade process, reboot the node after evacuation has been performed.

   1. Connect to the cluster VIP via SSH and log in as the admin user: ssh admin@<mgnt-ip-of-errored-node>
   2. Enter reboot into the CLI to continue the automatic rolling-upgrade process and reboot the node.

7. Navigate to Maintenance > Update.

8. Verify that the automatic rolling upgrade continues without error and that the status of the failed node returns Succeeded.