Increase PCIe ports to support multiple input/ output devices per virtual machine

Summary

When virtual machines need more than two network interfaces, the default allocation of PCIe ports can become a limitation. To resolve this, you need to manually increase the number of available PCIe ports in the VM configuration.

Key steps to resolve insufficient PCIe ports available

• Increase the default limit on the PCIe ports available to CubeCOS
• Add additional PCIe root port devices.
• Attach extra network interfaces to these newly created ports.
• Restart or redefine the VM for changes to take effect.

This ensures the VM has enough PCIe slots to support multiple NICs without conflicts.

Details

Use this article if attaching multiple network interfaces to a virtual machine fails and you have not configured additional PCIe ports.

Applicable versions

CubeCOS 3.0.0 and above.

This article applies to environments running CubeCOS version 3.0.0 or newer.

Resolution

Configure Nova computing service to use maximum available PCIe ports

The default allocation of PCIe ports may be insufficient. To resolve this issue, set the maximum number of available PCIe ports manually to the maximum supported by CubeCOS in the configuration.

1. Open a terminal session.

2. Log into the cluster VIP as the root user via SSH.

   ssh root@<your-cluster-vip>

3. Create a new configuration file to set the maximum number of available PCIe ports manually to the maximum supported by CubeCOS.

   touch /etc/nova/nova.d/pcie.conf

4. Oepn the Nova configuration using the command:

   vi /etc/nova/nova.d/pcie.conf

5. Enter the following value into pcie.conf with i to start insert mode.

   The default number of enabled PCIe ports on CubeCOS is 0, and the maximum supported configuration is 28.

   # Override number of enabled pcie ports
   [libvirt]
   num_pcie_ports = 28

6. Save and exit the editor after making changes with :wq.

Sync the PCIe ports configuration file to all computing nodes

Sync the vTPM with a cluster-wide synchronization task to ensure that all compute nodes support the same PCIe ports. Note that the sync command is limited to nodes running a compute service, such as AIO, Compute, or Compute with Storage.

1. Sync the configuration file to all CubeCOS nodes running the compute service.

   cubectl node -r compute rsync /etc/nova/nova.d/pcie.conf

Restart the Nova computing service on all compute service nodes

In order to implement the PCIe ports configuration, finish syncing the configuration file and then execute the following command to restart the service:

warning

Before attaching additional network interfaces, confirm that the synchronization process has finished for all compute nodes.

1. Restart the Nova compute service on all nodes hosting the compute service.

   cubectl node -r compute exec -p "hex_config restart_nova"

2. Check the Nova service health on all compute service nodes.

   cubectl node -r compute exec -p "systemctl status openstack-nova-compute | grep Active"

3. The openstack-nova-compute service status must return Active for all compute service nodes.

Attach additional network interfaces to existing virtual machines

After applying the configuration, the maximum number of PCIe ports will be supported by default for all newly created virtual machines. However, you must modify existing virtual machines before applying the configuration. Follow this procedure to configure an existing virtual machine to support the new maximum number of PCIe ports.

Use the GUI to re-configure existing virtual machines

To support attaching multiple network interfaces to existing virtual machines after increasing the maximum number of PCIe ports, the virtual machine must be shelved and unshelved.

warning

To avoid data loss and unexpected service disruptions, shut down the guest OS safely before proceeding.

You cannot perform shelving or unshelving operations while the virtual machine is active.

1. Log in to the CubeCOS management interface, and navigate to the OpenStack integration.
2. Navigate to Compute > Instances, and locate the instance to modify.
3. Prepare for shelving and unshelving by accessing the guest OS of the instance and shutting down the instance cleanly.
4. Verify that the instance status is Shutoff before proceeding.
5. Shelve the instance by hovering over More, clicking Instance Status, and selecting shelve.
6. Unshelve the instance by hovering over More, clicking Instance Status, and selecting unshelve.
7. Attach additional network interfaces as needed.
8. Restart the virtual machine.

Use the CLI to re-configure existing virtual machines

1. Log in to the CubeCOS management interface, and navigate to the OpenStack integration.
2. Navigate to Compute > Instances, and locate the instance to modify.
3. Clip on the instance ID to copy the instance ID.
4. Prepare for shelving and unshelving by accessing the guest OS of the instance and shutting down the instance cleanly.
5. Verify that the instance status is Shutoff before proceeding.
6. Open a terminal session.
7. Log into the cluster VIP as the root user via SSH with ssh root@<your-cluster-ip>.
8. Shelve the instance by executing openstack server shelve --offload <vm-id>
   1. Replace <vm-id> with the ID obtained in step 3.
9. Unshelve the instance by executing openstack server unshelve <vm-id>
10. Attach additional network interfaces as needed.
11. Restart the virtual machine.