Troubleshoot network issues

Use this guide to diagnose common CubeCOS networking problems. If you haven't already, start with the Troubleshooting overview to rule out a cluster-wide issue.

Before you begin

Some checks require the OpenStack CLI. Run these commands from a controller node, or from any host with /etc/admin-openrc.sh sourced and the openstack client installed.

An instance has no network connectivity

1. Confirm that the instance has a fixed IP on the expected network, and a floating IP if it needs to reach or be reached from outside the project network.
2. Confirm that the instance's security group allows the traffic you expect. See Security Groups for how rules are structured.
3. If the instance needs external access, confirm that its network is connected to a router with an external gateway, and that the router status is Active (the creation flow briefly shows this as Activate). See Create router.
4. Run cluster check_repair and check the Network row for a FIXING status.

A floating IP won't associate or doesn't work

1. Confirm the floating IP's status and the port it's associated with.

   List floating IPs

   openstack floating ip list

2. Confirm that the associated instance's security group allows inbound traffic on the port and protocol you're testing.

3. Confirm that the instance status is Active. A floating IP mapped to a stopped or errored instance won't respond.

SSH or ping requests time out

1. Confirm that the security group attached to the instance has an ingress rule for the protocol and port you're using (for example, TCP port 22 for SSH, or ICMP for ping). See Security Groups.
2. Confirm that you're connecting to the correct address: the floating IP if you're outside the project network, or the fixed IP if you're on the same private network.
3. Confirm that the guest OS's own firewall isn't blocking the connection.

A router shows the wrong status

The router creation flow can briefly display Activate instead of Active as a UI label rather than an error. Confirm the actual state before troubleshooting further.

Check router status

openstack router show <router-id> -c status

If the router isn't Active, confirm that it has an external gateway set, and that the external network it points to still exists.

DNS records aren't resolving

1. Confirm that the DNS zone exists and is active.

   List DNS zones

   openstack zone list

2. Confirm that the record was created in the zone you expect.

   List records in a zone

   openstack recordset list <zone-id>

3. If you're using automatic DNS, confirm that the network associated with the zone is correct. If you're using manual DNS, confirm that the record was created with the correct type and value.

A load balancer is unresponsive or shows an error

Run the load balancer recovery process from the CubeCOS admin CLI.

Recover a load balancer

ssh admin@<cluster-vip>

Fix a load balancer

controller> iaas lb fix

Select the domain, tenant, and load balancer to fix, then enter YES to confirm. For the full command output, see Fix Load Balancer.

Network services report a fault

Run cluster check_repair and check the Network row. A FIXING status with neutron listed identifies a networking service issue rather than a configuration problem on a specific resource. For more information, see Monitor and repair CubeCOS services using the web UI or CLI.