Fix load balancer service errors in clusters with more than three compute nodes

Summary

In CubeCOS version 3.0.0, clusters with more than three compute nodes may falsely be reported as 'Unhealthy' by Load Balancer as a Service (LBaaS).

Details

Issues description

Octavia workers are designed to run only on the first three compute nodes. However, the health checker evaluates all compute nodes. When it checks compute nodes beyond the first three, it detects that Octavia is not running on those nodes and incorrectly reports the LBaaS service as unhealthy.

Applicable versions

CubeCOS 3.0.0 and above.

This article applies to environments running CubeCOS version 3.0.0 or newer.

Resolution

Patch the LBaaS health monitor

Open a terminal session.
Log into your cluster management VIP as the root user via SSH.
```
ssh root@<your-cluster-vip>
```
Open the health monitor configuration file to edit the LBaaS health probe.

The SDK health module provides health checks and automated repair actions for CubeCOS services. The command will open the health monitor configuration through a file editor.
```
vim /usr/lib/hex_sdk/modules/sdk_health.sh
```
Locate the Octavia health check function
1. Search for the health_octavia_check function by typing /health_octavia_check()
2. Press Enter
3. When the health_octavia_check() function is highlighted, review the existing logic. You will modify this function to prevent false LBaaS health errors.
Update the compute-node check so it only validates Octavia services on the first three compute nodes.
1. In the health_octavia_check() function, wrap the Octavia compute-node checks with the following condition:
```
if remote_run $node hex_sdk is_first_three_compute_node ; then
...
fi
```
Apply the health check changes
1. Locate the line that sets the error log source
```
ERR_LOG="ip addr show"
```
2. Find the for loop that immediately follows this line
```
for node in "${CUBE_NODE_COMPUTE_HOSTNAMES[@]}" ; do
    ...
done
```

Add the conditional logic inside this loop, as shown in the following diff:

health_octavia_check()
{
    stale_api_check_repair octavia-api 9876 octavia-api python3

    local http_stats=$(influx -host $(shared_id) -database monasca -format json -execute "select last(value) from http_status where service = 'octavia'" | jq .results[0].series[0].values[0][1])
    if [ "$http_stats" != "0" ] ; then
        ERR_CODE=1
    else
        for node in "${CUBE_NODE_CONTROL_HOSTNAMES[@]}" ; do
            if ! is_remote_running $node octavia-api ; then
                ERR_MSG+="octavia-api on $node is not running\n"
                ERR_CODE=3
                ERR_LOG="journalctl -n $ERR_LOGSIZE -u octavia-api"
            elif ! is_remote_running $node octavia-housekeeping ; then
                ERR_MSG+="octavia-housekeeping on $node is not running\n"
                ERR_CODE=4
                ERR_LOG="journalctl -n $ERR_LOGSIZE -u octavia-housekeeping"
            fi
        done
        ERR_LOG="ip addr show"
        for node in "${CUBE_NODE_COMPUTE_HOSTNAMES[@]}" ; do
+            if remote_run $node hex_sdk is_first_three_compute_node ; then
+                if ! remote_run $node ovs-vsctl port-to-br octavia-hm0 >/dev/null 2>&1 ; then
+                    ERR_MSG+="no octavia-hm0 ovn port found on $node\n"
+                    ERR_CODE=5
+                elif remote_run $node ip link show octavia-hm0 | grep -q DOWN ; then
+                    ERR_MSG+="no link octavia-hm0 found on $node\n"
+                    ERR_CODE=6
+                elif ! remote_run $node route -n | grep -q octavia-hm0 ; then
+                    ERR_MSG+="no route from octavia-hm0 found on $node\n"
+                    ERR_CODE=7
+                elif ! is_remote_running $node octavia-worker ; then
+                    ERR_MSG+="octavia-worker on $node is not running\n"
+                    ERR_CODE=8
+                elif ! is_remote_running $node octavia-health-manager ; then
+                    ERR_MSG+="octavia-health-manager on $node is not running\n"
+                    ERR_CODE=9
+                fi
+            fi
-            if ! remote_run $node ovs-vsctl port-to-br octavia-hm0 >/dev/null 2>&1 ; then
-                ERR_MSG+="no octavia-hm0 ovn port found on $node\n"
-                ERR_CODE=5
-            elif remote_run $node ip link show octavia-hm0 | grep -q DOWN ; then
-                ERR_MSG+="no link octavia-hm0 found on $node\n"
-                ERR_CODE=6
-            elif ! remote_run $node route -n | grep -q octavia-hm0 ; then
-                ERR_MSG+="no route from octavia-hm0 found on $node\n"
-                ERR_CODE=7
-            elif ! is_remote_running $node octavia-worker ; then
-                ERR_MSG+="octavia-worker on $node is not running\n"
-                ERR_CODE=8
-            elif ! is_remote_running $node octavia-health-manager ; then
-                ERR_MSG+="octavia-health-manager on $node is not running\n"
-                ERR_CODE=9
-            fi
        done
    fi

    _health_fail_log
}

Sync the updated configuration.

After saving the file, sync the updated health module to all compute and control-converged nodes:
```
cubectl node rsync /usr/lib/hex_sdk/modules/sdk_health.sh -r compute
```
Verify that health check fix.
Run the health check manually:
```
hex_sdk health_octavia_check ; echo $?
```
If the command returns 0, the fix is complete. No service restart or node reboot is required.

Summary​

Details​

Issues description​

Applicable versions​

Resolution​

Patch the LBaaS health monitor​