# Fix load balancer service errors in clusters with more than three compute nodes

> Learn how to resolve false LBaaS health errors in clusters with more than three compute nodes running CubeCOS 3.0.0.

## Summary {#summary}

In CubeCOS version 3.0.0, clusters with more than three compute nodes may falsely be reported as 'Unhealthy' by Load Balancer as a Service (LBaaS).

## Details {#details}

### Issues description {#issues-description}

Octavia workers are designed to run only on the first three compute nodes. However, the health checker evaluates all compute nodes. When it checks compute nodes beyond the first three, it detects that Octavia is not running on those nodes and incorrectly reports the LBaaS service as `unhealthy`.

### Applicable versions {#applicable-versions}

CubeCOS 3.0.0 and above.

This article applies to environments running CubeCOS version 3.0.0 or newer.

## Resolution {#resolution}

### Patch the LBaaS health monitor {#patch-the-lbaas-health-monitor}

1. Open a terminal session.
2. Log into your cluster management VIP as the `root` user via SSH.

   ```bash
   ssh root@<your-cluster-vip>
   ```

3. Open the health monitor configuration file to edit the LBaaS health probe.

   The SDK health module provides health checks and automated repair actions for CubeCOS services. The command will open the health monitor configuration through a file editor.

   ```bash
   vim /usr/lib/hex_sdk/modules/sdk_health.sh
   ```

4. Locate the Octavia health check function
   1. Search for the `health_octavia_check` function by typing `/health_octavia_check()`
   2. Press `Enter`
   3. When the `health_octavia_check()` function is highlighted, review the existing logic. You will modify this function to prevent false LBaaS health errors.
5. Update the compute-node check so it only validates Octavia services on the first three compute nodes.
   1. In the health_octavia_check() function, wrap the Octavia compute-node checks with the following condition:

      ```shell
      if remote_run $node hex_sdk is_first_three_compute_node ; then
      ...
      fi
      ```

6. Apply the health check changes
   1. Locate the line that sets the error log source

      ```bash
      ERR_LOG="ip addr show"
      ```

   2. Find the `for` loop that immediately follows this line

      ```bash
      for node in "${CUBE_NODE_COMPUTE_HOSTNAMES[@]}" ; do
          ...
      done
      ```

7. Add the conditional logic inside this loop, as shown in the following diff:

   ```diff
   health_octavia_check()
   {
       stale_api_check_repair octavia-api 9876 octavia-api python3

       local http_stats=$(influx -host $(shared_id) -database monasca -format json -execute "select last(value) from http_status where service = 'octavia'" | jq .results[0].series[0].values[0][1])
       if [ "$http_stats" != "0" ] ; then
           ERR_CODE=1
       else
           for node in "${CUBE_NODE_CONTROL_HOSTNAMES[@]}" ; do
               if ! is_remote_running $node octavia-api ; then
                   ERR_MSG+="octavia-api on $node is not running\n"
                   ERR_CODE=3
                   ERR_LOG="journalctl -n $ERR_LOGSIZE -u octavia-api"
               elif ! is_remote_running $node octavia-housekeeping ; then
                   ERR_MSG+="octavia-housekeeping on $node is not running\n"
                   ERR_CODE=4
                   ERR_LOG="journalctl -n $ERR_LOGSIZE -u octavia-housekeeping"
               fi
           done
           ERR_LOG="ip addr show"
           for node in "${CUBE_NODE_COMPUTE_HOSTNAMES[@]}" ; do
   +            if remote_run $node hex_sdk is_first_three_compute_node ; then
   +                if ! remote_run $node ovs-vsctl port-to-br octavia-hm0 >/dev/null 2>&1 ; then
   +                    ERR_MSG+="no octavia-hm0 ovn port found on $node\n"
   +                    ERR_CODE=5
   +                elif remote_run $node ip link show octavia-hm0 | grep -q DOWN ; then
   +                    ERR_MSG+="no link octavia-hm0 found on $node\n"
   +                    ERR_CODE=6
   +                elif ! remote_run $node route -n | grep -q octavia-hm0 ; then
   +                    ERR_MSG+="no route from octavia-hm0 found on $node\n"
   +                    ERR_CODE=7
   +                elif ! is_remote_running $node octavia-worker ; then
   +                    ERR_MSG+="octavia-worker on $node is not running\n"
   +                    ERR_CODE=8
   +                elif ! is_remote_running $node octavia-health-manager ; then
   +                    ERR_MSG+="octavia-health-manager on $node is not running\n"
   +                    ERR_CODE=9
   +                fi
   +            fi
   -            if ! remote_run $node ovs-vsctl port-to-br octavia-hm0 >/dev/null 2>&1 ; then
   -                ERR_MSG+="no octavia-hm0 ovn port found on $node\n"
   -                ERR_CODE=5
   -            elif remote_run $node ip link show octavia-hm0 | grep -q DOWN ; then
   -                ERR_MSG+="no link octavia-hm0 found on $node\n"
   -                ERR_CODE=6
   -            elif ! remote_run $node route -n | grep -q octavia-hm0 ; then
   -                ERR_MSG+="no route from octavia-hm0 found on $node\n"
   -                ERR_CODE=7
   -            elif ! is_remote_running $node octavia-worker ; then
   -                ERR_MSG+="octavia-worker on $node is not running\n"
   -                ERR_CODE=8
   -            elif ! is_remote_running $node octavia-health-manager ; then
   -                ERR_MSG+="octavia-health-manager on $node is not running\n"
   -                ERR_CODE=9
   -            fi
           done
       fi

       _health_fail_log
   }
   ```

8. Sync the updated configuration.

   After saving the file, sync the updated health module to all compute and control-converged nodes:

   ```bash
   cubectl node rsync /usr/lib/hex_sdk/modules/sdk_health.sh -r compute
   ```

9. Verify that health check fix.
10. Run the health check manually:

    ```shell
    hex_sdk health_octavia_check ; echo $?
    ```

11. If the command returns `0`, the fix is complete.
    No service restart or node reboot is required.