Network offload guide

Network offload separates cluster traffic into dedicated interfaces, ensuring that no single link becomes a throughput bottleneck or a single point of failure. Following the OpenStack reference architectures, it isolates control-plane, tenant, and storage traffic onto separate physical paths, thus achieving network traffic segregation.

Network roles

CubeCOS assigns cluster traffic to four network roles:

• Management: Access to the CubeCOS management web UI, CLI, and root shell. This is control-plane traffic and needs external reachability, but not high bandwidth.
• Provider: The external network that carries internet-routable traffic, equivalent to a Neutron provider or external network used for floating IPs and public API endpoints.
• Overlay: Tenant (project) network traffic tunneled by the software-defined networking backend. This carries east-west traffic between instances on different compute nodes, so it's the most latency- and bandwidth-sensitive role.
• Storage (frontend): Client I/O between compute, Cinder, and Glance services and the software-defined storage cluster equivalent to a Ceph public network.
• Storage (backend): OSD-to-OSD replication and heartbeat traffic equivalent to a Ceph cluster network. Keeping this on its own interface means a rebalance or recovery doesn't compete with client I/O on the frontend.

Recommended interface layout

• Management: IF.1, 1 Gbps with internet access.
• Overlay: overlay.100, two 10 Gbps interfaces bonded, VLAN tag 100.
• Storage (frontend): data0.101, two 10 Gbps interfaces bonded, VLAN tag 101.
• Storage (backend): data1.102, two 10 Gbps interfaces bonded, VLAN tag 102.
• Provider: IF.1 is shared with management and has no VLAN tag. The provider network maps directly to an externally routable segment; therefore, tagging it adds unnecessary complexity. If management and provider traffic together approach the link's capacity, give it a dedicated bonded interface instead.

Bond every multi-NIC role with LACP (802.3ad), fast rate, and a layer3+4 transmit hash policy. LACP gives you both throughput aggregation and automatic failover if a link drops, and it requires the upstream switch ports to be configured as a matching LACP port channel.

tip

In smaller or lower-throughput deployments, you can combine management and provider on one dual-port 10 Gbps bonded NIC, and overlay and storage on a second. Larger or performance-sensitive clusters should keep all four roles on separate bonded interfaces as shown above. For NVMe-backed storage, use at least 25 Gbps for the storage interfaces, and 100 Gbps for large-scale or latency-sensitive deployments.

Configure interfaces during node setup

CubeCOS prompts for each network role when you assign a node's role. A role that doesn't carry tenant or storage traffic skips the corresponding prompt — for example, the control, storage, and moderator roles only prompt for management and storage interfaces, while control-converged, compute, and edge-core prompt for all four roles.

The following example configures a node with the offloaded interface layout described above:

CubeCOS interface dump during node setup

Specify external IP/domain [optional]:
Specify management interface: IF.1
Specify provider interface: IF.1
Specify overlay interface: overlay.100
Specify storage interface [frontend(,backend)]: data0.101,data1.102
Specify region ("RegionOne"):
Specify cluster secret seed: bigstack
Specify management CIDR ("10.254.0.0/16"):