Troubleshoot storage issues

Use this guide to diagnose common CubeCOS storage problems. If you haven't already, start with the Troubleshooting overview to rule out a cluster-wide issue.

Before you begin

Some checks require SSH access to a cluster node as the admin user. Once connected, use the storage command to check Ceph cluster and OSD health.

Connect to a cluster node

ssh admin@<cluster-vip>

The Ceph cluster reports HEALTH_WARN

1. Check the overall cluster status.

   Check storage cluster status

   cc1> storage status

2. Read the health line and the messages under it, such as N osds down or Degraded data redundancy. The STATE column in the OSD table shows which OSD is affected — a healthy OSD shows exists,up, while a failed one shows exists without up.

3. If a specific OSD is down, see An OSD is down below.

4. If no OSD is down but the cluster still reports degraded placement groups, the cluster may still be recovering from a recent change. Re-run storage status after a few minutes to confirm it returns to HEALTH_OK.

An OSD is down

1. List all OSDs and find the one with a down or missing up state.

   List all OSDs

   cc1> storage list_osd

2. Check the failing OSD's disk health.

   Get details for a specific OSD

   cc1> storage list_osd <osd-id>

   For more information, see Check OSD Status.

3. If the underlying disk has failed, remove the OSD so the cluster can rebalance around it.

   Remove a failed OSD

   cc1> storage remove_osd

   For the full procedure, see Remove OSD from Storage Pool.

4. If you need to physically replace the disk, remove it from the cluster first.

   Remove a failed disk

   cc1> storage remove_disk

   danger

   Force mode skips data migration and carries a high risk of data corruption. Use safe mode unless you understand the risk and need to remove the disk immediately. See Remove Hard Disks.

A volume is stuck in Creating status

1. Check the volume's status directly.

   Check volume status

   openstack volume show <volume-id> -c status

2. Check the Ceph cluster status. A volume can stay in Creating status if the underlying storage pool is degraded or rebalancing.

   Check storage cluster status

   cc1> storage status

A volume won't attach to or detach from an instance

1. Confirm the volume's status and current attachments.

   Check volume attachments

   openstack volume show <volume-id> -c status -c attachments

2. Confirm that the target instance status is Active. You can't attach or detach a volume on an instance that's still building, resizing, or in an error state.

3. Retry the operation from Compute > Instances > More > Related Resources > Attach Volume. For the full procedure, see Create and attach a volume.

Extending a volume doesn't reflect the new size inside the guest OS

Extending a volume in CubeCOS grows the underlying block device, but the guest OS still needs to extend its own partition and filesystem to use the new space. See Extend partition.

Rebalancing or removing a disk takes a long time or fails

• Safe mode removal migrates existing data off the disk before removing it, which takes longer on disks with more data. Let it finish rather than retrying.
• If a safe-mode removal fails because the cluster doesn't have enough remaining capacity or fault domains to migrate the data, add capacity first, or accept the risk of force mode. See Remove Hard Disks.
• If you changed the replication factor with storage rebalance and the cluster has been recovering for an extended period, check storage status for the current recovery rate before assuming it's stuck.

Object storage or S3 operations fail

1. Confirm that you're using valid EC2/S3 credentials for the project, and that the container's access policy allows the operation you're attempting.
2. Confirm the container exists and is visible under Project > Storage > Object Storage.

For setup and usage details, see Object Storage (Swift API) and Object Storage (S3 API).

A file share won't mount

1. Confirm that the share status is Active.
2. Confirm that the share network and access rules allow the client's IP address or network.

For setup details, see Tenant Share.