# Troubleshoot storage issues

> Diagnose and resolve common CubeCOS storage problems, including Ceph cluster health, OSD failures, volume attachment, and object or file storage errors.

Use this guide to diagnose common CubeCOS storage problems. If you haven't already, start with the [Troubleshooting overview](./overview) to rule out a cluster-wide issue.

## Before you begin {#before-you-begin}

Some checks require SSH access to a cluster node as the `admin` user. Once connected, use the `storage` command to check Ceph cluster and OSD health.

```bash title="Connect to a cluster node"
ssh admin@<cluster-vip>
```

## The Ceph cluster reports HEALTH_WARN {#the-ceph-cluster-reports-health_warn}

1. Check the overall cluster status.

   ```bash title="Check storage cluster status"
   cc1> storage status
   ```

2. Read the `health` line and the messages under it, such as `N osds down` or `Degraded data redundancy`. The `STATE` column in the OSD table shows which OSD is affected — a healthy OSD shows `exists,up`, while a failed one shows `exists` without `up`.
3. If a specific OSD is down, see [An OSD is down](#an-osd-is-down) below.
4. If no OSD is down but the cluster still reports degraded placement groups, the cluster may still be recovering from a recent change. Re-run `storage status` after a few minutes to confirm it returns to `HEALTH_OK`.

## An OSD is down {#an-osd-is-down}

1. List all OSDs and find the one with a `down` or missing `up` state.

   ```bash title="List all OSDs"
   cc1> storage list_osd
   ```

2. Check the failing OSD's disk health.

   ```bash title="Get details for a specific OSD"
   cc1> storage list_osd <osd-id>
   ```

   For more information, see [Check OSD Status](../storage/check_osd.md).

3. If the underlying disk has failed, remove the OSD so the cluster can rebalance around it.

   ```bash title="Remove a failed OSD"
   cc1> storage remove_osd
   ```

   For the full procedure, see [Remove OSD from Storage Pool](../storage/remove_osd.md).

4. If you need to physically replace the disk, remove it from the cluster first.

   ```bash title="Remove a failed disk"
   cc1> storage remove_disk
   ```

   :::danger

   Force mode skips data migration and carries a high risk of data corruption. Use safe mode unless you understand the risk and need to remove the disk immediately. See [Remove Hard Disks](../storage/remove_disk.md).

   :::

## A volume is stuck in Creating status {#a-volume-is-stuck-in-creating-status}

1. Check the volume's status directly.

   ```bash title="Check volume status"
   openstack volume show <volume-id> -c status
   ```

2. Check the Ceph cluster status. A volume can stay in `Creating` status if the underlying storage pool is degraded or rebalancing.

   ```bash title="Check storage cluster status"
   cc1> storage status
   ```

## A volume won't attach to or detach from an instance {#a-volume-wont-attach-to-or-detach-from-an-instance}

1. Confirm the volume's status and current attachments.

   ```bash title="Check volume attachments"
   openstack volume show <volume-id> -c status -c attachments
   ```

2. Confirm that the target instance status is `Active`. You can't attach or detach a volume on an instance that's still building, resizing, or in an error state.
3. Retry the operation from **Compute > Instances > More > Related Resources > Attach Volume**. For the full procedure, see [Create and attach a volume](../storage/block/volume.md).

## Extending a volume doesn't reflect the new size inside the guest OS {#extending-a-volume-doesnt-reflect-the-new-size-inside-the-guest-os}

Extending a volume in CubeCOS grows the underlying block device, but the guest OS still needs to extend its own partition and filesystem to use the new space. See [Extend partition](../operating_system/extend_partition.md).

## Rebalancing or removing a disk takes a long time or fails {#rebalancing-or-removing-a-disk-takes-a-long-time-or-fails}

- Safe mode removal migrates existing data off the disk before removing it, which takes longer on disks with more data. Let it finish rather than retrying.
- If a safe-mode removal fails because the cluster doesn't have enough remaining capacity or fault domains to migrate the data, add capacity first, or accept the risk of force mode. See [Remove Hard Disks](../storage/remove_disk.md).
- If you changed the replication factor with `storage rebalance` and the cluster has been recovering for an extended period, check `storage status` for the current recovery rate before assuming it's stuck.

## Object storage or S3 operations fail {#object-storage-or-s3-operations-fail}

1. Confirm that you're using valid EC2/S3 credentials for the project, and that the container's access policy allows the operation you're attempting.
2. Confirm the container exists and is visible under **Project > Storage > Object Storage**.

For setup and usage details, see [Object Storage (Swift API)](../storage/object/s3.md) and [Object Storage (S3 API)](../storage/object/s3_api.md).

## A file share won't mount {#a-file-share-wont-mount}

1. Confirm that the share status is `Active`.
2. Confirm that the share network and access rules allow the client's IP address or network.

For setup details, see [Tenant Share](../storage/file/tenant_share.md).