# Resolve k3s on VM — Disk High Latency / Slow IOPS

> k3s workloads on VMs backed by Ceph volumes cloned from snapshots experience high disk latency and slow IOPS due to copy-on-write overhead from an un-flattened parent image. Resolving requires flattening the RBD image or configuring k3s to use a lower-latency data path.

## Summary {#summary}

VMs running k3s exhibit abnormally high disk latency and degraded IOPS when their Ceph-backed block volume was cloned from a snapshot and the RBD image has not been flattened. Every write operation incurs copy-on-write overhead against the parent snapshot, saturating I/O even when disk tilisation appears low. Running `fstrim` does not resolve the issue. Flattening the RBD image resolves the issue.

## Details {#details}

Use the following symptoms to confirm this KB applies to your environment.

- **Symptom A — Disk shows low utilisation but high latency:** `df -h` reports normal usage (e.g. 13% on a 200 GiB volume), yet applications experience slow read/write performance.
- **Symptom B — Ceph volume has an active parent reference:** `rbd info` shows a `parent:` field pointing to a snapshot, and `rbd du` reports `USED` close to `PROVISIONED` despite low guest-side utilisation.
- **Symptom C — `fstrim` produces no improvement:** Running `fstrim -av` on the guest completes without error but latency remains unchanged.
- **Root Cause:** The Ceph RBD volume was cloned from a snapshot and the image was never flattened. All writes trigger copy-on-write against the parent snapshot chain, which adds significant latency and reduces effective IOPS.

### Confirming the parent reference {#confirming-the-parent-reference}

Run the following on the Ceph cluster to check for a parent relationship:

```bash
rbd du cinder-volumes/<volume-id>
```

If the output contains a `parent:` line (as shown below), the volume is still linked to its snapshot origin and must be flattened.

```bash
parent: cinder-volumes/volume-04d1fa55-7990-43a9-8018-9c1658fe87f9@snapshot-46874354-44b6-448e-be22-daa70ef24fc0
overlap: 200 GiB
```

### Applicable versions {#applicable-versions}

All CubeCOS versions.

## Resolution {#resolution}

Choose the remediation path that best fits your deployment schedule.

### Option 1: Flatten the RBD image (Recommended — support action) {#option-1-flatten-the-rbd-image-recommended--support-action}

Flattening the image severs the parent-snapshot dependency. All data is copied into the volume itself, eliminating copy-on-write overhead. This operation runs online and does not require a VM reboot, but it will temporarily increase Ceph I/O during the copy.

Run the following on the Ceph cluster:

```bash
rbd flatten cinder-volumes/<volume-id>
```

Monitor progress until completion:

```bash
Image flatten: 100% complete...done.
```

Once flattened, verify that `rbd info` no longer shows a `parent:` field.
Disk latency should return to baseline levels immediately.

### Option 2: Redirect k3s data directory (User-side workaround) {#option-2-redirect-k3s-data-directory-user-side-workaround}

If flattening is not immediately possible, you can reduce the impact by pointing k3s to a volume or path with lower latency. Use the `--data-dir` flag at server startup to specify an alternative location:

```bash
k3s server --data-dir /path/to/fast-volume/k3s
```

Refer to the [k3s server CLI reference](https://docs.k3s.io/cli/server#data) for full details on the `--data-dir` option and supported paths.

---

## Additional information {#additional-information}

- The copy-on-write penalty is proportional to the overlap between the child volume and its parent snapshot. Large overlaps (e.g. 200 GiB as shown in `rbd du`) produce the most severe latency impact.
- `fstrim` operates at the filesystem/guest level and has no effect on Ceph-layer copy-on-write chains. This is expected behaviour.
- RBD image features such as `object-map` and `fast-diff` (present in the example above) are not a contributing cause and do not need to be disabled.

---

## What's next {#whats-next}

After flattening the RBD image:

1. Verify baseline IOPS using `fio` or `iostat` on the guest to confirm latency has returned to expected levels.
2. Review your volume provisioning workflow to ensure future clones are flattened before being attached to production workloads.
3. If multiple VMs show similar symptoms, audit all Ceph volumes for active parent references using `rbd ls` and `rbd info`.