Resolve k3s on VM — Disk High Latency / Slow IOPS

Summary

VMs running k3s exhibit abnormally high disk latency and degraded IOPS when their Ceph-backed block volume was cloned from a snapshot and the RBD image has not been flattened. Every write operation incurs copy-on-write overhead against the parent snapshot, saturating I/O even when disk tilisation appears low. Running fstrim does not resolve the issue. Flattening the RBD image resolves the issue.

Details

Use the following symptoms to confirm this KB applies to your environment.

• Symptom A — Disk shows low utilisation but high latency: df -h reports normal usage (e.g. 13% on a 200 GiB volume), yet applications experience slow read/write performance.
• Symptom B — Ceph volume has an active parent reference: rbd info shows a parent: field pointing to a snapshot, and rbd du reports USED close to PROVISIONED despite low guest-side utilisation.
• Symptom C — fstrim produces no improvement: Running fstrim -av on the guest completes without error but latency remains unchanged.
• Root Cause: The Ceph RBD volume was cloned from a snapshot and the image was never flattened. All writes trigger copy-on-write against the parent snapshot chain, which adds significant latency and reduces effective IOPS.

Confirming the parent reference

Run the following on the Ceph cluster to check for a parent relationship:

rbd du cinder-volumes/<volume-id>

If the output contains a parent: line (as shown below), the volume is still linked to its snapshot origin and must be flattened.

parent: cinder-volumes/volume-04d1fa55-7990-43a9-8018-9c1658fe87f9@snapshot-46874354-44b6-448e-be22-daa70ef24fc0
overlap: 200 GiB

Applicable versions

All CubeCOS versions.

Resolution

Choose the remediation path that best fits your deployment schedule.

Option 1: Flatten the RBD image (Recommended — support action)

Flattening the image severs the parent-snapshot dependency. All data is copied into the volume itself, eliminating copy-on-write overhead. This operation runs online and does not require a VM reboot, but it will temporarily increase Ceph I/O during the copy.

Run the following on the Ceph cluster:

rbd flatten cinder-volumes/<volume-id>

Monitor progress until completion:

Image flatten: 100% complete...done.

Once flattened, verify that rbd info no longer shows a parent: field. Disk latency should return to baseline levels immediately.

Option 2: Redirect k3s data directory (User-side workaround)

If flattening is not immediately possible, you can reduce the impact by pointing k3s to a volume or path with lower latency. Use the --data-dir flag at server startup to specify an alternative location:

k3s server --data-dir /path/to/fast-volume/k3s

Refer to the k3s server CLI reference for full details on the --data-dir option and supported paths.

---

Additional information

• The copy-on-write penalty is proportional to the overlap between the child volume and its parent snapshot. Large overlaps (e.g. 200 GiB as shown in rbd du) produce the most severe latency impact.
• fstrim operates at the filesystem/guest level and has no effect on Ceph-layer copy-on-write chains. This is expected behaviour.
• RBD image features such as object-map and fast-diff (present in the example above) are not a contributing cause and do not need to be disabled.

---

What's next

After flattening the RBD image:

1. Verify baseline IOPS using fio or iostat on the guest to confirm latency has returned to expected levels.
2. Review your volume provisioning workflow to ensure future clones are flattened before being attached to production workloads.
3. If multiple VMs show similar symptoms, audit all Ceph volumes for active parent references using rbd ls and rbd info.