Storage
Recovering a full array disk without losing the array
One disk in the array hit one hundred percent while the array as a whole still had room. Files pin to the disk they were created on, so the fix was to move data off. Both of my first two attempts made things worse before they made them better.
The symptom
- A single disk in the array reported full while the array still showed free space overall.
- Files pin to the disk they were created on, so new writes targeting that disk failed even though capacity existed elsewhere.
What I tried first, and why it failed
- Moved data off the full disk with a naive copy, expecting space to be freed.
- Space was not actually reclaimed: hardlinked files kept the original data alive when only one of the pair was moved.
- Second problem surfaced during the move: throughput collapsed to a crawl.
Root cause
- Hardlink pairs have to be moved together or the underlying data is never released.
- Sparse files over reconstruct-write parity turn a naive rsync into a catastrophically slow operation, because every sparse region gets written out in full.
The fix
- Moved hardlinked files as a set so the links stayed intact and the space was genuinely freed.
- Used rsync with the sparse flag so sparse regions were not written out in full.
- Verified reclaimed capacity and array health after the move.
What I would do differently
- Measure before optimizing. What looked like a slow network was parity-write saturation, and I would have spent a day on the wrong layer if I had trusted the first guess.
- Freed space is the artifact worth verifying, not the exit code of the copy. The first move reported success and reclaimed nothing.
- Understand how the filesystem allocates before moving data around inside it. Hardlinks and sparse regions both broke the naive mental model of "a file is a file".
- Do the destructive half last, and only once the successor is confirmed to exist.