Storage

The last unprotected component

I had dual parity across two dozen drives and a real backup story, and I still had one component where a single failure would have cost me something I could not rebuild.


What I thought was covered

  • The array runs dual parity, so any two drives can fail at once with zero data loss.
  • The irreplaceable files are also backed up offsite, so hardware failure was not the only thing standing between me and my photos.
  • On that basis the storage side felt finished.

The thing I had missed

  • A cache sits in front of the array to absorb sustained writes, because parity writes are read-modify-write across three disks and would otherwise stall on ingest.
  • Cache is not part of the array, so array parity does not cover anything living on it.
  • What lives there is not just in-flight files. It is live databases, including the one Immich uses.
  • So dual parity was protecting my photo files while nothing was protecting the metadata describing them: the albums, the faces, the search index, the organization.
  • Losing that would not have lost the photos. It would have lost every bit of structure that made them findable, and no parity rebuild would bring it back.

The fix

  • Moved the cache to a mirrored pair, so a single device failure no longer takes the databases with it.
  • This was deliberate rather than incidental: the mirror exists specifically to cover what array parity structurally cannot.

The lesson

  • Redundancy protects the layer it is applied to, and nothing else. It is easy to read "the array is protected" as "my data is protected".
  • Ask what is not covered, component by component, rather than checking whether a protection scheme exists at all.
  • Data about data is still data. The index, the database, and the metadata deserve the same scrutiny as the files they describe.
  • The gap was not exotic. It sat in the middle of a setup I would have described as well protected.

More

← All writeups