The incremental backup is running before dawn and the storage can’t keep up: every server’s backup runs late, and so does any restore someone needs right then. Why does a storage server with spinning disks in parity RAID struggle during the window of an incremental backup made with rsync and hardlinks?
Why it matters to whoever runs the server
- The storage works even when nothing changed: with rsync and hardlinks, every unchanged file gets a new hardlink every night. On a server with hundreds of thousands of files, that’s hundreds of thousands of random writes, exactly what parity RAID does badly.
- By the estimate, a 6-disk array handles about 150 to 225 of those writes per second. When the storage saturates, the backup window of every server writing to it pays for it, and so does any restore at the same time.
See the data as a table
| Operation | Per second | Math |
|---|---|---|
| One spinning disk | ~150 | reference |
| Reads, 6 disks | ~900 | 6 × 150 |
| Small writes, RAID 50 | ~225 | 900 ÷ 4 |
| Small writes, RAID 60 | ~150 | 900 ÷ 6 |
Method
The comparison happens on the storage: for every file of every account, rsync checks size and date against the previous snapshot. Millions of lookups scattered across the disks, every night.
Unchanged files also cause writes: with --link-dest, every unchanged file gets a new hardlink in the day’s snapshot, and its link count is updated.
Retention repeats the cost: deleting the oldest snapshot removes millions of hardlinks.
Reindexing walks the whole snapshot tree: the same storm of random reads.
What we decided
In Salvo, the “changed or not” decision is made on the source server itself, using an index cached locally; the storage is not queried file by file. It receives a few large files written sequentially, which is the kind of write parity RAID handles well. And there is no reindexing: the index is part of the repository.