When a customer loses their whole account (a WordPress site, a database and 13 mailboxes, 10 GB), they are on the phone, waiting. We measured how long Salvo took to bring that account back at the worst possible moment: the hour when the backup storage is busiest, receiving backups from several other servers at the same time. The account’s server was calm; the bottleneck was the storage. Result: 29 min 47 s. We did not like it. We went after the cause, improved Salvo’s restore in two places and measured again, with the same account, at the same hour.
| At the backup storage’s worst moment | Before the improvements | Improvement 1 (8 connections) | Improvements 1 and 2 (backup in one go) |
|---|---|---|---|
| Fetch the backup from the storage | 26 min 57 s | 4 min 44 s | 2 min 39 s |
| cPanel recreating the account | 2 min 48 s | 2 min 57 s | 3 min 03 s |
| From deleted account to account back | 29 min 47 s | 7 min 44 s | 5 min 45 s |
| Faster than before | — | 3.9× | 5.2× |
Method
One real account: a WordPress site with a 12-table database, 13 mailboxes with 15.5 thousand messages and 24.9 thousand files. About 10 GB on the storage and 11 GB once restored.
The backup sits on a remote storage, reached over SSH, which also receives backups from other servers. The three restores ran within the same hour, that storage’s busiest: the account was deleted and restored without the improvements; with improvement 1; and with improvements 1 and 2.
Before and after each restore, a full snapshot of the account: the hash of every file, the checksum of every table and the messages in every folder. In all three, the customer’s account came back identical: 23,801 of 23,801 customer files (website, uploaded files and messages), 100%, the database with the same checksum and the email the same, mailbox by mailbox. The only differences are in files the system recreates on its own: mail server indexes, panel caches and copies of the webmail database that the panel renames with the restore date.
How we found the bottleneck. In the first measurement, the panel’s standard backup package, already on the server itself, came back in 3 min 04 s; Salvo, fetching the backup from the remote storage, took 58 min 40 s, and 93% of that was reading the storage. With the storage without that load, it dropped to 7 min 53 s, and multiplying the reads barely helped. That is when we saw the single-connection queue.
What we changed in Salvo
First, we looked at where the time went. cPanel recreating the account takes about 3 minutes in any scenario: that is the floor, and it does not depend on Salvo. The other 27 minutes were Salvo fetching the backup from the storage. The cause: it talked to the storage over a single connection, one request at a time, like a bank line with one cashier open; at the storage’s busiest hour, each request took longer and the line moved slowly. Improvement 1, eight cashiers open: Salvo now fetches the backup over 8 connections at once. That alone took the account from 29 min 47 s to 7 min 44 s. Improvement 2, the backup comes whole, in one go: instead of fetching pieces bit by bit while restoring, Salvo copies the account’s backup to the server in a single pass (110 s, at 91 MB/s) and restores from there. We got to 5 min 45 s, with no extra disk at the peak. With both, the restore at the worst hour was faster than the best one we had measured with the storage free (7 min 53 s). Both improvements are already in Salvo and were measured here; for now they are on only in testing, and the default is still one connection until we decide. And the backup? It does not gain from the extra connections (74 s with 1, 110 s with 8): in the backup, what limits is reading and compressing the data on the server.