A server went down and the WordPress sites have to go back online. The clock rules. In Lab 23, Salvo brought everything back, every message included, but restored more slowly than JetBackup, and we were not satisfied. This lab is what came next: can a whole account (site, database and email) be restored faster than the panel’s own backup package, already on the server? And many accounts at once?
Why it matters to whoever runs the server
- The clock. Every minute of restoring is a WordPress site offline, email stopped and a ticket open. With 3 accounts at a time, the 6 came back in a little over 4 minutes; JetBackup, in the same window, took 14.
- A large account came back faster than the panel’s own package, already sitting on the server: 81 s versus 126 s on a 3.2 GB account. No waiting on the storage and no copying everything twice.
- Trust in what came back. A fast restore that loses email is no use: in every Salvo round, every message and every database table came back identical, checked one by one. In the same window, JetBackup left messages out on 4 of the 6 accounts, as it had in Lab 23.See:Lab 23
- The backup storage. To bring the 6 accounts back, JetBackup made the storage disk read 192,900 times; Salvo, with 3 at a time, 8,780 times: 22 times fewer. A storage with room to spare can take other servers’ backups at the same time. It is the same effect we measured on backups:See:Lab 04Lab 15
- The price: more restores at once weigh more on the server. With 3 at a time, the average load was 24.9, versus 18.4 for JetBackup, and the disk used 8 GB more at peak, versus 1 GB. That is why Salvo’s default waits for the load to drop before the next account, and the admin chooses how many run together.
- The road here: storage reads got fast in Lab 21, and Lab 23 showed where we were still losing.See:Lab 21Lab 23
How to read it: the grey line is the panel restoring its own package; the blue one, Salvo. The height is how much the server writes to disk each second; the length, how long it took. The panel finished in 109 s. Salvo created the account in 25 s and then brought the content back piece by piece, until 263 s, and still left things out. It wrote 10.7 GB to the server’s disk, versus 6.3 GB for the package. Worse on both counts.
How to read it: same as before. Now the skeleton comes ready from the backup, the account exists in about 20 s, and everything came back. But the content went into a folder next to it and was then moved into place: the stretch at the end is moving 40 thousand files (66 s). Salvo finished in 152 s, versus 118 s for the package, writing 6.6 GB, versus 8.4 GB.
How to read it: same as before. As soon as the account exists, Salvo writes the site and email straight into place, once, and imports the databases at the same time. It finished in 81 s, versus 126 s for the package, and wrote 6.0 GB to the server’s disk, versus 8.2 GB: the panel’s package copies the content twice.
How to read it: each pair is one account, smallest to largest. Grey is the panel’s package, blue is Salvo in two tracks, one right after the other. On medium and large accounts Salvo wins; on small ones, creating the account already takes almost as long as the whole package, and the panel ties or wins by a few seconds.
How to read it: the 6 accounts restored from the storage, each round right after the other, in the same window. Neither tool waited for the server load to drop. JetBackup worked with 5 at a time, its default; Salvo with 1, 3 and 6. More at a time finishes sooner but weighs more on the server (table below). In this window JetBackup was much slower than in the morning, when it restored the same 6 in 309 s:See:Lab 23
How to read it: each point is how many reads per second the backup storage disk did in those 10 seconds. The four charts in this series use the same scale. JetBackup read 7.6 GB from the storage in 192,900 reads, over 14 minutes.
Salvo, one account at a time: 4.6 GB read in 11,520 reads, 17 times fewer than JetBackup, in 6 min 38 s.
Salvo with 3 at a time, the default: 5.2 GB in 8,780 reads, 22 times fewer, in 4 min 11 s.
Salvo with 6 at a time: 5.1 GB in 9,660 reads, 20 times fewer, in 2 min 58 s. More accounts at once do not multiply storage reads: Salvo reads a few large files instead of many small pieces. More on the storage’s workload:See:Lab 04Lab 15
| JetBackup 5 | Salvo 1 | Salvo 3 | Salvo 6 | |
|---|---|---|---|---|
| Time | 839 s | 399 s | 250 s | 179 s |
| Average load | 18.4 | 18.2 | 24.9 | 30.9 |
| Peak load | 28.9 | 23.3 | 31.7 | 41.8 |
| Extra disk at peak | 1 GB | 6 GB | 8 GB | 8 GB |
| Storage reads | 192,900 | 11,520 | 8,780 | 9,660 |
| Read from storage | 7.6 GB | 4.6 GB | 5.2 GB | 5.1 GB |
| All email back | 2 of 6 | 6 of 6 | 6 of 6 | 6 of 6 |
Method
The same 6 real accounts as Lab 23, on a cPanel test server with 8 cores: two small (130 to 150 MB), two medium (~1 GB) and two large (~3.2 GB), 8.6 GB in total, with WordPress sites, databases and mailboxes.
One account: each Salvo attempt was measured right after restoring the panel’s own package, in the same window, because the server load changed during the day. The account was deleted before each round.
Many accounts: the 6 restored from the storage, one round after the other, in the same window. JetBackup with its backup and 5 accounts at a time; Salvo with 1, 3 and 6. To compare, neither tool waited for the server load to drop; Salvo’s default does wait.
After each round, a snapshot of each account against the original: the hash of every customer file, the checksum of every WordPress table and the messages in every mailbox. In every Salvo round, 99.98% of customer files identical; what was missing is only what the default exclusion list leaves out on purpose. JetBackup stayed at 99.45%.
The server was measured every 2 seconds during every round: load, CPU and disk. In the many-account rounds, the backup storage disk was measured too, every 10 seconds: reads per second and volume read. Outside the rounds, the storage was doing no reads at all.
How we got there: what we tried and what changed in Salvo
Lab 23 showed that reading from the storage was already solved. The time was in copying the site and email twice: once to build the panel’s package and again when the panel restores it. The idea: the account is born in the panel with only what it needs (domains, DNS, cron, email accounts, database users) while, at the same time, Salvo fetches the content from the backup and writes it straight into place. Attempt 1: the panel created the account from that skeleton in 25 s, but bringing the content back piece by piece took 229 s and left things out: the account’s main mailbox, the default files the panel creates in a new account and a database with an old name. Slower than the panel’s package (108 s) and incomplete. Attempt 2: the skeleton is now stored ready-made in the backup itself (about 10 MB), and the account existed in about 20 s. The content came into a folder next to it and was moved into place. Everything came back, but it took 151 s, versus 118 s for the panel’s package: moving the 40 thousand files alone took 66 s. Attempt 3: as soon as the account exists, Salvo writes the site and email straight into place, once, and imports the databases at the same time. 80.6 s, versus 126 s for the panel’s package: 36% faster, with the same result. The file-by-file check found two defects in the new method, and both were fixed before the final measurements: skeleton files that did not go back over the ones the panel creates (133 of 140), and the mailboxes of an old domain on one account with the wrong group on two files, which would have left its email offline. That account was redone after the fix: 56 s, versus 68 s for the panel’s package. With one account solved, many at a time: 1, 3 and 6. The new default is 3, the balance between time and load. Planned: Salvo choosing the path by itself from the account size, because on small accounts the panel’s package ties.