The server went down, or someone deleted the accounts: the customer wants the WordPress site back, and the whole email, including the messages they haven’t read yet. With real customer accounts (WordPress sites, shops, real mailboxes), what happens when you back up, lose everything and restore, with JetBackup and with Salvo, on the same server and the same storage?
Why it matters to whoever runs the server
- Email is where the customer notices first. Getting the account back without unread messages, spam and trash turns into a ticket the same day. On Salvo, all 19,667 messages came back; on JetBackup, 648 were missing, and they were not in its backup. How we checked every file and message against the original in other restores:See:Lab 16Lab 18Lab 19Lab 21
- Space: the same 6 accounts took 5.26 GB on Salvo and 6.70 GB on JetBackup, 21% less on the very first backup. And the backup finished in 60 s, versus 98 s. To see where the savings come from: compression and one repository per account (Lab 02), what doesn’t need a backup (Lab 10) and 166× fewer files on the storage (Lab 14).See:Lab 02Lab 10Lab 14
- 199 files came back with a different owner or permission on JetBackup, and none on Salvo. A wrong permission is a WordPress site with an error and a customer opening a ticket.
- Where Salvo lost: the restore. One account at a time, waiting for the server load to drop, it took longer on the clock. With 5 at a time, like JetBackup, it came close on time but weighed more on the server. It is all below, with what already changed in Salvo because of it. We were not satisfied, and that led to new improvements. The new options came from Lab 21, and the next step is in Lab 24:See:Lab 21Lab 24
How to read it: grey is JetBackup, blue is Salvo. It is the first backup of the same 6 accounts, both tools with 3 accounts at a time, writing to the same storage. Salvo finished 38 s earlier and still took 21% less space (5.26 GB versus 6.70 GB).
How to read it: each bar is the restore of the 6 accounts. Blue is time spent restoring. Grey is Salvo waiting on purpose: before each account, it waits for the server load to drop, so it does not weigh on other customers’ sites. JetBackup does not wait: it restored 5 accounts together and pushed the load of the 8-core server to 24.5. From the 2nd to the 3rd bar, the new reading options nearly halved the time spent restoring (19 min 06 s to 10 min 35 s); the wait grew because that round fell at the server’s busiest hour.
How to read it: here Salvo worked like JetBackup, 5 accounts together and without waiting for the load to drop. It came in 59 s behind and weighed more on the server: peak load of 37.3, versus 23.0, and 19 GB more disk at peak, versus 7 GB. That is why Salvo does not use 5 at a time as the default. We were not satisfied with these results. They led us to rethink how to restore and to new improvements in Salvo, measured with the same 6 accounts in Lab 24.See:Lab 24
| JetBackup | Salvo | |
|---|---|---|
| 1st backup of the 6 accounts (3 at a time) | 98 s | 60 s |
| Space on the storage | 6.70 GB | 5.26 GB |
| Customer files identical (125,661) | 99.45% | 99.98% |
| Email messages (19,667) | 648 missing | all |
| WordPress database tables (321) | identical | identical |
| Files with a different owner or permission | 199 | 0 |
| Restoring the 6, each with its default | 309 s, 5 at a time | 635 s of work + 833 s waiting for the load to drop, one at a time |
| Peak load in that restore | 24.5 | 23.0 (the server was already at 12) |
| Restoring the 6, both with 5 at a time | 363 s | 422 s |
| Peak load, 5 at a time | 23.0 | 37.3 |
| Extra disk at peak, 5 at a time | +7 GB | +19 GB |
Method
Six accounts from a production server, copied to a cPanel test server with 8 cores: two small (130 to 150 MB, a WordPress site and database), two medium (~1 GB, WordPress, databases, mailboxes and several domains) and two large (~3.2 GB, WordPress, up to 3 databases and 10 to 18 mailboxes).
Before anything, a snapshot of each account, used as the answer key: the hash of every file, the checksum of every table and the messages in every mailbox. That is 125,661 customer files (sites, uploads, messages and mailbox settings), 321 tables and 19,667 messages. Files the system recreates on its own (email indexes, panel caches, logs) are left out of the count.
Backup of the six by JetBackup and by Salvo, one after the other, both tools with 3 accounts at a time, on the same storage. JetBackup ran with the configuration of a real job, with no changes for the test.
Then the six accounts were deleted and restored, comparing with the snapshot each time: by JetBackup (5 at a time, its default); by Salvo with the old options and with the new storage reading options (several connections, the backup fetched whole first), one account at a time; and finally both tools with 5 accounts at a time, one right after the other.
The server was measured from start to end of each phase, every 2 seconds; the space on the storage, at the end of the first backup.
What we found and changed in Salvo during the test
This lab was not a single measurement: each round showed something, and Salvo changed along the way. 1. To set up the test, we had to restore 6 cPanel packages at once, and Salvo did not do that. Now it does: several packages in a queue, from the screen, the command line and the API. 2. The check after the restore warned that messages were missing when the package kept repeated messages as hardlinks. It was a false warning: the messages were there. Fixed, with an automated test so it does not come back. 3. The queue showed an estimated start even while it was waiting for the server load to drop. Fixed. 4. Restoring one account at a time with the old options, Salvo spent more than 12 minutes just reading the backup of the 6 accounts from the storage. With the options born in Lab 21 (several connections and the backup fetched whole first), the same reading dropped to 40 seconds. What is left is the panel recreating each account. 5. Even so, the clock barely moved (25 min 08 s to 24 min 28 s): before each account, the queue waits for the server load to drop, and that round fell at the busiest hour. The wait was longer than the work itself. 6. So we put Salvo under the same conditions as JetBackup, 5 accounts together and no waiting: 422 s versus 363 s, with a load of 37.3 and 19 GB more disk. Not worth it. We were not satisfied with these results. They led us to a new way of restoring and to more improvements in Salvo, which we measured with the same 6 accounts in Lab 24. What did not need to change: in every Salvo round, messages, databases and permissions came back right. What was missing was only what the default exclusion list leaves out on purpose (25 .sql, .zip and .wpress files kept inside the site, and plugin temp folders), and JetBackup leaves those out too.