Short answer: on hosting servers, backups usually get slow because of the backup storage, not because of the CPU or the network. The common method copies every site file to the storage as its own file, every night. A server with a few hundred thousand files makes the storage do a few hundred thousand small operations, and that happens again on every incremental backup. We built Salvo to remove that work: it sends a few large packs instead of a tree of small files, and it checks what changed on the source server instead of on the storage.
Where Salvo came from
We run hosting servers, and our problem was the backup storage: the RAID array that receives the backups of many servers was saturated, and backups took too long. Every new server meant more load on a disk array that was already at its limit.
Faster disks were the obvious answer. Instead, we measured where the time goes. The answer was not the amount of data. It was the number of files and the operations each file costs on the storage.
Is this only our problem?
No. The vendor of the most popular hosting backup tool published the same story in 2018: a provider backing up 150 servers and 15,000+ cPanel accounts to one destination. Their notes, in order:
- "There is a major IO & network bottleneck. Having all the servers run at the same time is clearly slowing down the backup process." They widened the backup window from 7 to 12 hours and limited it to 20 servers at a time.
- On the storage, the disks were at their limit, and every server was running
findorcpcommands that "consumed most of the disks' IO". - The final fix was hardware: SSD write cache in front of a 10-disk RAID 5.
Hosting admins describe the same thing in public forums: "our backup server is running backups 24/7", and "the number of inodes plays a bigger role than the data size in terms of backup time".
What a hardlink snapshot costs the storage
The common file-based method (rsync with hardlinks) keeps one copy of each site file on the storage. Every incremental backup builds a new snapshot: a new folder for every folder of every account, and a hardlink for every file that didn't change. To decide what changed, rsync checks each file against the previous copy on the storage.
That means the storage does work proportional to the number of files, every night, even when almost nothing changed. Retention adds more: deleting an old snapshot means unlinking hundreds of thousands of entries.
What we measured
Same production cPanel server, same 10 accounts (about 55 GB and 660 thousand files), same storage, each tool on its own empty volume, measured every 5 seconds (Lab 14 and Lab 15):
| On the storage | Hardlink snapshots (JetBackup, SSH destination) | Salvo |
|---|---|---|
| Files and folders created by the first backup | 701,685 | 4,217 |
| Average write latency during the first backup | 90 ms | 42 ms |
| Disk reads during the next incremental | 23,580 | 155 |
| New files and folders from that incremental | +46,044 | +221 |
| Time of that incremental | 3 min 57 s | 1 min 28 s |
The storage is where the difference is largest: 166× fewer files and 152× fewer disk reads. Fewer files also means less to check and less to delete when retention runs.
How Salvo avoids the work
- Few large packs instead of a file tree. Salvo groups the account's data into packs before sending. The storage receives a few thousand objects, not hundreds of thousands.
- The check happens on the source server. Salvo keeps the file list from the previous backup and skips, without opening them, the files whose metadata shows they didn't change. In Lab 15 it read 369 of 662 thousand files. The storage practically only receives writes.
- Compressed and encrypted before leaving the server. Less data on the network and on the disks.
What Salvo costs in exchange
We publish the downsides too. Compression costs CPU on the source server: with 3 large accounts at once, load reached 23 on an 8-core server, against 14 for JetBackup, and Salvo used up to 1.3 GB of memory against 50 MB (Lab 14). Salvo runs at the lowest priority, so sites go first, but the load is real.
What this means for your backups
If your backup storage is always busy, check three numbers before buying disks:
- How many files your backups create on the storage per night (not GB).
- How many disk reads the storage does during an incremental, when almost nothing changed.
- How long retention cleanup takes.
If those numbers grow with the number of files, the bottleneck is the method, not the hardware.
Frequently asked questions
Why are my cPanel backups taking so long? Usually because of the number of files, not the size. File-based incremental backups check and link every file on the storage, every night.
Will faster disks fix it? They help, as the 2018 case shows, but the work per file stays. When you add servers, it grows again.
Is there an alternative to JetBackup that reduces storage load? Salvo is being built for that, and it is designed to run alongside JetBackup during a transition. It's in development, with a closed beta coming; every test we run is published in the Lab, with the method and what we don't know yet.
Salvo is in development. Join the waitlist to test it on your servers.
Sources: Lab 14 · Lab 15 · "A true story of a mission impossible", JetApps on the cPanel blog, 2018 (archived) · cPanel forum: "Worth switching to Jetbackup?" · LowEndTalk, 2024