Ugh, so in one of the replies to my experiments with #BorgBackup and associated tooling, @guerda asked "well, why not #restic?" and turns out, my main reasons against using restic (no compression and no exclude exceptions) are no longer valid, so I'm currently re-evaluating it.
Almost finished my writeup. And you know what? Both are really good.
My decision might boil down to performance against remote hosts and SSH vs SFTP, and whether "rclone serve restic" can save the day there.
@scy@guerda does Restic do some sanity checks on startup, like checking for typical symptoms of a defective RAM module?
I remember the main author of Borg telling me that Borg does that, because he wanted to get rid of related bug reports. I love that level of sophistication 😊
The main thing missing from #restic's compression feature¹ is what #BorgBackup calls "auto": try compressing a small part of the file to see if it makes sense at all (i.e. if the file is compressible), and skip compressing that file if it doesn't. Right now, restic can only compress all files during a run, or none.
¹ other than documentation; it's not cool that I had to dig through the source to find the difference between "auto" and "max" compression, and which algorithm it's using at all
@traumaphoenix Huh! Does it? On my machine, restic with compression "auto" appeared to be significantly slower than Borg with a compression setting of auto,zstd,3, even though restic was running on two cores instead of Borg's one.
Do you have any links about that? And does that also apply to the pure-Go implementation klauspost/compress that restic is using?
@scy don’t know, but there’s a fairly simple check you can do to test for this behavior (note the transfer speed). in restic you’ll want to use dry runs or fresh repos, since it dedupes.
@scy we tried to keep the UI as simple as possible. The Go zstd lib we're using is a noop for incompressible data, therefore we decided against adding a compressibility estimator ourselves.
@fd0 The klauspost/compress docs don't seem to mention that no-op behavior you're mentioning, can you point me to some piece of documentation or even code that confirms this?
I've looked into the design behind what zstd does here for a few minutes (not checking actual code), the behavior seems to be "if a block can't be compressed, don't store it compressed, but continue trying to compress the rest of the file". This is mainly designed to save CPU cycles during decompression, the reasoning being "the beginning of the file could be not representative of the rest".
@scy@traumaphoenix for restic, compression is handled on a "blob" level, which is a chunk of a few megabytes of a file. Due to its architecture, we cannot easily turn compression of just for a file, so each block is compressed individually.
In my tests, compression off and compression auto mostly had the same runtimes, I'm curious for your measurements.
So, I'd say the difference between off and auto on this particular machine is 10 % more time for 10 % compression. "max" however doesn't seem to pay off that much at all.
This is a Celeron 3965Y, two cores at 1.5 GHz, so not massively powerful.
@scy@guerda When evaluating restic, have a look at resticprofile as well. Neat little wrapper with features like profiles (with inheritance), systemd scheduling, Prometheus pushgateway integration, etc.
Add comment