> One way is to ensure that machines that must be backed up via "push" [..] can only access their own space. More importantly, the backup server, for security reasons, should maintain its own filesystem snapshots for a certain period. In this way, even in the worst-case scenario (workload compromised -> connection to backup server -> deletion of backups to demand a ransom), the backup server has its own snapshots
My preferred solution is to let client only write new backups, never delete. The deletion is handled separately (manually or cron on the target).
You can do this with rsync/ssh via the allowed command feature in .ssh/authorized_keys.
3eb7988a1663 · 38m ago
I fall into the "pull" camp so this is less of a worry. The server to be backed-up should have no permissions to the backup server. If an attacker can root your live server (with more code/services to exploit), they do not automatically also gain access to the backup system.
haiku2077 · 1h ago
This is also why I use rclone copy instead of rclone sync for my backups, using API keys without permission to delete objects.
binwiederhier · 1h ago
Thank you for sharing. A curious read. I am looking forward to the next post.
I've been working on backup and disaster recovery software for 10 years. There's a common phrase in our realm that I feel obligated to share, given the nature of this article.
> "Friends don't let friends build their own Backup and Disaster Recovery (BCDR) solution"
Building BCDR is notoriously difficult and has many gotchas. The author hinted at some of them, but maybe let me try to drive some of them home.
- Backup is not disaster recovery: In case of a disaster, you want to be up and running near-instantly. If you cannot get back up and running in a few minutes/hours, your customers will lose your trust and your business will hurt. Being able to restore a system (file server, database, domain controller) with minimal data loss (<1 hr) is vital for the survival of many businesses. See Recovery Time Objective (RTO) and Recovery Point Objective (RPO).
- Point-in-time backups (crash consistent vs application consistent): A proper backup system should support point-in-time backups. An "rsync copy" of a file system is not a point-in-time backup (unless the system is offline), because the system changes constantly. A point-in-time backup is a backup in which each block/file/.. maps to the same exact timestamp. We typically differentiate between "crash consistent backups" which are similar to pulling the plug on a running computer, and "application consistent backups", which involves asking all important applications to persist their state to disk and freeze operations while the backup is happening. Application consistent backups (which is provided by Microsoft's VSS, as mentioned by the author) significantly reduce the chances of corruption. You should never trust an "rsync copy" or even crash consistent backups.
- Murphy's law is really true for storage media: My parents put their backups on external hard drives, and all of r/DataHoarder seems to buy only 12T HDDs and put them in a RAID0. In my experience, hard drives of all kinds fail all the time (though NVMe SSD > other SSD > HDD), so having backups in multiple places (3-2-1 backup!) is important.
(I have more stuff I wanted to write down, but it's late and the kids will be up early.)
sandreas · 2h ago
Nice writeup... Although I'm missing a few points...
In my opinion a good backup (system) is only good, if it has been tested to be restorable as fast as possible and the procedure is clear (like in documented).
How often have I heard or seen backups that "work great" and "oh, no problem we have them" only to see them fail or take ages to restore, when the disaster has happened (2 days can be an expensive amount of time in a production environment). Quite too often only parts could be restored.
Another missing aspect is within the snapshots section... I like restic, which provides repository based backup with deduplicated snapshots for FILES (not filesystems). It's pretty much what you want if you don't have ZFS (or other reliable snapshot based filesystems) to keep different versions of your files that have been deleted on the filesystem.
The last aspect is partly mentioned, the better PULL than PUSH part. Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything... So you could either use readonly media (like Blurays) or PULL is mandatory. It is also helpful to have auto-snapshotting on ZFS via zfs-auto-snapshot, zrepl or sanoid to go back in time to where the ransomware has started its journey.
marcusb · 20m ago
> Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything
That depends on how you have access to your backup servers configured. I'm comfortable with append-only backup enforcement for push backups[0] with Borg and Restic via SSH, although I do use offline backup drive rotation as a last line of defense for my local backup set. YMMV.
Since you mentioned restic, is there something wrong with using restic append-only with occasional on-server pruning instead of pulling? I thought this was the recommended way of avoiding ransomware problems using restic.
> So you could either use readonly media (like Blurays) or PULL is mandatory.
Or like someone already commented you can use a server that allows push but doesn't allow to mess with older files. You can for example restrict ssh to only the scp command and the ssh server can moreover offer a chroot'ed environment to which scp shall copy the backups. And the server can for example daily rotate that chroot.
The push can then push one thing: daily backups. It cannot log in. It cannot overwrite older backups.
Short of a serious SSH exploit where the ransomware could both re-configure the server to accept all ssh (and not just scp) and escape the chroot box, the ransomware is simply not destroying data from before the ransomware found its way on the system.
My backup procedure does that for the one backup server that I have on a dedicated server: a chroot'ed ssh server that only accepts scp and nothing else. It's of course just one part of the backup procedure, not the only thing I rely on for backups.
P.S: it's not incompatible with also using read-only media
anonymars · 1h ago
I don't understand why this is dead..is it wrong advice? Is there some hidden flaw? Is it simply because the content is repeated elsewhere?
On the face of it "append-only access (no changes)" seems sound to me
quesera · 51m ago
TacticalCoder's comments appear to be auto-deaded for the last week or so.
I did not see a likely reason in a quick review of their comment history.
You can view a comment directly by following the "... ago" link, and from there you can use the "vouch" link to revive the comment. I vouched for a few of TacticalCoder's recent comments.
bambax · 3h ago
It's endlessly surprising how people don't care / don't think about backups. And not just individuals! Large companies too.
I'm consulting for a company that makes around €1 billion annual turnover. They don't make their own backups. They rely on disk copies made by the datacenter operator, which happen randomly, and which they don't test themselves.
Recently a user error caused the production database to be destroyed. The most recent "backup" was four days old. Then we had to replay all transactions that happened during those four days. It's insane.
But the most insane part was, nobody was shocked or terrified about the incident. "Business as usual" it seems.
polishdude20 · 3h ago
If it doesn't affect your bottom line enough to do it right, then I guess it's ok?
rapfaria · 31m ago
I'd go even a step further: For the big corp, having a point of failure that lives outside its structure can be a feature, and not a bug.
"Oh there goes Super Entrepise DB Partner again" turns into a product next fiscal year, that shutdowns the following year because the scope was too big, but at least they tried to make things better.
treetalker · 2h ago
Possibly for legal purposes? Litigation holds are a PITA and generators of additional liability exposure, and backups can come back to bite you.
haiku2077 · 1h ago
Companies that big have legal requirements to keep much of their data around for 5-7 years anyway.
daneel_w · 2h ago
It's also endlessly surprising how people over-think the process and requirements.
Shank · 23m ago
> Security: I avoid using mainstream cloud storage services like Dropbox or Google Drive for primary backups. Own your data!
What does this have to do with security? You shouldn't be backing up data in a way that's visible to the server. Use something like restic. Do not rely on the provider having good security.
rr808 · 3h ago
I dont need a backup system. I just need a standardized way to keep 25 years of photos for a family of 4 with their own phones, cameras, downloads, scans etc. I still haven't found anything good.
xandrius · 2h ago
Downloads and scans are generally trash unless deemed important.
For the phones and cameras, setup Nextcloud and have it automatically sync to your own home network. Then have a nightly backup to another disk with a health check after it finishes.
After that you can pick either a cloud host which your trust or get another drive of ours into someone else's server to have another locstion for your 2nd backup and you're golden.
sandreas · 2h ago
I use syncthing... it's great for that purpose, Android is not officially supported but there is a fork, that works fine. Maybe you want to combine it with either ente.io or immich (also available for self-hosted) for photo backup.
I would also distinguish between documents (like PDF and TIFF) and photos - there is also paperless ngx.
bravesoul2 · 1h ago
Isn't that like a Dropbox approach? If you have 2tb photos this means you need 2tb storage on everything?
setopt · 2h ago
I like Syncthing but it’s not a great option on iOS.
It's an option... But still beholden to the arbitrary restriction apple has on data access.
bravesoul2 · 1h ago
Struggling too.
For me one win/mac with backblaze. Dump everything to that machine. Second ext. Drive backup just in case.
rsolva · 2h ago
Check out ente.io - it is really good!
nor-and-or-not · 1h ago
I second that, and you can even self-host it.
palata · 2h ago
I recently found that Nextcloud is good enough to "collect" the photos from my family onto my NAS. And my NAS makes encrypted backups to a cloud using restic.
haiku2077 · 1h ago
A NAS running Immich, maybe?
bambax · 3h ago
You do need a backup. But before that, you need a family NAS. There are plenty of options. (But a NAS is not a backup.)
daneel_w · 2h ago
My valuable data is less than 100 MiB. I just tar+compress+encrypt a few select directories/files twice a week and keep a couple of months of rotation. No incremental hassle necessary. I store copies at home and I store copies outside of home. It's a no-frills setup that costs nothing, is just a few lines of *sh script, takes care of itself, and never really needed any maintenance.
mavilia · 2h ago
This comment made me rethink what I have that is actually valuable data. My photos alone even if culled down to just my favorites would probably be at least a few gigs. Contacts from my phone would be small. Other than that I guess I wouldn't be devastated if I lost anything else. Probably should put my recovery keys somewhere safer but honestly the accounts most important to me don't have recovery keys.
Curious what you consider valuable data?
Edit: I should say for pictues I have around 2Tb right now (downside of being a hobby photographer)
daneel_w · 1h ago
With valuable I should've elaborated that it's my set of constantly changing daily-use data. Keychain, documents and notes, e-mail, bookmarks, active software projects, those kinds of things.
I have a large amount of memories and "mathom" as well, in double copies, but I connect and add to this data so rarely that it absolutely does not have to be part of any ongoing backup plan.
bob1029 · 1h ago
I think the cleanest, most compelling backup strategies are those employed by RDBMS products. [A]sync log replication is really powerful at taking any arbitrary domain and making sure it exists in the other sites exactly.
You might think this is unsuitable for your photo/music/etc. collection, but there's no technical reason you couldn't use the database as the primary storage mechanism. SQLite will take you to ~281 terabytes with a 64k page size. MSSQL supports something crazy like 500 petabytes. The blob data types will choke on your 8k avengers rip, but you could store it in 1 gig chunks - There are probably other benefits to this anyways.
kernc · 1h ago
Make your own backup system—is exactly what I did. I felt git porcelain had a stable-enough API to accommodate this popular use case.
How data changes, and what changes it, matters when trying to optimize backups.
A full OS installation may not change a lot, or change with security updates that anyway are stored elsewhere.
Configurations have their own lifecycle, actors, and good practices on how to keep and backup them. Same with code.
Data is what matters if you have saved somewhat everything else. And it could have a different treatment file tree backups from I.e. database backups.
Logs is something that frequently changes, but you can have a proper log server for which logs are data.
Things can be this granular, or go for storage backup. But the granularity, while may add complexity, may lower costs and increase how much of what matters you can store for longer periods of time.
o11c · 1h ago
Other things that matter (some overlap):
* Is the file userland-compressed, filesystem-or-device-compressed, or uncompressed?
* What are you going to do about secret keys?
* Is the file immutable, replace-only (most files), append-only (not limited to logs; beware the need to defrag these), or fully mutable (rare - mostly databases or dangerous archive software)?
* Can you rely on page size for (some) chunking, or do you need to rely entirely on content-based chunking?
* How exactly are you going to garbage-collect the data from no-longer-active backups?
* Does your filesystem expose an accurate "this file changed" signal, or better an actual hash? Does it support chunk sharing? Do you know how those APIs work?
* Are you crossing a kernel version that is one-way incompatible?
* Do you have control of the raw filesystem at the other side? (e.g. the most efficient backup for btrfs is only possible with this)
My preferred solution is to let client only write new backups, never delete. The deletion is handled separately (manually or cron on the target).
You can do this with rsync/ssh via the allowed command feature in .ssh/authorized_keys.
I've been working on backup and disaster recovery software for 10 years. There's a common phrase in our realm that I feel obligated to share, given the nature of this article.
> "Friends don't let friends build their own Backup and Disaster Recovery (BCDR) solution"
Building BCDR is notoriously difficult and has many gotchas. The author hinted at some of them, but maybe let me try to drive some of them home.
- Backup is not disaster recovery: In case of a disaster, you want to be up and running near-instantly. If you cannot get back up and running in a few minutes/hours, your customers will lose your trust and your business will hurt. Being able to restore a system (file server, database, domain controller) with minimal data loss (<1 hr) is vital for the survival of many businesses. See Recovery Time Objective (RTO) and Recovery Point Objective (RPO).
- Point-in-time backups (crash consistent vs application consistent): A proper backup system should support point-in-time backups. An "rsync copy" of a file system is not a point-in-time backup (unless the system is offline), because the system changes constantly. A point-in-time backup is a backup in which each block/file/.. maps to the same exact timestamp. We typically differentiate between "crash consistent backups" which are similar to pulling the plug on a running computer, and "application consistent backups", which involves asking all important applications to persist their state to disk and freeze operations while the backup is happening. Application consistent backups (which is provided by Microsoft's VSS, as mentioned by the author) significantly reduce the chances of corruption. You should never trust an "rsync copy" or even crash consistent backups.
- Murphy's law is really true for storage media: My parents put their backups on external hard drives, and all of r/DataHoarder seems to buy only 12T HDDs and put them in a RAID0. In my experience, hard drives of all kinds fail all the time (though NVMe SSD > other SSD > HDD), so having backups in multiple places (3-2-1 backup!) is important.
(I have more stuff I wanted to write down, but it's late and the kids will be up early.)
In my opinion a good backup (system) is only good, if it has been tested to be restorable as fast as possible and the procedure is clear (like in documented).
How often have I heard or seen backups that "work great" and "oh, no problem we have them" only to see them fail or take ages to restore, when the disaster has happened (2 days can be an expensive amount of time in a production environment). Quite too often only parts could be restored.
Another missing aspect is within the snapshots section... I like restic, which provides repository based backup with deduplicated snapshots for FILES (not filesystems). It's pretty much what you want if you don't have ZFS (or other reliable snapshot based filesystems) to keep different versions of your files that have been deleted on the filesystem.
The last aspect is partly mentioned, the better PULL than PUSH part. Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything... So you could either use readonly media (like Blurays) or PULL is mandatory. It is also helpful to have auto-snapshotting on ZFS via zfs-auto-snapshot, zrepl or sanoid to go back in time to where the ransomware has started its journey.
That depends on how you have access to your backup servers configured. I'm comfortable with append-only backup enforcement for push backups[0] with Borg and Restic via SSH, although I do use offline backup drive rotation as a last line of defense for my local backup set. YMMV.
0 - https://marcusb.org/posts/2024/07/ransomware-resistant-backu...
Or like someone already commented you can use a server that allows push but doesn't allow to mess with older files. You can for example restrict ssh to only the scp command and the ssh server can moreover offer a chroot'ed environment to which scp shall copy the backups. And the server can for example daily rotate that chroot.
The push can then push one thing: daily backups. It cannot log in. It cannot overwrite older backups.
Short of a serious SSH exploit where the ransomware could both re-configure the server to accept all ssh (and not just scp) and escape the chroot box, the ransomware is simply not destroying data from before the ransomware found its way on the system.
My backup procedure does that for the one backup server that I have on a dedicated server: a chroot'ed ssh server that only accepts scp and nothing else. It's of course just one part of the backup procedure, not the only thing I rely on for backups.
P.S: it's not incompatible with also using read-only media
On the face of it "append-only access (no changes)" seems sound to me
I did not see a likely reason in a quick review of their comment history.
You can view a comment directly by following the "... ago" link, and from there you can use the "vouch" link to revive the comment. I vouched for a few of TacticalCoder's recent comments.
I'm consulting for a company that makes around €1 billion annual turnover. They don't make their own backups. They rely on disk copies made by the datacenter operator, which happen randomly, and which they don't test themselves.
Recently a user error caused the production database to be destroyed. The most recent "backup" was four days old. Then we had to replay all transactions that happened during those four days. It's insane.
But the most insane part was, nobody was shocked or terrified about the incident. "Business as usual" it seems.
"Oh there goes Super Entrepise DB Partner again" turns into a product next fiscal year, that shutdowns the following year because the scope was too big, but at least they tried to make things better.
What does this have to do with security? You shouldn't be backing up data in a way that's visible to the server. Use something like restic. Do not rely on the provider having good security.
For the phones and cameras, setup Nextcloud and have it automatically sync to your own home network. Then have a nightly backup to another disk with a health check after it finishes.
After that you can pick either a cloud host which your trust or get another drive of ours into someone else's server to have another locstion for your 2nd backup and you're golden.
I would also distinguish between documents (like PDF and TIFF) and photos - there is also paperless ngx.
https://mobiussync.com/
For me one win/mac with backblaze. Dump everything to that machine. Second ext. Drive backup just in case.
Curious what you consider valuable data?
Edit: I should say for pictues I have around 2Tb right now (downside of being a hobby photographer)
I have a large amount of memories and "mathom" as well, in double copies, but I connect and add to this data so rarely that it absolutely does not have to be part of any ongoing backup plan.
You might think this is unsuitable for your photo/music/etc. collection, but there's no technical reason you couldn't use the database as the primary storage mechanism. SQLite will take you to ~281 terabytes with a 64k page size. MSSQL supports something crazy like 500 petabytes. The blob data types will choke on your 8k avengers rip, but you could store it in 1 gig chunks - There are probably other benefits to this anyways.
https://kernc.github.io/myba/
A full OS installation may not change a lot, or change with security updates that anyway are stored elsewhere.
Configurations have their own lifecycle, actors, and good practices on how to keep and backup them. Same with code.
Data is what matters if you have saved somewhat everything else. And it could have a different treatment file tree backups from I.e. database backups.
Logs is something that frequently changes, but you can have a proper log server for which logs are data.
Things can be this granular, or go for storage backup. But the granularity, while may add complexity, may lower costs and increase how much of what matters you can store for longer periods of time.
* Is the file userland-compressed, filesystem-or-device-compressed, or uncompressed?
* What are you going to do about secret keys?
* Is the file immutable, replace-only (most files), append-only (not limited to logs; beware the need to defrag these), or fully mutable (rare - mostly databases or dangerous archive software)?
* Can you rely on page size for (some) chunking, or do you need to rely entirely on content-based chunking?
* How exactly are you going to garbage-collect the data from no-longer-active backups?
* Does your filesystem expose an accurate "this file changed" signal, or better an actual hash? Does it support chunk sharing? Do you know how those APIs work?
* Are you crossing a kernel version that is one-way incompatible?
* Do you have control of the raw filesystem at the other side? (e.g. the most efficient backup for btrfs is only possible with this)