I feel this is fixating on the wrong problem. Even with immutable flags there are various ways an attacker with root access could, after getting what they want from the system, cover their tracks by trashing the whole system⁰, and as usual if someone has physical access all bets are off. I see filesystem level flags like that to be more tools to stop you or a bug accidentally doing something stupid, than to get in the way of a malicious action by someone else.
While the standard might effectively call for immutable logs¹, he needs to read between the lines one step further: those logs do not need to be on the same machine. You could stream logs to another system that stores them immutably from the PoV of anyone except those with root or physical access to it. You still have a problem if an attacker gets access to both the source system(s) and the log sinks², there might be a latency issue meaning you could easily lose the last few log entries in the case of a complete disaster, and you have an extra moving part in your infrastructure to monitor, but it satisfies the requirement where immutable filesystem flags can not.
----
[0] Yes, you'll know something happened, and you might guess it was malicious and not random corruption, but enough tracks might be covered to stop you working out the initial who & how.
[1] and some standars explicitly call for them
[2] Careful granular access management should largely mitigate that risk. That could be a problem if you are a small organisation trying to protect against internal disgruntled admins³, but you could use a a 3rd party log-sink service in that case.
[3] This may seem overly paranoid, but if it is required for the standard your target audience wants you to have a certificate for…, and TBH it isn't that paranoid.
khaki54 · 11m ago
Yep use syslog server or similar in conjuction with this, which basically gives you something like immutability since the data is on a remote server with hopefully different security controls. You really don't want to be trying to sort an attack out after the fact on attacker-controlled machine. They could of course turn off network links or syslog eventually, but you'd at least have the early stages of the attack and or perhaps be able to detect it before they actually get full access.
Rygian · 6h ago
That point should not require "reading between the lines" and that's why other standards (e.g. PCI) require explicitly that the logs are sent to a separate "central server" that provides guarantees of immutability.
JdeBP · 5h ago
Indeed. That was exactly what I was thinking when I read the article, from experience of PCI compliance as a matter of fact. And clearly from comments here a lot of people are thinking the same. It may be a fun "Look! OpenBSD can do something!" thing, but the reality is that defence against the dark arts goes a lot deeper, and (as ever) one often has to read more than one standard/specification. (-:
anthk · 2h ago
Linux has chattr, similar features.
tptacek · 3h ago
I came here to say the same thing; trying to make logs immutable on an attacker-controlled machine feels like a very 2000s-era OpenBSD thing to do.
accrual · 2h ago
I enjoyed reading the article, I didn't realize I could have this layer of immutability on my OpenBSD systems so easily. But after reading the comments here, indeed the real solution is to export the logs to a central server in another security domain à la PCI requirements.
On the other hand it's great to have documentation like this. I feel there's a gradient between convenience and security and immutable local logs could provide a layer of defense without requiring another server for logging. Maybe a "nice to have" for a small homelab, security practice, etc.
johnisgood · 8h ago
It is not the same, but I do use "chattr +i" on a file (which applies the immutable attribute) on Linux to a file that otherwise would have been overwritten by programs that do not give a damn whether I want it to or not, and in my case it was easier to just make that file immutable, mainly: /etc/resolv.conf.
claviola · 2h ago
Are you using systemd-resolved? If so, it's actually a symlink to /run/systemd/resolve/stub-resolv.conf, and all you need to do to manage it yourself is to turn it into a regular file of your own. This is explained in the admittedly very long commented out text above the actual directives.
johnisgood · 1h ago
No, it is a runit-based system. I think the issue was related to resolvconf + package updates overwriting the file.
That said, I am sure your comment will be useful to some!
mmsc · 6h ago
.bash_history
mzajc · 6h ago
Better yet, `shopt -s 'histappend'` in your .bashrc and `chattr +a .bash_history`. This will still allow bash to add to the history, but it won't be able to trim the file.
mmsc · 2h ago
Yes! And for macOS, `chflags uappnd .bash_history`
comex · 8h ago
> Once the system reaches normal security level, even root cannot tamper with these logs without rebooting into single-user mode
What stops the attacker from just editing /etc/rc.securelevel and then doing a normal reboot?
kstrauser · 7h ago
Make that file immutable so that you can only edit it in single-user mode.
This is definitely one of those “security vs convenience” situations where you can easily shoot yourself in the foot, but it’s great to have the option when you need it.
dgl · 4h ago
Except it is sourced from /etc/rc, and that’s a shell script which obviously depends on the shell and some other pieces. If you want an immutable base you kind of need to make the whole (base) system immutable (and that is possibly best designed as such to start with).
I don’t think this is “security vs convenience”, I’d more argue it’s possible to think you’ve made this secure but you’ve missed something and haven’t configured it to be as secure as you think. An approach like others have suggested with remote logging is at least easier to reason about.
TacticalCoder · 7h ago
> What stops the attacker from just editing /etc/rc.securelevel and then doing a normal reboot?
Certainly a full reboot leaves more tracks than no full reboot? So it's harder to hide?
h43z · 7h ago
Do I understand that correctly that in order for logs to rotate you have to reboot?
jelder · 5h ago
My thoughts exactly. And couldn’t an attacker just fill the logging volume with uninteresting events to prevent certain other events from being recorded?
gertrunde · 4h ago
That would be where something like auditd would come in, configured so that if the audit logs location runs low on space (or out of space), it will halt the system.
(Yes, quite harsh, but for some use cases it may be the right thing to do, i.e. to fail closed).
jorvi · 5h ago
Log filtering via severity / keywords prevents this, assuming the logs are regularly and properly checked.
eternauta3k · 7h ago
Is root prevented from directly writing to the underlying block device?
kstrauser · 7h ago
Yes.
messe · 6h ago
Only if securelevel is 2. If securelevel = 1, then only mounted filesystems are RO. An attacker could conceivably forcibly unmount /var/log as root, and make the changes directly to the block device.
louwrentius · 7h ago
If you want immutable logs, you log to an external log server. Anything else seems security theater to me.
That log server is properly firewalled/hardened so a hacked server can’t be used as a stepping stone to compromise the log server.
Maybe you even have access restrictions in place for the log server so people can’t wipe their own misdeeds (4-eyes principle).
This is how it’s been done for 35+ years, nothing special about this.
holowoodman · 7h ago
Yes, so much this. It used to be that important logs (filtered by severity and keywords) were even continuously live-printed by a line printer, so that there was always a current paper copy of the really important stuff for forensics.
tsch! tsch! tsch! tsch! "Ah, someone is trying to login as root again"
pjmlp · 5h ago
Exactly the right approach.
mrtesthah · 4h ago
macOS has always had these chflags attributes, by the way, long before System Integrity Protection existed. Changing or removing the system-immutable/append-only flags required booting into single user mode. Even the macOS installer application itself was unable to clear them otherwise.
spauldo · 29m ago
That's not surprising given MacOS' lineage.
bananapub · 6h ago
immutable is a handy advisory feature, but the actual answer for log tampering is "get them off the box in to a different security domain", e.g. a log server this machine can't access and is securely backed up so logs that make it there can be fairly well trusted.
While the standard might effectively call for immutable logs¹, he needs to read between the lines one step further: those logs do not need to be on the same machine. You could stream logs to another system that stores them immutably from the PoV of anyone except those with root or physical access to it. You still have a problem if an attacker gets access to both the source system(s) and the log sinks², there might be a latency issue meaning you could easily lose the last few log entries in the case of a complete disaster, and you have an extra moving part in your infrastructure to monitor, but it satisfies the requirement where immutable filesystem flags can not.
----
[0] Yes, you'll know something happened, and you might guess it was malicious and not random corruption, but enough tracks might be covered to stop you working out the initial who & how.
[1] and some standars explicitly call for them
[2] Careful granular access management should largely mitigate that risk. That could be a problem if you are a small organisation trying to protect against internal disgruntled admins³, but you could use a a 3rd party log-sink service in that case.
[3] This may seem overly paranoid, but if it is required for the standard your target audience wants you to have a certificate for…, and TBH it isn't that paranoid.
On the other hand it's great to have documentation like this. I feel there's a gradient between convenience and security and immutable local logs could provide a layer of defense without requiring another server for logging. Maybe a "nice to have" for a small homelab, security practice, etc.
That said, I am sure your comment will be useful to some!
What stops the attacker from just editing /etc/rc.securelevel and then doing a normal reboot?
This is definitely one of those “security vs convenience” situations where you can easily shoot yourself in the foot, but it’s great to have the option when you need it.
I don’t think this is “security vs convenience”, I’d more argue it’s possible to think you’ve made this secure but you’ve missed something and haven’t configured it to be as secure as you think. An approach like others have suggested with remote logging is at least easier to reason about.
Certainly a full reboot leaves more tracks than no full reboot? So it's harder to hide?
(Yes, quite harsh, but for some use cases it may be the right thing to do, i.e. to fail closed).
That log server is properly firewalled/hardened so a hacked server can’t be used as a stepping stone to compromise the log server.
Maybe you even have access restrictions in place for the log server so people can’t wipe their own misdeeds (4-eyes principle).
This is how it’s been done for 35+ years, nothing special about this.
See e.g. https://www.youtube.com/watch?v=FiEGoVzmyvs but dot-matrix was also used and at least a little less noisy.