This is an excellent excellent resource regardless of whether you agree/disagree with the author's conclusions, simply by virtue of being a great list of broken down problems, well described, & accompanied by good technical descriptions of proposed fixes (again independent of your opinion on those fixes).
Just an excellent example of how to approach & elucidate a problem domain.
h1fra · 1h ago
I think the big problem is when secrets can be anywhere in a string and you don't control the input (e.g, library stacktraces, HTTP responses, JSON that was stringified).
You need to pass the secrets to the logger so it can be redacted, it's heavily dependent on the dev and easy to forget during review.
And an exact match is just part of the problem; if a dev redacts the end and another dev redacts the start, you can still reassemble the secret with enough logs.
Bender · 57m ago
One direction to venture would be running rsyslog on every node, using regex to match all the known patterns and use various plugins/addons to send all the applications to the local rsyslog instance using a local spooler and then encrypt the rsyslog upstream to centralized logging servers. Rsyslog supports using a spooler so that if the up-stream server is offline for whatever reason the logs are spooled locally and then resume when upstream is online.
Regex matching on logs is slow but if performed on every node the CPU load is distributed vs. doing this upstream. Configuration management can push the regex rules to all the nodes. This won't help with unknown-unknowns but those can be added quickly to all nodes through configuration management after peer review.
Rsyslog also supports encrypting the log stream so that secret leakage is limited to the sending nodes and the central nodes and it checks a few boxes.
Another thing that helps is limiting to warn and above sent upstream and using an agent on the local nodes to monitor for keywords in the range of info to debug to let someone know to go check the node logs. Less junk on the centralized servers that may have SOC1/SOC2/PCI/FEDRAMP log retention requirements. One can not leak what is not sent in the first place.
dataflow · 2h ago
As far as run-time exposure prevention goes, I feel like in-band signaling might work better than out-of-band for this problem. Along the lines of the taint checking technique mentioned, you can insert some magic string (say, some recognizable prefix + a randomly generated UUID) into your sensitive strings at the source, that you then strip out at the sink. (Or wrap your secrets in a pair of such magic strings.) Then block or mask any strings containing that magic string from making it into any persisted data, including logs. And it will be easy to identify the points of exposure, since they will be wherever you call your respective seal()/unseal() function or such.
mgaunard · 2h ago
One particular thing to be careful of are core dumps.
What I did at a previous shop was remove the passwords as part of a smart gdb script that runs when the core is dumped, before it gets written to a readable location.
Writing the script also helped to demonstrate how to extract the passwords in the first place.
kjs3 · 1h ago
Stack traces, too. I did some work with a heavy Java shop and pretty much everything sensitive ended up in a stack trace at some point.
carlmr · 44m ago
Java is just too verbose in every possible way.
bilalq · 21m ago
This is an excellent write-up of the problem. New hires out of college/bootcamps often have no awareness of the risks here at all. Sometimes even engineers with years of experience but no operational mentorship in their career.
The kitchen sink example in particular is one that trips up people. Without knowing the specifics of how a library may deal with failure edge cases, it can catch you off guard (e.g., axios errors including API key headers).
A lot of these problems come from architectures where secrets go over the wire instead of just using signatures/ids. But in cases where you have to use some third party platform, there's often no choice.
munchler · 3h ago
I certainly agree with the desire to keep secrets out of logs, but isn’t the entire log itself also considered to be secret? Even a perfectly sanitized log probably contains lots of data about your production environment that you wouldn’t want to share with adversaries (e.g. peak usage hours).
advisedwang · 3h ago
Logs probably need to be exposed to support teams, oncalls for sister-teams (if you are a large org), all your devs etc. That is many MANY more people than need access to secrets. Secrets in logs therefore puts you are much wider risk of internal threats and makes it MUCH easier for an attacker who phishes someone to pivot to higher credentials.
Also if you have audit records, you want accessing a secret to be logged separately from accessing logs.
jauer · 3h ago
There’s secret from an adversary and then there’s internal compartmentalization.
You could have 100s of people who have a business need to look at syslog from a router, but approximately nobody who should have access to login creds of administrative users and maybe 10s of people with access to automation role account creds.
dmurray · 3h ago
Yes, but think defense in depth. Your team member who leaves for a competitor could tell them your peak usage hours, but he shouldn't be able to tell them all your customers' passwords.
> And while people will write the code that accidentally introduces sensitive data into logs, they’re also the ones that will report, respond, and fix them.
This should probably be the first point and not the last.
blkhawk · 2h ago
oh god - I had that come up in an issue at work just about a month ago. A development system used really simple usernames and passwords since it was just for testing but all the lines with one of those got gobbled up because they had "secrets" in them.
I have very strong opinions on this issue that boils down to. _why are you logging everything you lazy asses_ and _adding all the secrets into another tool just to scan for them in logs just adds another point for them to leak_...
Especially since the ability of lines getting censored even when the secrets were just part of words showed that probably no hashing was involved.
But its a security tool so it stays. I kinda feel like Cassandra but I think I can already predict a major security issue with it or others with the same functionality in the future. its like some goddamn blind spot that software that is to prevent X cannot be vulnerable to X but somehow often is vulnerable because prevention of X and not being vulnerable to X are two separate things somehow.
pavel_lishin · 2h ago
Why is logging everything considered lazy?
tonymet · 1h ago
for one it's extremely costly, in vcpu , storage , transfer rates. and if you're paying a third-party logger , multiply each by 10x
shakna · 1h ago
If you're in a testing environment, where your SIT and UAT are looking to break stuff though, don't you usually want to be able to look to a log of everything?
tonymet · 44m ago
I could see a couple reasons against. For one, it's expensive to seralize/encode your objects into the logger , even if you reduce logging level on prod.
Secondly, you can't represent the heap & stack well as strings. Concurrent threads and object trees are better debugged with a debugger (e.g. gdb).
pavel_lishin · 1h ago
That makes it foolish, but I'm not sure if it's lazy.
tonymet · 46m ago
the lazy part comes from the fact that it's easier to be foolish in this case than to be selective about what gets logged. So lazy & foolish.
NeutralForest · 3h ago
Just excellent. Lots of (common from my experience) examples, potential fixes and self-contained explanations. Nice.
debarshri · 2h ago
I think secrets ending up in the log is an issue but who should have access to view logs of what log should also be an important that is often ignored. This is also scope down the surface area of leakage.
stretchwithme · 53m ago
A user's password is something I shouldn't see in a log, even if I'm in control of what gets logged and frequently access them to do my job.
Even if I trust me.
Audits happen. I assume other people will eventually see this bad practice.
aduwah · 3h ago
Great article!
I will definitely reference it in my upcoming discussions. I had some hard time defending having an EU based o11y stack for our EU based infra. I found it hard to articulate on the spot that there are myriads of places where sensitive/personal data can get in the logs and cause leaks, or make GDPR angry.
jazzyjackson · 2h ago
Why do I have to know how many letters are in observability? is this some kind of in group signaling?
That presumes you know all secrets ahead of time. A risk in and of itself. But from a practical point of view you will never know all secrets, because they are generated constantly in real time.
Just an excellent example of how to approach & elucidate a problem domain.
And an exact match is just part of the problem; if a dev redacts the end and another dev redacts the start, you can still reassemble the secret with enough logs.
Regex matching on logs is slow but if performed on every node the CPU load is distributed vs. doing this upstream. Configuration management can push the regex rules to all the nodes. This won't help with unknown-unknowns but those can be added quickly to all nodes through configuration management after peer review.
Rsyslog also supports encrypting the log stream so that secret leakage is limited to the sending nodes and the central nodes and it checks a few boxes.
Another thing that helps is limiting to warn and above sent upstream and using an agent on the local nodes to monitor for keywords in the range of info to debug to let someone know to go check the node logs. Less junk on the centralized servers that may have SOC1/SOC2/PCI/FEDRAMP log retention requirements. One can not leak what is not sent in the first place.
What I did at a previous shop was remove the passwords as part of a smart gdb script that runs when the core is dumped, before it gets written to a readable location.
Writing the script also helped to demonstrate how to extract the passwords in the first place.
The kitchen sink example in particular is one that trips up people. Without knowing the specifics of how a library may deal with failure edge cases, it can catch you off guard (e.g., axios errors including API key headers).
A lot of these problems come from architectures where secrets go over the wire instead of just using signatures/ids. But in cases where you have to use some third party platform, there's often no choice.
Also if you have audit records, you want accessing a secret to be logged separately from accessing logs.
You could have 100s of people who have a business need to look at syslog from a router, but approximately nobody who should have access to login creds of administrative users and maybe 10s of people with access to automation role account creds.
> And while people will write the code that accidentally introduces sensitive data into logs, they’re also the ones that will report, respond, and fix them.
This should probably be the first point and not the last.
I have very strong opinions on this issue that boils down to. _why are you logging everything you lazy asses_ and _adding all the secrets into another tool just to scan for them in logs just adds another point for them to leak_...
Especially since the ability of lines getting censored even when the secrets were just part of words showed that probably no hashing was involved.
But its a security tool so it stays. I kinda feel like Cassandra but I think I can already predict a major security issue with it or others with the same functionality in the future. its like some goddamn blind spot that software that is to prevent X cannot be vulnerable to X but somehow often is vulnerable because prevention of X and not being vulnerable to X are two separate things somehow.
Secondly, you can't represent the heap & stack well as strings. Concurrent threads and object trees are better debugged with a debugger (e.g. gdb).
Even if I trust me.
Audits happen. I assume other people will eventually see this bad practice.
secrets.forEach(secret => logMessage = logMessage.replaceAll(secret, '**'))