I think that Security fuckups of this disastrous scale should get classified as "breaches" or "incidents" and be required to be publicly disclosed by the news media, in order to protect consumers.
Here is a tool with 7,000+ customers and access to 1 million code repositories which was breached with an exploit a clever 11 year old could created. (edit: 1 million repos, not customers)
When the exploit is so simple, I find it likely that bots or Black Hats or APTs had already found a way in and established persistence before the White Hat researchers reported the issue. If this is the case, patching the issue might prevent NEW bad actors from penetrating CodeRabbit's environment, but it might not evict any bad actors which might now be lurking in their environment.
I know Security is hard, but come on guys
Lionga · 1h ago
Code Rabbit is a vibe coder company, what would you expect? Then they try to hide the breach and instead post marketing fluff on google cloud blog not even mentioning they got hacked and can not even give any proof there is no backdoor still running all the time.
What a piece of shit company.
moomoo11 · 32m ago
I got so much heat for calling out that Tea app for being imbeciles who couldn’t bother finishing reading the firebase docs.
People were quick to blame firebase instead of the devs.
Vibrators are so fucking annoying, mostly dumb, and super lame.
willejs · 15m ago
This is a great read, but unfortunately does not surprise me really, it was bound to happen given how people blindly add apps with wide permissions and githubs permissions model.
It amazes me how many people will install github apps that have wide scopes, primarily write permissions to their repositories. Even with branch protection, often people will allow privilaged access to their cloud in github actions from pull requests. To properly configure this, you need to change the github oidc audience and that is not well documented.
When you enquire with the company who makes an app and ask them to provide a different app with less scope to disable some features which require write, they often have no interest what so ever and don't understand the security concerns and potential implications.
I think github need to address this in part by allowing more granular app access defined by the installer, but also more granular permissions in general.
vadepaysa · 42m ago
I cancelled my coderabbit paid subscription, because it always worries me when a post has to go viral on HN for a company to even acknowledge an issue occurred. Their blogs are clean of any mention of this vulnerability and they don't have any new posts today either.
I understand mistakes happen, but lack of transparency when these happen makes them look bad.
codedokode · 26m ago
One of the problems is that code analyzers, bundlers, compilers (like Rust compiler) allow running arbitrary code without any warning.
Imagine following case: an attacker pretending to represent a company sends you a repository as a test task before the interview. You run something like "npm install" or run Rust compiler, and your computer is controlled by an attacker now.
Or imagine how one coworker's machine gets hacked, the malicious code is written into a repository and whole G, F or A is now owned by foreign hackers. All thanks to npm and Rust compiler.
Maybe those tools should explicitly confirm executing every external command (with caching allowed commands list in order to not ask again). And maybe Linux should provide an easy to use and safe sandbox for developers. Currently I have to make sandboxes from scratch myself.
Also in maybe cases you don't need the ability to run external code, for example, to install a JS package all you need to do is to download files.
Also this is an indication why it is a bad idea to use environment variables for secrets and configuration. Whoever wrote "12 points app" doesn't know that there are command-line switches and configuration files for this.
gpm · 55s ago
> compilers (like Rust compiler) allow running arbitrary code without any warning.
It's safe to assume that the Rust compiler (like any compiler built on top of LLVM) has arbitrary code execution vulnerabilities, but as an intended feature I think this only exists in cargo, the popular/official build system, not rustc, the compiler.
criemen · 15m ago
> Maybe those tools should explicitly confirm executing every external command
This wouldn't work - it's not external commands that's the problem, it's arbitrary code that's being executed. That code has access to all regular system APIs/syscalls, so there's no way of explicitly confirming external commands.
Python/pip suffers the same problem btw, so I think that ship has sailed.
codedokode · 14m ago
Then explicitly confirming running every hook with displaying module and function name.
> Python/pip suffers the same problem btw, so I think that ship has sailed.
If I ever find time to write a package manager for C, it won't support hooks.
ketzo · 3h ago
> While running the exploit, CodeRabbit would still review our pull request and post a comment on the GitHub PR saying that it detected a critical security risk, yet the application would happily execute our code because it wouldn’t understand that this was actually running on their production system.
What a bizarre world we're living in, where computers can talk about how they're being hacked while it's happening.
Also, this is pretty worrisome:
> Being quick to respond and remediate, as the CodeRabbit team was, is a critical part of addressing vulnerabilities in modern, fast-moving environments. Other vendors we contacted never responded at all, and their products are still vulnerable. [emphasis mine]
Props to the CodeRabbit team, and, uh, watch yourself out there otherwise!
htrp · 30m ago
You mean the anthropic model talked about an exploit... the coderabbit system just didn't listen
progforlyfe · 3h ago
Beautiful that CodeRabbit reviewed an exploit on its own system!
chanon · 2h ago
Oh my god. I haven't finished reading that yet, it became too much to comprehend. Too stressful to take in the scope. The part where he could have put malware into release files of 10s of thousands (or millions?) of open source tools/libraries/software. That could have been a worldwide catastrophe. And who knows what other similar vulnerabilities might still exist elsewhere.
chanon · 1h ago
I'm starting to think these 'Github Apps' are a bad idea. Even if CodeRabbit didn't have this vulnerability, what guarantee do we have that they will always be good actors? That their internal security measures will ensure that none of their employees may do any malicious things?
Taking care of private user data in a typical SaaS is one thing, but here you have the keys to make targetted supply chain attacks that could really wreak havoc.
gz09 · 1h ago
Correct me if I'm wrong, but the problem here is not with GitHub Apps, instead CodeRabbit violated the principle of least privilege: ideally the private key of their app should never end up in the environment of a job for a client but rather a short lived token should be minted from it (for just a single repo (for which the job is running)) so it never gets anywhere near those areas where one of their clients has any influence over what runs.
filleokus · 1h ago
I agree, this seems like straight up bad design from a security perspective.
But at the same time, me as a customer of Github, would prefer if Github made it harder for vendors like CodeRabbit to make misstakes like this.
If you have an app with access to more than 1M repos, it would make sense for Github to require a short lived token to access a given repository and only allow the "master" private key to update the app info or whatever.
And/or maybe design mechanisms that only allow minting of these tokens for the repo whenever a certain action is run (i.e not arbitrarily).
But at the end of the day, yes, it's impossible for Github to both allow users to grant full access to whatever app and at the same time ensure stuff like this doesn't happen.
lukevp · 1h ago
The private key isn’t a key in the “API KEY” sense, it’s a key in the “public/private key pair” sense. It’s not sent to github and there’s no way for them to know if the signing of the token used to make the call happened in a secure manner or not, because github doesn’t receive the key as part of the request at all.
0x457 · 51m ago
GH Apps already use short-lived tokens that can be scoped per repo. You mint a token using your private key and exchange it for a token via API. Then you use that token and dispose of it. That's the only way to use GH Apps (User Access Tokens which are the same thing, but require user interaction) Those tokens always expire.
I'd rather GitHub finally fix their registry to allow these GH Apps to push/pull with that instead of PAT.
mook · 48m ago
There's also no reason why they needed to have write access to post code review comments. But for some reason they ask for it and you can't deny that part when hooking up their thing.
codedokode · 17m ago
Just don't grant write access to random apps and you are safe.
risyachka · 1h ago
Software industry really needs at least some guardrails/regulations at this point.
It is absurd that anyone can mess up anything and have absolutely 0 consequences.
ianbutler · 9m ago
If you're a concerned user and you're looking for a solution founded by 2 people with a security background who have sandboxed execution (and network limited) so stuff like this can't happen you should check us out.
We even offer a self hosted deployment which sidesteps this entirely. (feel free to reach out).
www.bismuth.sh
thyrfa · 2h ago
It is incredibly bad practice that their "become the github app as you desire" keys to the kingdom private key was just sitting in the environment variables. Anybody can get hacked, but that's just basic secrets management, that doesn't have to be there. Github LITERALLY SAYS on their doc that storing it in an environment variable is a bad idea. Just day 1 stuff. https://docs.github.com/en/apps/creating-github-apps/authent...
doesnt_know · 1h ago
If it’s not a secret that is used to sign something, then the secret has to get from the vault to the application at some point.
What mechanism are you suggesting where access to the production system doesn’t let you also access that secret?
Like I get in this specific case where you are running some untrusted code, that environment should have been isolated and these keys not passed in, but running untrusted code isn’t usually a common feature of most applications.
Nextgrid · 1h ago
If you actually have a business case for defense in depth (hint: nobody does - data breaches aren't actually an issue besides temporarily pissing off some nerds, as Equifax' and various companies stock prices demonstrate), what you'd do is have a proxy service who is entrusted with those keys and can do the operations on behalf of downstream services. It can be as simple as an HTTP proxy that just slaps the "Authorization" header on the requests (and ideally whitelists the URL so someone can't point it to https://httpbin.org/get and get the secret token echoed back).
This would make it so that even a compromised downstream service wouldn't actually be able to exfiltrate the authentication token, and all its misdeeds would be logged by the proxy service, making post-incident remediation easier (and being able to definitely prove whether anything bad has actually happened).
roywiggins · 58m ago
In this specific case running linters doesn't even need that much I think, it's never going to need to reach out to GitHub on its own, let alone Anthropic etc. The linter process likely doesn't even need network access, just stdout so you can gather the result and fire that back to GitHub or whenever it needs to go. Just executing it with an empty environment would have helped things (though obviously an RCE would still be bad)
wahnfrieden · 1h ago
It is a national security concern more than a business ownership & market concern
Nextgrid · 58m ago
Unless "national security" is going to either pay people proactively to pass gov-mandated pentests, or enforce actual, business-threatening penalties for breaches, it doesn't really matter from a company owner perspective. They're not secure, but neither are their competitors, so it's all good.
curuinor · 2h ago
hey, this is Howon from CodeRabbit. We use a cloud-provider-provided key vault for application secrets, including GH private key.
ipython · 1h ago
This reply, while useful, only serves to obfuscate and doesn’t actually answer the question.
You can store the credentials in a key vault but then post them on pastebin. The issue is that the individual runner has the key in its environment variables. Both can be true- the key can be given to the runner in env and the key is stored in a key vault.
The important distinction here is - have you removed the master key and other sensitive credentials from the environment passed into scanners that come in contact with customer untrusted code??
thyrfa · 1h ago
Not at that time though, right, considering it was dumped? You have changed since, which is good, but under a year ago had it as just an env var
kmarc · 12m ago
My nightmare is that one of those auto updating vim/vscode/your-favorite-IDE plug-ins That many of us happily use on all the monorepos we work on, at one point invoke a "linter" (or as in this case, configure a linter maliciously) and we start leaking the precious IP to random attackers :-(
> No customer data was accessed and the vulnerability was quickly remediated within hours of disclosure
How do they know this -- Do they have any audit logs confirming this? A malicious actor could have been using this for months for all they know
cnst · 42m ago
Can someone explain how is this not GitHub's fault that they don't allow the end-user to modify the permissions that all these services require? E.g., fine-grained permission control?
For example, why a tool like this code analysis service would need git write permission access in the first place?
The only consolation here is that it'd be difficult to forge git repositories because of the SHA hash conflicts for any existing checkout, although presumably even there, the success rates would still be high enough, especially if they attack front-end repositories where the maintainers may not understand what has happened, and simply move on with the replaced repo without checking what went on.
brainless · 3h ago
I did not understand something: why did CodeRabbit run external tools on external code within its own set of environment variables? Why are these variables needed for this entire tooling?
tadfisher · 2h ago
> Why are these variables needed for this entire tooling?
They are not. The Github API secret key should never be exposed in the environment, period; you're supposed to keep the key in an HSM and only use it to sign the per-repo access token. Per the GH docs [0]:
> The private key is the single most valuable secret for a GitHub App. Consider storing the key in a key vault, such as Azure Key Vault, and making it sign-only. This helps ensure that you can't lose the private key. Once the private key is uploaded to the key vault, it can never be read from there. It can only be used to sign things, and access to the private key is determined by your infrastructure rules.
> Alternatively, you can store the key as an environment variable. This is not as strong as storing the key in a key vault. If an attacker gains access to the environment, they can read the private key and gain persistent authentication as the GitHub App.
Environment variables used to be standard practice for API keys. It seems like every time someone finds a way to get a key, standard practice gets more convoluted.
progbits · 12m ago
It's not convoluted. Env vars are fine for places where you need the value. If your application talks to service X with API key then sure, give it that via env var (mounted from some secret manager, so it's only mounted in production).
But there are two very wrong things here:
1. You don't send the private key to github like an API key, you use it to sign requests. So there is no reason for any application, even your trusted backend, to ever see that key. Just have it request signatures from a vault, and the vault can log each access for audit etc.
2. Even if you really trust your backend and give it the key, why the fuck does the sandboxed runner get it? And don't tell me it's easy to make a mistake and accidentally inherit it somehow. The runner should be on completely separate node, separate network, everything, it only gets the untrusted code to run as input and nothing more, and gives output back.
codedokode · 15m ago
A standard practice imho is configuration files. It is better almost in every aspect.
gdbsjjdn · 3h ago
It sounds like they were putting these processes in a chroot jail or something and not allowing them to access the parent process env vars. There's a continuum of ways to isolate child processes in Linux that don't necessarily involve containers or docker.
The_Fox · 3h ago
Their own tools would need the various API keys, of course, and they did build a method to filter out those variables and managed most user code through it, but it sounds like they forgot to put Rubocop through the special method.
So this researcher may have gotten lucky in choosing to dig into the tool that CodeRabbit got unlucky in forgetting.
chuckadams · 3h ago
It sounds like a pretty bad approach in general to have to "filter out the bad stuff" on a case-by-case basis. It should be as simple as launching everything from a sanitized parent environment, and making it impossible to launch any tool otherwise. Or better, make that sanitized environment the default and make privileged operations be the thing that jumps through hoops to talk to a bastion/enclave/whatever that holds the actual keys.
The_Fox · 2h ago
Yes although somewhere there will be an `if` statement to determine if the process being started should get the complete environment or a key to get the other keys or whatever. Best to make that `if` at the highest level of the architecture as possible and wrapped in something that makes it obvious, like a `DangerousUserCodeProcess` class.
The only other safety I can think of is a whitelist, perhaps of file pathnames. This helps to maintain a safe-by-default posture. Taking it further, the whitelist could be specified in config and require change approval from a second team.
elpakal · 3h ago
presuming they take the output of running these linters and pass it for interpretation to Claude or OpenAI
roywiggins · 46m ago
It's very silly that the linter process was handed those environment variables, since it wasn't going to do anything with them and didn't need them.
elpakal · 3h ago
> After responsibly disclosing this critical vulnerability to the CodeRabbit team, we learned from them that they had an isolation mechanism in place, but Rubocop somehow was not running inside it.
Curious what this (isolation mechanism) means if anyone knows.
diggan · 3h ago
> Curious what this (isolation mechanism) means if anyone knows.
If they're anything like the typical web-startup "developing fast but failing faster", they probably are using docker containers for "security isolation".
No comments yet
benmmurphy · 2h ago
What a lucky coincidence that the tool the researcher attacked because it allowed code execution was not sandboxed.
kachapopopow · 3h ago
you could say that they have vibe forgotten to sandbox it.
(likely asked AI to implement x and ai completely disregarded the need to sandbox).
nphardon · 2h ago
Oh, it really makes my day when we get hacker post here on the top. This is so well written too, no mystique, just a simple sequence of logical steps, with pictures.
curuinor · 2h ago
hey, this is Howon from CodeRabbit here. we wish to note that this RCE was reported and fixed in January. it was entirely prospective and no customer data was affected. we have extensive sandboxing for basically any execution of anything now, including any and every tool and all generated code of any kind under the CodeRabbit umbrella.
Where can we find the blog post you made back in January about the RCE fix explaining what measures you took to check if any customer data had been affected?
progbits · 8m ago
> no customer data was affected
Someone could have taken the private github key and cloned your customers' private repos.
You would need to audit every single access to github made via your app since the beginning and link it somehow to your side. Did you do this?
frankfrank13 · 2h ago
Reading this, its not clear how your blog posts relates:
1. You run git clone inside the GCR function, so, you have at the very least a user token for the git provider
2. RCE exploit basically used the external tools, like a static analysis checker, which again, is inside your GCR function
3. As a contrived example, if I could RCE `console.log(process.env)` then seemingly I could do `fetch(mywebsite....`
I get it, you can hand wave some amount of "VPC" and "sandbox" here. But, you're still executing code, explicitly labeling it "untrusted" and "sandboxed" doesn't excuse it.
cleverwebb · 2h ago
how do you know that no customer data was affected? did you work with github and scan all uses of your keys? how do you know if a use of your github key was authentic or not? did you check with anthroipic/openai/etc to scan logs usage?
It's really hard to trust a "hey we got this guys" statement after a fuckup this big
Xunjin · 1h ago
That's why countries should start to legislate on these matters, there are no incentives in focusing on security and properly report to the customers such vulnerability.
thyrfa · 2h ago
How can you guarantee that nobody ripped the private key before the researcher told you about the issue though?
KingOfCoders · 2h ago
Or has a backdoor installed somewhere?
KingOfCoders · 2h ago
The chuzpe to use this as PR.
Xunjin · 1h ago
Silicon Valley sitcom comedy moment right here.
elpakal · 2h ago
> Sandboxing: All Cloud Run instances are sandboxed with two layers of sandboxing and can be configured to have minimal IAM permissions via dedicated service identity. In addition, CodeRabbit is leveraging Cloud Run's second generation execution environment, a microVM providing full Linux cgroup functionality. Within each Cloud Run instance, CodeRabbit uses Jailkit to create isolated processes and cgroups to further restrict the privileges of the jailed process.
In case you don't want to read through the PR
0x457 · 23m ago
I don't get it, if you're running on linux then just use Landlock LSM inside a VM.
tadfisher · 2h ago
But do you still store your GH API private key in environment variables?
curuinor · 2h ago
hey, this is Howon from CodeRabbit. We use a cloud-provider-provided key vault for application secrets, including GH private key.
musicnarcoman · 1h ago
So the CodeRabbit application with access to application secrets still runs in the same virtual machine as untrusted code from the outside?
yunohn · 2h ago
While I fully understand that things sometimes get missed, it just seems really bizarre to me that somehow “sandboxing/isolation” was never considered prior to this incident. To me, it feels like the first thing to implement in a system that is explicitly built to run third party untrusted code?
wging · 2h ago
The article seems to imply that something of the sort had actually been attempted prior to the incident, but was either incomplete or buggy. I'm not sure the details would be entirely exculpatory, but unless you want to flatly disbelieve their statements, "not considered" isn't quite right.
> After responsibly disclosing this critical vulnerability to the CodeRabbit team, we learned from them that they had an isolation mechanism in place, but Rubocop somehow was not running inside it.
roywiggins · 51m ago
It seems to me that they thought the linter would be safe to run as it wasn't meant to actually run untrusted code, just statically analyze it.
jsbg · 44m ago
wild to comment this
dpacmittal · 1h ago
I hope the author received a nice well deserved bounty for this find. Could have been catastrophic in the wrong hands.
cube00 · 1h ago
When they're spinning it [1] as a PR opportunity with no mention of the breach there won't be a bounty.
Why does CodeRabbit need write access to the git repo? Why doesn't Github let me limit it's access?
dpcx · 3h ago
Because it has the ability to write tests for the PR in question.
tadfisher · 2h ago
Then it should open a PR for those tests so it can go through the normal CI and review process.
tedivm · 2h ago
Doing that requires write access if you're a Github Application. You can't just fork repositories back into another org, since Github Applications only have the permissions of the single organization that they work with. Rulesets that prevent direct pushes to specific branches can help here, but have to be configured for each organization.
dpcx · 2h ago
It updates the existing PR with the tests, I believe. They'd still get reviewed and go through CI.
tadfisher · 2h ago
Right, the downside being that the app needs write access to your repository.
rahkiin · 2h ago
Writing to PR branches should really be some new kind of permission.
flippyhead · 1h ago
It's more than that. If can suggest fixes which you can directly commit.
iTokio · 1h ago
That’s why I’m worried about the growing centralization of things such as Chrome, Gmail, AWS, Cloudflare…
It’s very efficient to delegate something to one major actor but we are introducing single points of failure and are less resilient to vulnerabilities.
Critical systems should have defenses in depth, decentralized architectures and avoid trusting new providers with too many moving parts.
eranation · 1h ago
This is very similar to a CVE I discovered in cdxgen (CVE-2024-50611), which is similar to another CVE in Snyk's plugin (CVE-2022-24441). tl;dr if you run a scanner on untrusted code, ensure it doesn't have a way of executing that code.
Some ways to prevent this from happening:
1. Don't let spawned processes have access to your env, there are ways to allowlist a set of env vars that are needed for a sub process in all major languages
2. Don't store secrets in env vars, use a good secrets vault (with a cache)
3. Tenant isolation as much as you can
4. And most obviously - don't run processes that can execute the code they are scanning, especially if that code is not your code (harder to tell, but always be paranoid)
edm0nd · 2h ago
No bounty was paid for this?
cube00 · 2h ago
I can't say I'm surprised they didn't pay a bounty when they couldn't even own up to this on their own blog [1].
Instead they took it as an opportunity to market their new sandboxing on Google's blog [2] again with no mention of why their hand was forced into building the sandboxing they should have had before they rushed to onboard thousands of customers.
I have no idea what their plan was. They had to have known the researchers would eventually publish this. Perhaps they were hoping it wouldn't get the same amount of attention it would if they posted it on their own blog.
First thing I looked for... this is an absolutely critical vulnerability that if exploited would have completely ruined their business. No bounty!?
vntok · 2h ago
Why would they pay anything? The researchers offered them the vuln analysis for free, unprompted.
If anything, they got paid in exposure.
cube00 · 1h ago
Let's hope the grants keep coming in because those researchers will start getting offers from the darker corners of the web if bounties aren't paid.
cleverwebb · 2h ago
I had a visceral and (quite audible) reaction when I got to the environment variable listing.
curuinor · 2h ago
hey, this is Howon from CodeRabbit. We use a cloud-provider-provided key vault for application secrets, including GH private key.
tnolet · 1h ago
what does that mean? Were the leaked keys irrelevant?
hotsauceror · 41m ago
This is the third or fourth time you’ve spammed this exact comment in response to people’s perfectly legitimate questions. What is this clown-show bullshit?
kachapopopow · 3h ago
Unrelated to the article, but the first time I saw them was in a twitter ad with a completely comically bull** suggestion. I cannot take a company seriously that had something like that inside an ad that is supposed to show the best they're capable of.
sitzkrieg · 2h ago
comically bad. get used to more of this
thewisenerd · 2h ago
global scoped installations or keys always scare me for this reason
i believe the answer here was to exchange the token for something scoped to the specific repo coderabbit is running in, but alas, that doesn't remove the "RCE" _on_ the repo
tadfisher · 2h ago
They do that, this is how GH apps work. There is no reason to expose the app's private key in the environment for the code that actually runs on the PR.
thewisenerd · 1h ago
even if they did not have the PEM file left in the environment, the token is still widely scoped and has the same scope as the PEM
what i'm clearly mis-remembering is being able to exchange the token for a smaller scope e.g., hey~ sign this jwt, with scopes=[org/repo1, org/repo2, permissions=write]
pengaru · 1h ago
This third party app gets write access to your repository, so it can do automated reviews of PRs?
Why would you even grant it such permissions? this is ridiculous.
kmarc · 7m ago
Besides that this was clearly a security f*ckup, in my mind it's almost equivalent to running those third party liters in our Internet-connection-enabled editors and IDEs. Other than one banking project, I don't think I ever had to sandbox my editor in any way.
Scary.
binarydreams · 2h ago
I've noticed CodeRabbit at times does reviews that are super. It is able to catch bugs that even claude code misses on our Github PRs. Blows my mind at times tbh.
Based on the env vars seems like they're using anthropic, openai, etc. only?
jacobsenscott · 1h ago
> catch bugs that even claude code misses on our Github PRs
Is that good? I assume it just catches a different 10% of the bugs.
tnolet · 2h ago
Interesting. We removed it as it was mostly too verbose, catching too many false positives and never really added anything useful.
flippyhead · 1h ago
You can comment on this in the PR and supposedly it'll remember and get better.
Here is a tool with 7,000+ customers and access to 1 million code repositories which was breached with an exploit a clever 11 year old could created. (edit: 1 million repos, not customers)
When the exploit is so simple, I find it likely that bots or Black Hats or APTs had already found a way in and established persistence before the White Hat researchers reported the issue. If this is the case, patching the issue might prevent NEW bad actors from penetrating CodeRabbit's environment, but it might not evict any bad actors which might now be lurking in their environment.
I know Security is hard, but come on guys
What a piece of shit company.
People were quick to blame firebase instead of the devs.
Vibrators are so fucking annoying, mostly dumb, and super lame.
It amazes me how many people will install github apps that have wide scopes, primarily write permissions to their repositories. Even with branch protection, often people will allow privilaged access to their cloud in github actions from pull requests. To properly configure this, you need to change the github oidc audience and that is not well documented.
When you enquire with the company who makes an app and ask them to provide a different app with less scope to disable some features which require write, they often have no interest what so ever and don't understand the security concerns and potential implications.
I think github need to address this in part by allowing more granular app access defined by the installer, but also more granular permissions in general.
I understand mistakes happen, but lack of transparency when these happen makes them look bad.
Imagine following case: an attacker pretending to represent a company sends you a repository as a test task before the interview. You run something like "npm install" or run Rust compiler, and your computer is controlled by an attacker now.
Or imagine how one coworker's machine gets hacked, the malicious code is written into a repository and whole G, F or A is now owned by foreign hackers. All thanks to npm and Rust compiler.
Maybe those tools should explicitly confirm executing every external command (with caching allowed commands list in order to not ask again). And maybe Linux should provide an easy to use and safe sandbox for developers. Currently I have to make sandboxes from scratch myself.
Also in maybe cases you don't need the ability to run external code, for example, to install a JS package all you need to do is to download files.
Also this is an indication why it is a bad idea to use environment variables for secrets and configuration. Whoever wrote "12 points app" doesn't know that there are command-line switches and configuration files for this.
It's safe to assume that the Rust compiler (like any compiler built on top of LLVM) has arbitrary code execution vulnerabilities, but as an intended feature I think this only exists in cargo, the popular/official build system, not rustc, the compiler.
This wouldn't work - it's not external commands that's the problem, it's arbitrary code that's being executed. That code has access to all regular system APIs/syscalls, so there's no way of explicitly confirming external commands.
Python/pip suffers the same problem btw, so I think that ship has sailed.
> Python/pip suffers the same problem btw, so I think that ship has sailed.
If I ever find time to write a package manager for C, it won't support hooks.
What a bizarre world we're living in, where computers can talk about how they're being hacked while it's happening.
Also, this is pretty worrisome:
> Being quick to respond and remediate, as the CodeRabbit team was, is a critical part of addressing vulnerabilities in modern, fast-moving environments. Other vendors we contacted never responded at all, and their products are still vulnerable. [emphasis mine]
Props to the CodeRabbit team, and, uh, watch yourself out there otherwise!
Taking care of private user data in a typical SaaS is one thing, but here you have the keys to make targetted supply chain attacks that could really wreak havoc.
But at the same time, me as a customer of Github, would prefer if Github made it harder for vendors like CodeRabbit to make misstakes like this.
If you have an app with access to more than 1M repos, it would make sense for Github to require a short lived token to access a given repository and only allow the "master" private key to update the app info or whatever.
And/or maybe design mechanisms that only allow minting of these tokens for the repo whenever a certain action is run (i.e not arbitrarily).
But at the end of the day, yes, it's impossible for Github to both allow users to grant full access to whatever app and at the same time ensure stuff like this doesn't happen.
I'd rather GitHub finally fix their registry to allow these GH Apps to push/pull with that instead of PAT.
It is absurd that anyone can mess up anything and have absolutely 0 consequences.
We even offer a self hosted deployment which sidesteps this entirely. (feel free to reach out).
www.bismuth.sh
What mechanism are you suggesting where access to the production system doesn’t let you also access that secret?
Like I get in this specific case where you are running some untrusted code, that environment should have been isolated and these keys not passed in, but running untrusted code isn’t usually a common feature of most applications.
This would make it so that even a compromised downstream service wouldn't actually be able to exfiltrate the authentication token, and all its misdeeds would be logged by the proxy service, making post-incident remediation easier (and being able to definitely prove whether anything bad has actually happened).
You can store the credentials in a key vault but then post them on pastebin. The issue is that the individual runner has the key in its environment variables. Both can be true- the key can be given to the runner in env and the key is stored in a key vault.
The important distinction here is - have you removed the master key and other sensitive credentials from the environment passed into scanners that come in contact with customer untrusted code??
In fact, I use rubocop every day lately LOL
https://www.coderabbit.ai/blog/our-response-to-the-january-2...
How do they know this -- Do they have any audit logs confirming this? A malicious actor could have been using this for months for all they know
For example, why a tool like this code analysis service would need git write permission access in the first place?
The only consolation here is that it'd be difficult to forge git repositories because of the SHA hash conflicts for any existing checkout, although presumably even there, the success rates would still be high enough, especially if they attack front-end repositories where the maintainers may not understand what has happened, and simply move on with the replaced repo without checking what went on.
They are not. The Github API secret key should never be exposed in the environment, period; you're supposed to keep the key in an HSM and only use it to sign the per-repo access token. Per the GH docs [0]:
> The private key is the single most valuable secret for a GitHub App. Consider storing the key in a key vault, such as Azure Key Vault, and making it sign-only. This helps ensure that you can't lose the private key. Once the private key is uploaded to the key vault, it can never be read from there. It can only be used to sign things, and access to the private key is determined by your infrastructure rules.
> Alternatively, you can store the key as an environment variable. This is not as strong as storing the key in a key vault. If an attacker gains access to the environment, they can read the private key and gain persistent authentication as the GitHub App.
[0]: https://docs.github.com/en/apps/creating-github-apps/authent...
But there are two very wrong things here:
1. You don't send the private key to github like an API key, you use it to sign requests. So there is no reason for any application, even your trusted backend, to ever see that key. Just have it request signatures from a vault, and the vault can log each access for audit etc.
2. Even if you really trust your backend and give it the key, why the fuck does the sandboxed runner get it? And don't tell me it's easy to make a mistake and accidentally inherit it somehow. The runner should be on completely separate node, separate network, everything, it only gets the untrusted code to run as input and nothing more, and gives output back.
So this researcher may have gotten lucky in choosing to dig into the tool that CodeRabbit got unlucky in forgetting.
The only other safety I can think of is a whitelist, perhaps of file pathnames. This helps to maintain a safe-by-default posture. Taking it further, the whitelist could be specified in config and require change approval from a second team.
Curious what this (isolation mechanism) means if anyone knows.
If they're anything like the typical web-startup "developing fast but failing faster", they probably are using docker containers for "security isolation".
No comments yet
(likely asked AI to implement x and ai completely disregarded the need to sandbox).
if you want to learn how CodeRabbit does the isolation, here's a blog post about how: https://cloud.google.com/blog/products/ai-machine-learning/h...
Someone could have taken the private github key and cloned your customers' private repos.
You would need to audit every single access to github made via your app since the beginning and link it somehow to your side. Did you do this?
1. You run git clone inside the GCR function, so, you have at the very least a user token for the git provider
2. RCE exploit basically used the external tools, like a static analysis checker, which again, is inside your GCR function
3. As a contrived example, if I could RCE `console.log(process.env)` then seemingly I could do `fetch(mywebsite....`
I get it, you can hand wave some amount of "VPC" and "sandbox" here. But, you're still executing code, explicitly labeling it "untrusted" and "sandboxed" doesn't excuse it.
It's really hard to trust a "hey we got this guys" statement after a fuckup this big
In case you don't want to read through the PR
> After responsibly disclosing this critical vulnerability to the CodeRabbit team, we learned from them that they had an isolation mechanism in place, but Rubocop somehow was not running inside it.
[1]: https://news.ycombinator.com/item?id=44954242
It’s very efficient to delegate something to one major actor but we are introducing single points of failure and are less resilient to vulnerabilities.
Critical systems should have defenses in depth, decentralized architectures and avoid trusting new providers with too many moving parts.
Some ways to prevent this from happening:
1. Don't let spawned processes have access to your env, there are ways to allowlist a set of env vars that are needed for a sub process in all major languages
2. Don't store secrets in env vars, use a good secrets vault (with a cache)
3. Tenant isolation as much as you can
4. And most obviously - don't run processes that can execute the code they are scanning, especially if that code is not your code (harder to tell, but always be paranoid)
Instead they took it as an opportunity to market their new sandboxing on Google's blog [2] again with no mention of why their hand was forced into building the sandboxing they should have had before they rushed to onboard thousands of customers.
I have no idea what their plan was. They had to have known the researchers would eventually publish this. Perhaps they were hoping it wouldn't get the same amount of attention it would if they posted it on their own blog.
[1]: https://news.ycombinator.com/item?id=44954560
[2]: https://news.ycombinator.com/item?id=44954242
If anything, they got paid in exposure.
i believe the answer here was to exchange the token for something scoped to the specific repo coderabbit is running in, but alas, that doesn't remove the "RCE" _on_ the repo
what i'm clearly mis-remembering is being able to exchange the token for a smaller scope e.g., hey~ sign this jwt, with scopes=[org/repo1, org/repo2, permissions=write]
Why would you even grant it such permissions? this is ridiculous.
Scary.
Based on the env vars seems like they're using anthropic, openai, etc. only?
Is that good? I assume it just catches a different 10% of the bugs.