I think the point around incorporating MFA into the automated publishing flow isn't getting enough attention.
I've got no problem with doing an MFA prompt to confirm publish by a CI workflow - but last I looked this was a convoluted process of opening a https tunnel out (using a third party solution) such that you could provide the code.
I'd love to see either npm or GitHub provide an easy, out the box way, for me to provide/confirm a code during CI.
tcoff91 · 3m ago
Publishing a package involves 2 phases: uploading the package to npmjs, and making it availble to users. Right now these 2 phases are bundled together into 1 operation.
I think the right way to approach this is to unbundle uploading the packages & publishing packages so that they're available to end-users.
CI systems should be able to build & upload packages in a fully automated manner.
Publishing the uploaded packages should require a human to log into npmjs's website & manually publish the package and go through MFA.
darkamaul · 19m ago
> That repo still contained a GitHub Actions secret — a npm token with broad publish rights.
One of the advantages of Trusted Publishing [0] is that we no longer need long-lived tokens with publish rights. Instead, tokens are generated on the CI VM and are valid for only 15 minutes.
This has already been implemented in several ecosystems (PyPI, npm, Cargo, Homebrew), and I encourage everyone to use it, it actually makes publishing a bit _easier_.
More importantly, if the documentation around this still feels unclear, don’t hesitate to ask for help. Ecosystem maintainers are usually eager to see wider adoption of this feature.
> A while ago, I collaborated on angulartics2, a shared repository where multiple people still had admin rights. That repo still contained a GitHub Actions secret — a npm token with broad publish rights. This collaborator had access to projects with other people which I believe explains some of the other 40 initial packages that were affected.
> A new Shai-Hulud branch was force pushed to angulartics2 with a malicious github action workflow by a collaborator. The workflow ran immediately on push (did not need review since the collaborator is an admin) and stole the npm token. With the stolen token, the attacker published malicious versions of 20 packages. Many of which are not widely used, however the @ctrl/tinycolor package is downloaded about 2 million times a week.
I still don't get it. An admin on angulartics2 gets hacked, his Github access is used to push a malicious workflow that extracts an npm token. But why would an npm token in angulartics2 have publication rights to tinycolor?
hinkley · 16m ago
I have admin rights on someone else’s npm repo and I’ve done most of the recent releases. Becoming admin lit a fire under me to fix all of the annoying things and shitty design decisions that have been stuck in the backlog for years so most of the commits are also mine. I don’t want my name on broken code that “works”.
I had just about convinced myself that we should be using a GitHub action to publish packages because there was always the possibility that publishing directly via 2FA, that one (or specifically I) could fuck up and publish something that wasn’t a snapshot of trunk.
But I worried about stuff like this and procrastinated on forcing the issue with the other admins. And it looks like the universe has again rewarded my procrastination. I don’t know what the answer is but giving your credentials to a third party clearly isn’t it.
tetha · 1h ago
> But why would an npm token in angulartics2 have publication rights to tinycolor?
Imo, this is one of the most classical ways organizations get pwned: That one sin from your youth years ago comes to bite you in the butt.
We also had one of these years ago. It wasn't the modern stack everyone was working to scan and optimize and keep us secure that allowed someone to upload stuff to our servers. It was the editor that had been replaced years and years ago, and it's replacement had also been replaced, the way it was packaged wasn't seen by the build-time security scans, but eventually someone found it with a URL scan. Whoopsie.
Terr_ · 48m ago
Thinking of biology, the reason often given for the disappearance of "unused" genes is that there's a metabolic cost to keeping them around and copying them on every cell division.
I wonder if someday we'll find there's also an active process that resembles "remove old shit because it may contain security vulnerabilities."
STRiDEX · 1h ago
Sorry if that wasn't clear. This was a token with global publish rights to my npm packages.
Scaevolus · 1h ago
I was confused too. Was it your npm token stored in angulartics2 as a Github Actions secret, so it could publish new angulartics2 versions?
STRiDEX · 1h ago
Yes, exactly.
indigodaddy · 1h ago
Anyone know of a published tool/script to check for the existence of any of the vulnerable npm packages? I don't see anything like that in the stepsecurity page.
retlehs · 21m ago
This won’t protect against everything, but it still seems like a good idea to implement:
Yep I did see that, but I'm not planning on pushing anything, just want a tool to scan for any of the offending packages. Could make my own but feel like somebody must have already made something (and probably better than I can)
rectang · 1h ago
Two-factor auth for publishing is helpful, but requiring cryptographically signed approval by multiple authors would be more helpful. Then compromising a single author wouldn't be enough.
tcoff91 · 1h ago
Many packages have only 1 author.
KronisLV · 11m ago
I'm not sure why we never got around to more human in the loop with 2FA when it comes to this sort of stuff: "Oh, you want to publish a new package? Okay, confirm it on this app on your device/phone to make sure." Surely a button press on a pre-approved device wouldn't be too hard, pretty much how every user initiated online banking payment over here goes like.
I once heard from a sysadmin that didn't want to automate certificate renewal and other things, because he believed that doing so would take away useful skills or some inner knowledge of how the system works. Because of the human error risk, I thought that was stupid, but when it comes to approval processes, I think it makes sense. Especially because pushing code doesn't necessarily mean the same thing as such an approval, or the main device that you push code from could also get compromised, using your phone as 2FA could save you.
Then again, maybe I'm also stupid and the way we build our software is messed up on a fundamental level with all of the dependencies and nobody being able to practically audit all of the code they import, given deadlines, limited skills and resources and so on. Maybe it's all just fighting against a windmill.
rectang · 1h ago
The conclusion I'm coming to is that depending on packages which only have a single author is problematic. There are too many ways that packages published by one person can be compromised.
Packages which don't have approval and review by a reliable third party shouldn't be visible by default in a package manager.
Hackbraten · 55m ago
How are you supposed to gain collaborators for a project that no one can possibly find?
rectang · 51m ago
There are ways, but at a high level, I don't care. I hate how modern package managers have come to value author convenience over downstream user security.
Hackbraten · 7m ago
Fair enough.
In the meantime, I'm trying to do my part through occasional random spot inspections when there's an update to a package, and encourage others to do the same for swarm coverage.
chrisweekly · 1h ago
and (as in this case), that 1 author may use a single token to authz publishing many packages
bikeshaving · 1h ago
> Local 2FA based publishing isn’t sustainable...
Why is local 2FA unsustainable?! The real problem here is automated publishing workflows. The overwhelming majority of NPM packages do not publish often enough or have complicated enough release steps to justify tokens with the power to publish without human intervention.
What is so fucking difficult about running `npm publish` manually with 2FA? If maintainers are unwilling to do this for their packages, they should reconsider the number of packages they maintain.
STRiDEX · 1h ago
That's fair, I'm referring to the number of mistakes that happen with local publishing. Publishing the wrong branch, not building from latest etc
skydhash · 51m ago
So add a wrapper for that, a quick script that checks which branch and revision you are publishing from. The issue here is publishing from a CI you do not control that well and with automated events.
paxys · 42m ago
You can run the exact same script locally as you do in CI, with the only difference being the addition of a 2FA prompt.
STRiDEX · 10m ago
That's a good point, I would lose package provenance that way. I guess that is fine since it didn't prevent anything here.
I can look into that.
cyberax · 1h ago
> exfiltrated a npm token with broad publish rights
I freaking HATE tokens. I hate them.
There should be a better way to do authentication than a glorified static password.
These machine-to-machine OIDC flows seem secure, and maybe they are when they’re implemented properly, but they’re really difficult to configure. And I can’t shake the feeling that they’re basically just “tokens with more moving parts,” at least for a big chunk of exploitation paths. Without a human in the loop, there’s still some “thing” that gets compromised, whether it’s a token or something that generates time-limited tokens.
In the case of this worm, the OIDC flow wouldn’t even help. The GitHub workflow was compromised. If the workflow was using an OIDC credential like this to publish to npm, the only difference would be the npm publish command wouldn’t use any credential because the GitHub workflow would inject some temporary identity into the environment. But the root problem would remain: an untrusted user shouldn’t be able to execute a workflow with secret parameters. Maybe OIDC would limit the impact to be more fine-grained, but so would changing the token permissions.
tetha · 21m ago
Hence you need to start thinking about threat models and levels of compromise, even in your build system.
If I control the issuing and governance of these short-lived secrets, they very much help against many attacks. Go ahead and extract an upload token for one project which lives for 60 seconds, be my guest. Once I lose control how these tokens are created, most of these advantages go away - you can just create a token every minute, for any project this infrastructure might be responsible for.
If I maintain control about my pipeline definition, I can again do a lot of work to limit damage. For example, if I am in control, I can make sure the stages running untrusted codes have as little access to secrets as possible, and possibly isolate them in bubblewrap, VMs, ..., minimize the code with access to publishing rights. Once I lose control about the pipeline structure, all that goes away. Just add a build step to push all information and secrets to mastodon in individual toots, yey.
To me, this has very much raised questions about keeping pipeline definitions and code in one repository. Or at least, to keep a publishing/release process in there. I don't have a simple solution there, especially for OSS software with little infrastructure - it's not an easy topic. But with these supply chain attacks coming hot and fast every 2 weeks, it's something to think about.
chatmasta · 18m ago
But the token wasn’t the primary source of compromise here. It was the GitHub workflow which had the token embedded into it. There was no need for the actor to exfiltrate the token from the workflow to somewhere else, because they could simply run arbitrary code within the workflow.
It would have made little difference if the environment variable was NPM_WEBIDENTITY instead of NPM_TOKEN. The workflow was still compromised.
cyberax · 15m ago
Universal OIDC tokens would slow down the lateral expansion and make it more difficult.
You won't be able to exfiltrate a token that allows you to publish an NPM package outside of a workflow, the infection has to happen during a build on GH.
er4hn · 1h ago
Well the idea behind tokens is that they should be time and authZ limited. In most cases they are not so they degrade to a glorified static password.
Solutions like generating them live with a short lifetime, using solutions like oauth w/ proper scopes, biscuits that limit what they can do in detail, etc, all exist and are rarely used.
I've got no problem with doing an MFA prompt to confirm publish by a CI workflow - but last I looked this was a convoluted process of opening a https tunnel out (using a third party solution) such that you could provide the code.
I'd love to see either npm or GitHub provide an easy, out the box way, for me to provide/confirm a code during CI.
I think the right way to approach this is to unbundle uploading the packages & publishing packages so that they're available to end-users.
CI systems should be able to build & upload packages in a fully automated manner.
Publishing the uploaded packages should require a human to log into npmjs's website & manually publish the package and go through MFA.
One of the advantages of Trusted Publishing [0] is that we no longer need long-lived tokens with publish rights. Instead, tokens are generated on the CI VM and are valid for only 15 minutes.
This has already been implemented in several ecosystems (PyPI, npm, Cargo, Homebrew), and I encourage everyone to use it, it actually makes publishing a bit _easier_.
More importantly, if the documentation around this still feels unclear, don’t hesitate to ask for help. Ecosystem maintainers are usually eager to see wider adoption of this feature.
[0] https://docs.pypi.org/trusted-publishers/
> A new Shai-Hulud branch was force pushed to angulartics2 with a malicious github action workflow by a collaborator. The workflow ran immediately on push (did not need review since the collaborator is an admin) and stole the npm token. With the stolen token, the attacker published malicious versions of 20 packages. Many of which are not widely used, however the @ctrl/tinycolor package is downloaded about 2 million times a week.
I still don't get it. An admin on angulartics2 gets hacked, his Github access is used to push a malicious workflow that extracts an npm token. But why would an npm token in angulartics2 have publication rights to tinycolor?
I had just about convinced myself that we should be using a GitHub action to publish packages because there was always the possibility that publishing directly via 2FA, that one (or specifically I) could fuck up and publish something that wasn’t a snapshot of trunk.
But I worried about stuff like this and procrastinated on forcing the issue with the other admins. And it looks like the universe has again rewarded my procrastination. I don’t know what the answer is but giving your credentials to a third party clearly isn’t it.
Imo, this is one of the most classical ways organizations get pwned: That one sin from your youth years ago comes to bite you in the butt.
We also had one of these years ago. It wasn't the modern stack everyone was working to scan and optimize and keep us secure that allowed someone to upload stuff to our servers. It was the editor that had been replaced years and years ago, and it's replacement had also been replaced, the way it was packaged wasn't seen by the build-time security scans, but eventually someone found it with a URL scan. Whoopsie.
I wonder if someday we'll find there's also an active process that resembles "remove old shit because it may contain security vulnerabilities."
https://github.com/danielroe/provenance-action
I once heard from a sysadmin that didn't want to automate certificate renewal and other things, because he believed that doing so would take away useful skills or some inner knowledge of how the system works. Because of the human error risk, I thought that was stupid, but when it comes to approval processes, I think it makes sense. Especially because pushing code doesn't necessarily mean the same thing as such an approval, or the main device that you push code from could also get compromised, using your phone as 2FA could save you.
Then again, maybe I'm also stupid and the way we build our software is messed up on a fundamental level with all of the dependencies and nobody being able to practically audit all of the code they import, given deadlines, limited skills and resources and so on. Maybe it's all just fighting against a windmill.
Packages which don't have approval and review by a reliable third party shouldn't be visible by default in a package manager.
In the meantime, I'm trying to do my part through occasional random spot inspections when there's an update to a package, and encourage others to do the same for swarm coverage.
Why is local 2FA unsustainable?! The real problem here is automated publishing workflows. The overwhelming majority of NPM packages do not publish often enough or have complicated enough release steps to justify tokens with the power to publish without human intervention.
What is so fucking difficult about running `npm publish` manually with 2FA? If maintainers are unwilling to do this for their packages, they should reconsider the number of packages they maintain.
I can look into that.
I freaking HATE tokens. I hate them.
There should be a better way to do authentication than a glorified static password.
An example of how to do it correctly: Github as a token provider for AWS: https://aws.amazon.com/blogs/security/use-iam-roles-to-conne... But this is an exception, rather than a rule.
In the case of this worm, the OIDC flow wouldn’t even help. The GitHub workflow was compromised. If the workflow was using an OIDC credential like this to publish to npm, the only difference would be the npm publish command wouldn’t use any credential because the GitHub workflow would inject some temporary identity into the environment. But the root problem would remain: an untrusted user shouldn’t be able to execute a workflow with secret parameters. Maybe OIDC would limit the impact to be more fine-grained, but so would changing the token permissions.
If I control the issuing and governance of these short-lived secrets, they very much help against many attacks. Go ahead and extract an upload token for one project which lives for 60 seconds, be my guest. Once I lose control how these tokens are created, most of these advantages go away - you can just create a token every minute, for any project this infrastructure might be responsible for.
If I maintain control about my pipeline definition, I can again do a lot of work to limit damage. For example, if I am in control, I can make sure the stages running untrusted codes have as little access to secrets as possible, and possibly isolate them in bubblewrap, VMs, ..., minimize the code with access to publishing rights. Once I lose control about the pipeline structure, all that goes away. Just add a build step to push all information and secrets to mastodon in individual toots, yey.
To me, this has very much raised questions about keeping pipeline definitions and code in one repository. Or at least, to keep a publishing/release process in there. I don't have a simple solution there, especially for OSS software with little infrastructure - it's not an easy topic. But with these supply chain attacks coming hot and fast every 2 weeks, it's something to think about.
It would have made little difference if the environment variable was NPM_WEBIDENTITY instead of NPM_TOKEN. The workflow was still compromised.
You won't be able to exfiltrate a token that allows you to publish an NPM package outside of a workflow, the infection has to happen during a build on GH.
Solutions like generating them live with a short lifetime, using solutions like oauth w/ proper scopes, biscuits that limit what they can do in detail, etc, all exist and are rarely used.