> But if you want an existence proof: Maven. The Java library ecosystem has been going strong for 20 years, and during that time not once have we needed a lockfile. And we are pulling hundreds of libraries just to log two lines of text, so it is actively used at scale.
Do not pretend, for even half a second, that dependency resolution is not hell in maven (though I do like that packages are namespaced by creators, npm shoulda stolen that).
Karrot_Kream · 6h ago
When I used to lead a Maven project I'd take dependency-upgrade tickets that would just be me bumping up a package version then whack-a-moling overrides and editing callsites to make dependency resolution not pull up conflicting packages until it worked. Probably lost a few days a quarter that way. I even remember the playlists I used to listen to when I was doing that work (:
Lockfiles are great.
RobRivera · 5h ago
> I even remember the playlists I used to listen to when I was doing that work (:
Im a big fan of anything Aphex Twin for these type of sessions.
yawaramin · 4h ago
How do lockfiles solve this problem? You would still have dependency-upgrade tickets and whack-a-mole, no? Or do you just never upgrade anything?
hyperpape · 2h ago
I think the difference is that since libraries do not specify version ranges, you must manually override their choices to find a compatible set of dependencies.
The solution is version ranges, but this then necessitates lockfiles, to avoid the problem of uncontrolled upgrades.
Note: maven technically allows version ranges, but they're rarely used.
chowells · 3h ago
The difference is that the data is centralized with a single source of truth, and you have tools for working with it automatically. It doesn't mean lockfiles are cheap to update, but it does mean it's a much more streamlined process when it's time.
yawaramin · 3h ago
The data is also centralized without lockfiles though...it's in the package spec file itself, where all version can be upgraded together. If you are saying the only difference is some tooling for automation, that's a temporary problem, not a fundamental one.
chowells · 1h ago
By "centralized" I mean a single file that contains all transitive dependency version information and nothing else. This is very different than having to recursively examine tiny portions of your dependencies' package specs recursively to find all transitive dependencies.
growse · 2h ago
Without a lock file your transitive dependencies are.... by definition not centralised?
Or have I misunderstood?
Muromec · 3h ago
you press a button which triggers a pipeline which does npm update on all root dependencies and produces a new lockfile, then creates a PR you need to approve. creating a PR triggers running all the tests you bothered to write to also flag things that didn't go well automagically.
yawaramin · 3h ago
Is this a joke? What's the difference between this and pressing a button which triggers a pipeline which does npm update on all root dependencies and produces a new package.json file which you approve in a PR? Is the only difference that some convenient tooling already exists so you don't have to update each root dependency by hand?
crote · 2h ago
A package spec defines your desires for your dependencies, a lockfile defines a resolution of those desires.
For example, as a developer I want Spring to stay on 6.3.x and not suddenly jump to 6.4 - as that is likely to break stuff. I do not care whether I get 6.3.1 or 6.3.6, as they are quite unlikely to cause issues. I do not care the slightest what version of libfoobar I get 5 dependencies down the line.
However, I do not want packages to suddenly change versions between different CI runs. Breakage due to minor version bumps are unlikely, but they will happen. That kind of stuff is only going to cause noise when a rebuild of a PR causes it to break with zero lines changed, so you only want version upgrades to happen specifically when you ask for them. On top of that there's the risk of supply chain attacks, where pulling in the absolute latest version of every single package isn't exactly a great idea.
The package spec allows me to define "spring at 6.3.x", the lockfile stores that we are currently using "spring 6.3.5, libfoobar 1.2.3". I ask it to look for upgrades, it resolves to "spring 6.3.6, libfoobar 1.4.0". There's also a "spring 6.4.0" available, but the spec says that we aren't interested so it gets ignored. All tests pass, it gets merged, and we'll stay at those versions until we explicitly ask for another upgrade.
The whole flow exists for things which aren't root dependencies. The major versions of those are trivial to keep track of manually, and you'll only have a handful of them. It's all the minor versions and downstream dependencies which are the issue: tracking them all manually is a nightmare, and picking whatever happens to be the latest version at time-of-build is a massive no-no.
Muromec · 2h ago
The difference is transitive dependences with ranges, which results in less frequent updates of direct dependences. And tooling of course too
adrianmsmith · 5h ago
If you use two dependencies, and one requires Foo 1.2.3 and the other Foo 1.2.4 then 99% of the time including either version of Foo will work fine. (I was a Java developer and used Maven for about 10 years.)
For those times where that's not the case, you can look at the dependency tree to see which is included and why. You can then add a <dependency> override in your pom.xml file specifying the one you want.
It's not an "insane" algorithm. It gives you predictability. If you write something in your pom.xml that overrides whatever dependency your dependency requires, because you can update your pom.xml if you need to.
And because pom.xml is hand-written there are very few merge conflicts (as much as you'd normally find in source code), vs. a lock file where huge chunks change each time you change a dependency, and when it comes to a merge conflict you just have to delete the lot and redo it and hope nothing important has been changed.
zdragnar · 4h ago
> You can then add a <dependency> override in your pom.xml file specifying the one you want.
Isn't that basically a crappy, hand-rolled equivalent to a lock file?
Muromec · 2h ago
Of course it is
int_19h · 2h ago
What happens when one requires Foo 1.0 and the other requires Foo 2.0, and the two are incompatible on ABI level?
jameslars · 1h ago
Notably, a lockfile does not solve this problem either.
Muromec · 2h ago
Then you sit and cry of course
ffsm8 · 4h ago
Depending on the dependency, you can also use shadow versions right? Essentially including both versions, and providing each dependency with it's own desired version. I believe it's done with the maven shade plug-in
Never used it myself though, just read about it but never had an actual usecase
xg15 · 4h ago
The irony is that it even has something like lockfiles as well: The <dependencyManagement> section:
> Dependency management - this allows project authors to directly specify the versions of artifacts to be used when they are encountered in transitive dependencies or in dependencies where no version has been specified.
It's just less convenient because you have to manage it yourself.
kemitchell · 2h ago
npm got around to `@{author}/{package}` and `@{org}/{package}` beyond just global `{package}`, albeit midstream, rather than very early on. The jargon is "scoped packages". I've seen more adoption recently, also with scopes for particular projects, like https://www.npmjs.com/package/@babel/core
reactordev · 2h ago
The issue is what happens when libX@latest is updated and uses libd@2.0 but your other dependency libA@latest uses libd@1.3.1? In maven, crazy things happen. Sometimes it’s fine but if you have any kind of security, the version mismatch has different signatures and blows up. Ask any spring developer what happens when they have more than 1 slf4j in their classpath.
oftenwrong · 3h ago
To add: if you would like to avoid depending on Maven's dependency mediation behaviour, then a useful tool is Maven Enforcer's dependencyConvergence rule.
Seems like there's room then in the Maven ecosystem that does what maven-enforcer-plugin does, but which just looks at a lockfile to make its decisions.
arcbyte · 6h ago
I have YEARS of zero problems with maven dependencias. And yet i cant leave up a node project for more than a month without immediately encountering transitive dependency breakage that take days to resolve.
Maven is dependency heaven.
creesch · 5h ago
You might not have issues, but they definitely happen. Especially once you bring in heavy frameworks like Spring, which for some reason ship with a ton of dependencies, some of which are surprisingly outdated.
I had this happen with JUnit after a JDK upgrade. We needed to update to a newer major version of JUnit to match the new JDK, so we updated the test code accordingly. But then things broke. Methods were missing, imports couldn't be resolved, etc. Turned out something else in the dependency tree was still pulling in JUnit 4, and Maven's “nearest-wins” logic just silently went with the older version. No error, no warning. Just weird runtime/classpath issues. This something else turned out to be spring, for some odd reason it was an ancient version of Junit 4 as well.
And yeah, you can eventually sort it out, maybe with mvn dependency:tree and a lot of manual overrides, but it is a mess. And Maven still doesn't give you anything like a lockfile to consistently reproduce builds over time. That's fine if your whole org pins versions very strictly, but it's naive to claim it "just works" in all cases. Certainly because versions often don't get pinned that strictly and it is easy to set up things in such a way that you think you have pinned the version while that isn't the case. Really fun stuff..
nemothekid · 6h ago
That's a problem with node's culture, not with lockfiles. I've never experienced the level of bitrot Node suffers from in Rust, Ruby, Go, or PHP, which do have lockfiles.
potetm · 8h ago
The point isn't, "There are zero problems with maven. It solves all problems perfectly."
The point is, "You don't need lockfiles."
And that much is true.
(Miss you on twitter btw. Come back!)
hyperpape · 8h ago
I think Maven's approach is functionally lock-files with worse ergonomics. You can only use the dependency from the libraries you use, but you're waiting for those libraries to update.
As an escape hatch, you end up doing a lot of exclusions and overrides, basically creating a lockfile smeared over your pom.
P.S. Sadly, I think enough people have left Twitter that it's never going to be what it was again.
Alupis · 5h ago
> P.S. Sadly, I think enough people have left Twitter that it's never going to be what it was again.
Majority of those people came back after a while. The alternatives get near-zero engagement, so it's just shouting into the wind. For the ones that left over political reasons, receiving near-zero engagement takes all the fun out of posting... so they're back.
stack_framer · 5h ago
I'd be willing to bet that 95% of my "followers" on Twitter are bots. So I get near-zero engagement, or engagement that is worth zero.
potetm · 8h ago
Of course it's functionally lock files. They do the same thing!
There's a very strong argument that manually managing deps > auto updating, regardless of the ergonomics.
P.S. You're, right, but also it's where the greatest remnant remains. :(
shadowgovt · 6h ago
I fear it says something unfortunate about our entire subculture if the greatest remnant remains at the Nazi bar. :(
(To be generous: it might be that we didn't build our own bar the moment someone who is at least Nazi-tolerant started sniffing around for the opportunity to purchas the deed to the bar. The big criticism might be "we, as a subculture, aren't punk-rock enough.")
KerrAvon · 6h ago
JFC, get off Twitter. It's a Nazi propaganda site and you are going to be affected by that even if you think you're somehow immune.
jeltz · 8h ago
You don't need package management by the same token. C is proof of that.
Having worked professionally in C, Java, Rust, Ruby, Perl, PHP I strongly prefer lock files. They make it so much nicer to manage dependencies.
potetm · 8h ago
"There is another tool that does exactly the job of a lockfile, but better."
vs
"You can use make to ape the job of dependency managers"
wat?
jeltz · 8h ago
I have worked with Maven and dependency management is a pain. Not much nicer than vendoting dependencies like you do for C. When I first was introduced to lock files that was amazing. It solved so many problems I had with vendored dependencies, CPAN and Maven.
Just because thousands of programmers manage to suffer through your bad system every day does not make it good.
aidenn0 · 7h ago
Now you're moving the goalposts; I think lockfiles that are checked-in to version control are superior to Maven's "Let's YOLO it if your transitive dependencies conflict." Version ranges are more expressive than single-versions, and when you add lockfiles you get deterministic builds.
deepsun · 6h ago
I don't understand how Maven's YOLO is different from NPM's range.
If you force a transitive dependency in Maven, then yes, some other library may get incompatible with it. But in NPM when people declare dependency as, say, ~1.2.3 the also don't know if they will be compatible with a future 1.2.4 version. They just _assume_ the next patch release won't break anything. Yes npm will try to find a version that satisfies all declarations, but library devs couldn't know the new version would be compatible because it wasn't published at that time.
And my point is that it's _exactly_ the same probability that the next patch version is incompatible in both Maven and NPM. That's why NPM users are not afraid to depend on ~x.x or even ^x.x, they basically YOLOing.
eptcyka · 5h ago
Yeah, npm people expect that semantic versioning will be abided by. Obviously, it will not work if a minor version bump introduces a breaking change. Obviously this is better than pinning the same one dependency in literally every package - imagine the churn and the amount of life lost to bumping dependencies in any given ecosystem if every package had to pin a specific version of a dependency.
Ultimately, these are imperfect solutions to practical problems, and I know that I much prefer the semantic versioning and lockfile approach to whatever the java people are into.
beart · 4h ago
From my experience, this is a self-policing issue in the npm ecosystem. When major packages break semver, the maintainers take a ton of heat. When minor packages do it, they quickly fall out of the ecosystem entirely. It's not "YOLO"ing, but rather following the ecosystem conventions.
But anyway.. isn't that exactly the purpose of lock files? If you don't trust the semver range, it shouldn't matter because every `npm ci` results in the same package versions.
aidenn0 · 5h ago
> I don't understand how Maven's YOLO is different from NPM's range.
The person who wrote the range selected a range that they deem likely to work.
I don't use NPM, but in Python it definitely happens that you see e.g.:
foo >= 0.3.4, <= 0.5.6
Which can save a lot of headaches early on for packages that use ZeroVer[1]
Maven builds are deterministic (so long as you don't have SNAPSHOT dependencies). The version resolution is insane but deterministic. You'll only break that determinism if you change the dependencies.
That's precisely because maven doesn't support version ranges. Maven artifacts are also immutable.
Maven also supports manual override when the insane resolution strategy fails that's the "dependencymanagement" section.
aidenn0 · 5h ago
Lockfile builds are also deterministic. You only break that determinism if you change the lockfile.
I typically just add an <exclusion> for one of my top level dependency, so Maven picks from some other.
trjordan · 8h ago
There is absolutely a good reason for version ranges: security updates.
When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).
But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.
I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.
I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.
seniorsassycat · 4h ago
Yeah, this felt like a gap in the article. You'd have to wait for every package to update from the bottom up before you could update you top levels to remove a risk (or you could patch in place, or override)
But what if all the packages had automatic ci/cd, and libinsecure 0.2.1 is published, libuseful automatically tests a new version of itself that uses 0.2.1, and if it succeeds it publishes a new version. And consumers of libuseful do the same, and so on.
CognitiveLens · 3h ago
The automatic ci/cd suggestion sounds appealing, but at least in the NPM ecosystem, the depth of those dependencies would mean the top-level dependencies would constantly be incrementing. On the app developer side, it would take a lot of attention to figure when it's important to update top-level dependencies and when it's not.
skybrian · 2h ago
Go has a deterministic package manager and handles security bugs by letting library authors retract versions [1]. The 'go get' command will print a warning if you try to retrieve a retracted version. Then you can bump the version for that module at top level.
You also have the option of ignoring it if you want to build the old version for some reason, such as testing the broken version.
What if libinsecure 0.2.1 is the version that introduces the vulnerability, do you still want your application to pick up the update?
I think the better model is that your package manager let you do exactly what you want -- override libuseful's dependency on libinsecure when building your app.
trjordan · 6h ago
Of course there's no 0-risk version of any of this. But in my experience, bugs tend to get introduced with features, then slowly ironed out over patches and minor versions.
I want no security bugs, but as a heuristic, I'd strongly prefer the latest patch version of all libraries, even without perfect guarantees. Code rots, and most versioning schemes are designed with that in mind.
MarkusQ · 5h ago
Except the only reason code "rots" is that the environment keeps changing as people chase the latest shiny thing. Moreover, it rots _faster_ once the assumption that everyone is going to constantly update get established, since it can be used to justify pushing non-working garbage, on the assumption "we'll fix it in an update".
This may sound judgy, but at the heart it's intended to be descriptive: there are two roughly stable states, and both have their problems.
guhcampos · 5h ago
The author hints very briefly that Semantic Version is a hint, not a guarantee, to which I agree - but then I think we should be insisting on library maintainers that semantic versioning *should* be a guarantee, and in the worst case scenario, boycott libraries that claim to be semantically versioned but don't do it in reality.
andix · 1h ago
It can't be a guarantee. Even the smallest patches for vulnerabilities change the behavior of the code. Most of the time this is not a problem, but weird things happen all the time. Higher memory usage, slower performance, some regressions that are only relevant for a tiny amount of users, ...
SchemaLoad · 1h ago
Pretty much. Everything is a breaking change to someone. Best to just ignore sem ver and have a robust automated test suite and deployment process that minimises issues with a bad build.
oiWecsio · 4h ago
I don't understand why major.minor.patchlevel is a "hint". It had been an interface contract with shared libraries written in C when I first touched Linux, and that was 25+ years ago; way before the term "semantic version" was even invented (AFAICT).
michaelt · 2h ago
Imagine I make a library for loading a certain format of small, trusted configuration files.
Some guy files a CVE against my library, saying it crashes if you feed it a large, untrusted file.
I decide to put out a new version of the library, fixing the CVE by refusing to load conspicuously large files. The API otherwise remains unchanged.
Is the new release a major, minor, or bugfix release? As I have only an approximate understanding of semantic versioning norms, I could go for any of them to be honest.
Some other library authors are just as confused as me, which is why major.minor.patchlevel is only a hint.
shwestrick · 1h ago
I like this example.
The client who didn't notice a difference would probably call it a bugfix.
The client whose software got ever-so-slightly more reliable probably would call it a minor update.
The client whose software previously was loading large files (luckily) without issue would call it major, because now their software just doesn't work anymore.
michaelt · 1h ago
It's also an almost-real situation (although I wasn't the library developer involved)
You can Google "YAMLException: The incoming YAML document exceeds the limit" - an error introduced in response to CVE-2022-38752 - to see what happens when a library introduces a new input size limit.
What happened in that case is: the updated library bumps their version from 1.31 to 1.32; then a downstream application updates their dependencies, passes all tests, and updates their version from 9.3.8.0 to 9.3.9.0
PhilipRoman · 5h ago
Slightly off topic but we need to normalize the ability to patch external dependencies (especially transitive ones). Coming from systems like Yocto, it was mind boggling to see a company bugging the author of an open source library to release a new version to the package manager with a fix that they desperately needed.
In binary package managers this kind of workflow seems like an afterthought.
eitau_1 · 4h ago
nixpkgs shines especially bright in this exact scenario
tonsky · 8h ago
It’s totally fine in Maven, no need to rebuild or repackage anything. You just override version of libinsecure in your pom.xml and it uses the version you told it to
zahlman · 8h ago
So you... manually re-lock the parts you need to?
aidenn0 · 7h ago
Don't forget the part where Maven silently picks one version for you when there are transitive dependency conflicts (and no, it's not always the newest one).
deredede · 7h ago
Sure, I'm happy with locking the parts I need to lock. Why would I lock the parts I don't need to lock?
zaptheimpaler · 5h ago
Dependency management is a deep problem with a 100 different concerns, and every time someone says "oh here it's easy, you don't need that complexity" it turns out to only apply to a tiny subset of dependency management that they thought about.
Maven/Java does absolutely insane things, it will just compile and run programs with incompatible version dependencies and then they crash at some point, and pick some arbitrary first version of a dependency it sees. Then you start shading JARs and writing regex rules to change import paths in dependencies and your program crashes with a mysterious error with 1 google result and you spend 8 hours figuring out WTF happened and doing weird surgery on your dependencies dependencies in an XML file with terrible plugins.
This proposed solution is "let's just never use version ranges and hard-code dependency versions". Now a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?
It doesn't even understand lockfiles fully. They don't make your build non-reproducible, they give you both reproducible builds (by not updating the lockfile) and an easy way to update dependencies if and when you want to. They were made for the express purpose of making your build reproducible.
I wish there was a mega article explaining all the concerns, tradeoffs and approaches to dependency management - there are a lot of them.
yawaramin · 3h ago
> compile and run programs with incompatible version dependencies and then they crash at some point
Just because Java does this doesn't mean every language has to. It's not strongly tied to the dependency management system used. You could have this even with a Java project using lockfiles.
> a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?
Alternatively, just specify the required version in the top-level project's dependency set, as suggested in the article.
adrianmsmith · 4h ago
1) "it will just compile and run programs with incompatible version dependencies and then they crash at some point"
2) "Now a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?"
You can't solve both of these simultaneously.
If you want a library's dependences to be updated to versions other than the original library author wanted to use (e.g. because that library is unmaintained) then you're going to get those incompatibilities and crashes.
I think it's reasonable to be able to override dependencies (e.g. if something is unmaintained) but you have to accept there are going to be surprises and be prepared to solve them, which might be a bit painful, but necessary.
nixosbestos · 4h ago
Yeah, you have to bump stuff and use packages that are actually compatible. Like Rust. Which does not do the insane things that Maven does, that the post author is presumably advocating for.
andy99 · 7h ago
In case the author is reading, I can't read your article because of that animation at the bottom. I get it, it's cute, but it makes it too distracting to concentrate on the article, so I ended up just closing it.
fellowniusmonk · 6h ago
I've never seen something so egregious before, it made it impossible to read without covering it with my hand.
But I realized something by attempting to read this article several times first.
If I ever want to write an article and reduce peoples ability to critically engage with the argument in it I should add a focus pulling animation that thwarts concerted focus.
It's like the blog equivalent of public speakers who ramble their audience into a coma.
I think the same blog used to show you the cursor position of every other reader on your screen. Surprised that's been removed.
IrishTechie · 6h ago
> I've never seen something so egregious before
Streaming comments on YouTube give it a run for its money, what absolute garbage.
masklinn · 4h ago
Do you mean the live chat? Those are, appropriately, for live streams. They do replay afterwards as depending on the type of stream the video may not make complete sense without them (and they're easy enough to fold if they don't have any value e.g. premieres).
politelemon · 5h ago
Thankfully that can be collapsed.
_verandaguy · 3h ago
I'll also add that the "night mode" is obnoxious as hell anyway.
Inverted colours would've been _mostly fine._ Not great, but mostly fine, but instead, the author went out of their way to add this flashlight thing that's borderline unusable?
What the hell is this website?
a57721 · 1h ago
I'm sure this feature is a joke, not intended to be usable. It's a personal blog.
nerdjon · 5h ago
Agreed, and the fact that there is not an easy "x" to close it is even worse.
If you want to do something cute and fun, whatever its your site. But if you actually want people to use your site make it easy to dismiss. We already have annoying ads and this is honestly worse than many ads.
Also, from the bio that I can barely see he writes about "UI Design" and... included this?
somehnguy · 7h ago
I read the article but that animation was incredibly distracting. I don't even understand what it's for - clicking it does nothing. Best guess is a representation of how many people active on page.
ddejohn · 5h ago
It's downright awful and I'm having a hard time imagining the author proof reading their own page and thinking "yeah, that's great".
As an aside, I have an article in my blog that has GIFs in it, and they're important for the content, but I'm not a frontend developer by any stretch of the imagination so I'm really at wit's end for how to make it so that the GIFs only play on mouse hover or something else. If anybody reading has some tips, I'd love to hear them. I'm using Zola static site generator, and all I've done is make minor HTML and CSS tweaks, so I really have no idea what I'm doing where it concerns frontend presentation.
yladiz · 5h ago
As someone who does like tonsky’s stuff sometimes: I immediately closed the article when I saw it. I’m less charitable than you: it’s not cute, it’s just annoying, and it should be able to be switched off. For me it goes into the same box as his “dark mode” setting but it’s worse because it can’t be disabled. Why should I, as the reader, put in effort to overcome something the author found “cute” just to read their words? It’s akin to aligning the words to the right, or vertically: I can read it but it’s so much work that I’d rather just not.
hans_castorp · 4h ago
On sites like that, I typically just switch to "reader view" which the leaves only the interesting content.
mvieira38 · 6h ago
Give in to the noJS movement, there's no animation and it's a beautiful minimalistic site if you disable javascript
hackrmn · 4h ago
So Tonsky's punishing us for leaving JavaScript enabled?
fennecbutt · 6h ago
It also covers a whole 1/4 of the screen on mobile...
Even worse, it exposes the city all viewers within the HTML, even if the country code is only displayed on the webpage.
Obviously, the server gets your IP when you connect but ideally it doesn't share that with all visitors. This isn't as bad as that, still concerning.
dom96 · 3h ago
The animation? For me it was the blinding yellow background
jerhewet · 3h ago
Reader mode. Don't leave home without it.
jedahan · 4h ago
I wonder if it respects prefers-reduced-motion, though I don't know if I have that set in my browser, I do have it set with my OS.
appease7727 · 6h ago
Wow, that's one of the most abhorrent web designs I've ever seen
rs186 · 3h ago
When I saw the yellow background, I knew this is the website where I read the Unicode article [1]. Sure it is. With great pain I finished it.
I mean, just the fact that the background is yellow is a terrible UX decision. Not to mention that ridiculous "dark mode". No, it's not funny. It's stupid and distracting.
I did document.querySelector('#presence').remove();
daveidol · 4h ago
Yeah I just popped into devtools and added "display: none" to the CSS. It was crazy distracting.
pak9rabid · 4h ago
Yes is horrible, this idea.
jraph · 4h ago
To all people in this sub thread: suggestion to try reader mode.
hn-acct · 3h ago
Thanos snap it if you’re using ios
sneak · 1h ago
My favorite websites are the weird ones that make people complain about stuff.
J37T3R · 6h ago
In addition, the solid yellow background is another readability impediment.
sethpurc · 6h ago
Same here, I also closed it within a few seconds.
sitkack · 4h ago
In ublock origin
tonsky.me##.container
epage · 8h ago
Let's play this out in a compiled language like Cargo.
If every dependency was a `=` and cargo allowed multiple versions of SemVer compatible packages.
The first impact will be that your build will fail. Say you are using `regex` and you are interacting with two libraries that take a `regex::Regex`. All of the versions need to align to pass `Regex` between yourself and your dependencies.
The second impact will be that your builds will be slow. People are already annoyed when there are multiple SemVer incompatible versions of their dependencies in their dependency tree, now it can happen to any of your dependencies and you are working across your dependency tree to get everything aligned.
The third impact is if you, as the application developer, need a security fix in a transitive dependency. You now need to work through the entire bubble up process before it becomes available to you.
Ultimately, lockfiles are about giving the top-level application control over their dependency tree balanced with build times and cross-package interoperability. Similarly, SemVer is a tool any library with transitive dependencies [0]
> All of the versions need to align to pass `Regex` between yourself and your dependencies.
No, they don't. As the article explains, the resolution process will pick the version that is 'closest to the root' of the project.
> The second impact will be that your builds will be slow....you are working across your dependency tree to get everything aligned.
As mentioned earlier, no you're not. So there's nothing to support the claim that builds will be slower.
> You now need to work through the entire bubble up process before it becomes available to you.
No you don't, because as mentioned earlier, the version that is 'closest to root' will be picked. So you just specify the security fixed version as a direct dependency and you get it immediately.
hosh · 8h ago
Wasn’t the article suggesting that the top level dependencies override transitive dependencies, and that could be done in the main package file instead of the lock file?
junon · 7h ago
You should not be editing your cargo.lock file manually. Cargo gives you a first-class way of overriding transitive dependencies.
richardwhiuk · 4h ago
You can also do cargo update -p
oblio · 6h ago
Java is compiled, FYI.
deepsun · 6h ago
And interpreted.
Some call transforming .java to .class a transpilation, but then a lot of what we call compilation should also be called transpilation.
Well, Java can ALSO be AOT compiled to machine code, more popular nowadays (e.g. GraalVM).
maxmcd · 5h ago
For what it's worth I think Go's MVS somewhat meets the desire here. It does not require lockfiles, but also doesn't allow use of multiple different minor/patch versions of a library: https://research.swtch.com/vgo-mvs
I believe Zig is also considering adopting it.
If there are any dependencies with the same major version the algorithm simply picks the newest one of them all (but not the newest in the package registry), so you don't need a lockfile to track version decisions.
Go's go.sum contains checksums to validate content, but is not required for version selection decisions.
vl · 4h ago
Indirect require section in go.mod file is essentially a lockfile. Once decision is made by tool, it's codified for future builds.
maxmcd · 3h ago
The //indirect dependencies I believe are just there to track dependencies that are not in the project, or to help with caching: https://github.com/golang/go/issues/36460
In go 1.17 they were added so that project loading did not require downloading the go.mod of every dependency in the graph.
nycticorax · 5h ago
Strongly endorse. That paper is really wonderful. It seems to me that MVS is the solution to the version selection problem, and now we just have to wait for awareness of this to fully percolate through the developer community.
jchw · 5h ago
Go MVS ought to be deterministic, but it still benefits from modules having lockfiles as it allows one to guarantee that the resolution of modules is consistent without needing to trust a central authority.
Go's system may be worth emulating in future designs. It's not perfect (still requires some centralized elements, module identities for versions ≥2 are confusing, etc.) but it does present a way to both not depend strongly on specific centralized authorities without also making any random VCS server on the Internet a potential SPoF for compiling software. On the other hand, it only really works well for module systems that purely deal with source code and not binary artifacts, and it also is going to be the least hazardous when fetching and compiling modules is defined to not allow arbitrary code execution. Those constraints together make this system pretty much uniquely suited to Go for now, which is a bit of a shame, because it has some cool knock-on effects.
(Regarding deterministic MVS resolution: imagine a@1.0 depending on b@1.0, and c@1.0 depending on a@1.1. What if a@1.1 no longer depends on b? You can construct trickier versions of this possibly using loops, but the basic idea is that it might be tricky to give a stable resolution to version constraints when the set of constraints that are applied depends on the set of constraints that are applied. There are possible deterministic ways to resolve this of course, it's just that a lot of these edge cases are pretty hard to reason about and I think Go MVS had a lot of bugs early on.)
lalaithion · 8h ago
What if your program depends on library a1.0 and library b1.0, and library a1.0 depends on c2.1 and library b1.0 depends on c2.3? Which one do you install in your executable? Choosing one randomly might break the other library. Installing both _might_ work, unless you need to pass a struct defined in library c from a1.0 to b1.0, in which case a1.0 and b1.0 may expect different memory layouts (even if the public interface for the struct is the exact same between versions).
The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.
tonsky · 8h ago
One of the versions will be picked up. If that version doesn’t work, you can try another one. The process is exactly the same
Joker_vD · 8h ago
> If that version doesn’t work, you can try another one.
And how will this look like, if your app doesn't have library C mentioned in its dependencies, only libraries A and B? You are prohibited from answering "well, just specify all the transitive dependencies manually" because it's precisely what a lockfile is/does.
tonsky · 7h ago
Maven's version resolution mechanism determines which version of a dependency to use when multiple versions are specified in a project's dependency tree. Here's how it works:
- Nearest Definition Wins: When multiple versions of the same dependency appear in the dependency tree, the version closest to your project in the tree will be used.
- First Declaration Wins: If two versions of the same dependency are at the same depth in the tree, the first one declared in the POM will be used.
Joker_vD · 7h ago
Well, I guess this works if one appends their newly-added dependencies are appended at the end of the section in the pom.xml instead of generating it alphabetically sorted just in time for the build.
yawaramin · 3h ago
Maven pom.xml files are maintained by hand, not by some code generation tool.
deredede · 7h ago
It's not "all the transitive dependencies". It's only the transitive dependencies you need to explicitly specify a version for because the one that was specified by your direct dependency is not appropriate for X reason.
deredede · 7h ago
Alternative answer: both versions will be picked up.
It's not always the correct solution, but sometimes it is. If I have a dependency that uses libUtil 2.0 and another that uses libUtil 3.0 but neither exposes types from libUtil externally, or I don't use functions that expose libUtil types, I shouldn't have to care about the conflict.
shadowgovt · 6h ago
This points to a software best-practice: "Don't leak types from your dependencies." If your package depends on A, never emit one of A's structs.
Good luck finding a project of any complexity that manages to adhere to that kind of design sensibility religiously.
(I think the only language I've ever used that provided top-level support for recognizing that complexity was SML/NJ, and it's been so long that I don't remember exactly how it was done... Modules could take parameters so at the top level you could pass to each module what submodule it would be using, and only then could the module emit types originating from the submodule because the passing-in "app code" had visibility on the submodule to comprehend those types. It was... Exactly as un-ergonomic as you think. A real nightmare. "Turn your brain around backwards" kind of software architecting.)
deredede · 4h ago
I can think of plenty situations where you really want to use the dependency's types though. For instance the dependency provides some sort of data structure and you have one library that produces said data structure and a separate library that consumes it.
What you're describing with SML functors is essentially dependency injection I think; it's a good thing to have in the toolbox but not a universal solution either. (I do like functors for dependency injection, much more than the inscrutable goo it tends to be in OOP languages anyways)
shadowgovt · 3h ago
I can think of those situations too, and in practice this is done all the time (by everyone I know, including me).
In theory... None of us should be doing it. Emitting raw underlying structures from a dependency coupled with ranged versioning means part of your API is under-specified; "And this function returns an argument, the type of which is whatever this third-party that we don't directly communicate with says the type is." That's hard to code against in the general case (but it works out often enough in the specific case that I think it's safe to do 95-ish percent of the time).
int_19h · 2h ago
It works just fine in C land because modifying a struct in any way is an ABI breaking change, so in practice any struct type that is exported has to be automatically deemed frozen (except for major version upgrades where compat is explicitly not a goal).
Alternatively, it's a pointer to an opaque data structure. But then that fact (that it's a pointer) is frozen.
Either way, you can rely on dependencies not just pulling the rug from under you.
shadowgovt · 2h ago
I like this answer. "It works just fine in C land because this is a completely impossible story in C land."
(I remember, ages ago, trying to wrap my head around Component Object Model. It took me awhile to grasp it in the abstract because, I finally realized, it was trying to solve a problem I'd never needed to solve before: ABI compatibility across closed-source binaries with different compilation architectures).
Revisional_Sin · 3h ago
So you need to test if the version worked yourself (e.g. via automated tests)? Seems better to have the library author do this for you and define a range.
RangerScience · 5h ago
No, no, a thousand times no.
The package file (whatever your system) is communication to other humans about what you know about the versions you need.
The lockfile is the communication to other computers about the versions you are using.
What you shouldn't have needed is fully defined versions in your package files (but you do need it, in case some package or another doesn't do a good enough job following semver)
So, this:
package1: latest
# We're stuck on an old version b/c of X, Y, Z
package2: ~1.2
(Related: npm/yarn should use a JSON variant (or YAML, regular or simplified) that allows for comments for precisely this reason)
skybrian · 2h ago
With deterministic version control, library authors are supposed to document the exact version that a library was tested with. (Or the oldest version that they tested with and still works.)
People who use a library might use newer versions (via diamond dependencies or because they use latest), but it will result in a combination of dependencies that wasn't tested by the library's authors. Often that's okay because libraries try to maintain backward compatibility.
Old libraries that haven't had a new release in a while are going to specify older dependencies and you just have to deal with that. The authors aren't expected to guess which future versions will work. They don't know about security bugs or broken versions of dependencies that haven't been released yet. There are other mechanisms for communicating about that.
nemothekid · 5h ago
Most of the issues in this thread and the article, are, IMO, problems with Node, not with lockfiles.
>How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.
Yes, this is a social contract. Not everything in the universe can be locked into code, and with Semantic versioning, we hope that our fellow humans won't unnecessarily break packages in non-major releases. It happens, and people usually apologize and fix, but it's rare.
This has worked successfully if you look at RubyGems which is 6 years older than npm (although Gemfile.lock was introduced in 2010, npm didn't introduce it until 2017).
RubyGems doesn't have the same reputation for dysfunction as Node does. Neither does Rust, Go, PHP, and Haskell. Even more that I probably don't use a daily basis. Node is the only language that I will come back and find a docker container that straight up won't build or a package that requires the entire dependency tree to update because one package pushed a minor-version change that ended up requiring a minor version change to Node, then that new version of Node isn't compatible with some hack that another package did in it's C extension.
In fact, I expect some Node developer to read this article and deploy yet another tool that will break _everything_ in the build process. In other languages I don't even think I've ever really thought about dependency resolution in years.
bravesoul2 · 1h ago
Semver is best effort. It may work but spacebar heating to change VIM mode (or whatever the XKCD joke was) may get lost in your minor update.
But it is still useful. It's like a bloom filter. Most of the time you can happily pull minor or patch upgrades with no issue. Occasionally it will break. But that's less work than analysing every upgrade.
simonw · 8h ago
I see lockfiles as something you use for applications you are deploying - if you run something like a web app it's very useful to know exactly what is being deployed to production, make sure it exactly matches staging and development environments, make sure you can audit new upgrades to your dependencies etc.
This article appears to be talking about lockfiles for libraries - and I agree, for libraries you shouldn't be locking exact versions because it will inevitably pay havoc with other dependencies.
Or maybe I'm missing something about the JavaScript ecosystem here? I mainly understand Python.
kaelwd · 8h ago
The lockfile only applies when you run `npm install` in the project directory, other projects using your package will have their own lockfile and resolve your dependencies using only your package.json.
aidenn0 · 8h ago
I think you missed the point of the article. Consider Application A, that depends on Library L1. Library L1 in turn depends on Library L2:
A -> L1 -> L2
They are saying that A should not need a lockfile because it should specify a single version of L1 in its dependencies (i.e. using an == version check in Python), which in turn should specify a single version of L2 (again with an == version check).
Obviously if everybody did this, then we wouldn't need lockfiles (which is what TFA says). The main downsides (which many comments here point out) are:
1. Transitive dependency conflicts would abound
2. Security updates are no longer in the hands of the app developers (in my above example, the developer of A1 is dependent on the developer of L1 whenever a security bug happens in L2).
3. When you update a direct dependency, your transitive dependencies may all change, making what you that was a small change into a big change.
(FWIW, I put these in order of importance to me; I find #3 to be a nothingburger, since I've hardly ever updated a direct dependency without it increasing the minimum dependency of at least one of its dependencies).
yawaramin · 3h ago
> Transitive dependency conflicts would abound
They would be resolved by just picking the version 'closest to root', as explained in the article.
> Security updates are no longer in the hands of the app developers
It is, the app developers can just put in a direct dependency on the fixed version of L2. As mentioned earlier, this is the version that will be resolved for the project.
> When you update a direct dependency, your transitive dependencies may all change, making what you that was a small change into a big change.
This is the same even if you use a lockfile system. When you update dependencies you are explicitly updating the lockfile as well, so a bunch of transitive dependencies can change.
crote · 1h ago
> They would be resolved by just picking the version 'closest to root', as explained in the article.
Which is going to lead to horrible issues when that library isn't compatible with all your other dependencies. What if your app directly depends on both L1 and L2, but L1 is compatible with L3 1.2 ... 1.5 while L2 is compatible with L3 1.4 ... 1.7? A general "stick to latest" policy would have L1: "L3==1.5", L2: "L3==1.7" (which breaks L1 if L2 wins). A general "stick to oldest compatible" policy would have L1: "L3==1.2", L2: "L3==1.4" (which breaks L2 if L1 wins).
The obvious solution would be to use L3 1.4 ... 1.5 - but that will never happen without the app developer manually inspecting the transitive dependencies and hardcoding the solution - in essence reinventing the lock file.
> It is, the app developers can just put in a direct dependency on the fixed version of L2. As mentioned earlier, this is the version that will be resolved for the project.
And how is that going to work out in practice? Is that direct dependency supposed to sit in your root-level spec file forever? Will there be a special section for all the "I don't really care about this, but we need to manually override it for now" dependencies? Are you going to have to manually specify and bump it until the end of time because you are at risk of your tooling pulling in the vulnerable version? Is there going to be tooling which automatically inspects your dependencies and tells you when it is safe to drop?
> This is the same even if you use a lockfile system. When you update dependencies you are explicitly updating the lockfile as well, so a bunch of transitive dependencies can change.
The difference is that in the lockfile world any changes to transitive dependencies are well-reasoned. If every package specifies a compatibility range for its dependencies, the dependency management system can be reasonably sure that any successful resolution will not lead to issues and that you are getting the newest package versions possible.
With a "closest-to-root" approach, all bets are off. A seemingly-trivial change in your direct dependencies can lead to a transitive dependency completely breaking your entire application, or to a horribly outdated library getting pulled in. Moreover, you might not even be aware that this is happening. After all, if you were keeping track of the specific versions of every single transitive dependency, you'd essentially be storing a lockfile - and that's what you were trying to avoid...
hosh · 8h ago
Is the article also suggesting that if there are version conflicts, it goes with the top level library? For example, if we want to use a secure version of L2, it would be specified at A, ignoring the version specified by L1?
Or maybe I misread the article and it did not say that.
aidenn0 · 7h ago
It's maybe implied since Maven lets you do that (actually it uses the shallowest dependency, with the one listed first winning ties), but the thrust of the article seems to be roughly: "OMGWTFBBQ we can't use L2 with 0.7.9 if L1 was only tested with 0.7.9!" so I don't know how the author feels about that.
[edit]
The author confirmed that they are assuming Maven's rules and added it to the bottom of their post.
boscillator · 8h ago
Ok, but what happens when lib-a depends on lib-x:0.1.4 and lib-b depends on lib-x:0.1.5, even though it could have worked with any lib-x:0.1.*? Are these libraries just incompatible now? Lockfiles don't guarantee that new versions are compatible, but it guarantees that if your code works in development, it will work in production (at least in terms of dependencies).
I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)
aidenn0 · 7h ago
> I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)
You are wrong; Maven just picks one of lib-x:0.1.4 or lib-x:0.1.5 depending on the ordering of the dependency tree.
Tadpole9181 · 4h ago
Maven will also silently choose different minor and major versions, destroying your application. Sometimes at compile time, sometimes at runtime.
Java dependency management is unhinged, antiquated garbage to anyone who has used any other ecosystem.
oftenwrong · 34m ago
Maven is not Java, though.
Tadpole9181 · 2m ago
Gradle suffers the same exact issue by default, because it inherits it from Maven (they use the same repository). You need to go out of your way to enable strict versioning policies and lock files.
Maven and Gradle make up the vast majority of all Java projects in the wild today. So, effectively, Maven is Java in terms of dependency management.
yladiz · 4h ago
How do you change the order?
adrianmsmith · 4h ago
You go into your pom.xml file (bunch of <dependency>) using a text editor and change the order.
shadowgovt · 6h ago
> Are these libraries just incompatible now?
Python says "Yes." Every environment manager I've seen, if your version ranges don't overlap for all your dependencies, will end up failing to populate the environment. Known issue; some people's big Python apps just break sometimes and then three or four open source projects have to talk to each other to un-fsck the world.
npm says "No" but in a hilarious way: if lib-a emits objects from lib-x, and lib-b emits objects from lib-x, you'll end up with objects that all your debugging tools will tell you should be the same type, and TypeScript will statically tell you are the same type, but don't `instanceof` the way you'd expect two objects that are the same type should. Conclusion: `instanceof` is sus in a large program; embrace the duck typing (and accept that maybe your a-originated lib-x objects can't be passed to b-functions without explosions because I bet b didn't embrace the duck-typing).
pnt12 · 56m ago
To me, it's the opposite. In Python, every time I see 'requirements.txt' without lock files, I cry a bit.
A: You're handling problem X and then unrelated problem Y suddenly arises because you're not locking package versions thoroughly. It's not fun.
B: Now the opposite. You lock all versions of the libs you use. You use renovate or schedule time for updates periodically. You have a thorough test suite that you can automatically exercise when trying the new updates. You can apply the updates and deoy the new version to a test environment to run a final test manually. Things look good. You deploy to production and, quite often, things go smoothly.
A is the blue pill, easy to taste but things are out of your control and will bite you eventually. B is the red pill: you're in control, for the better or worst.
pnt12 · 55m ago
(sorry for the bad Matrix analogy)
_verandaguy · 3h ago
> But... why would libpupa’s author write a version range that includes versions that don’t exist yet? How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.
> For that, kids, I have no good answer.
Because semantic version is good enough for me, as a package author, to say with a good degree of confidence, "if security or stability patches land within the patch (or sometimes, even minor) fields of a semver version number, I'd like to have those rolled out with all new installs, and I'm willing to shoulder the risk."
You actually kind-of answer your own question with this bit. Semver not being a guarantee of anything is true, but I'd extend this (and hopefully it's not a stretch): package authors will republish packages with the same version number, but different package contents or dependency specs. Especially newer authors, or authors new to a language or packaging system, or with packages that are very early in their lifecycle.
There are also cases where packages get yanked! While this isn't a universally-available behaviour, many packaging systems acknolwedge that software will ship with unintentional vulnerabilities or serious stability/correctness issues, and give authors the ability to say, "I absolutely have to make sure that nobody can install this specific version again because it could cause problems." In those cases, having flexible subdependency version constraints helps.
It might be helpful to think by analogy here. If a structure is _completely rigid,_ it does have some desirable properties, not the least of which being that you don't have to account for the cascading effects of beams compressing and extending, elements of the structure coming under changing loads, and you can forget about accounting for thermal expansion or contraction and other external factors. Which is great, in a vacuum, but structures exist in environments, and they're subject to wear from usage, heat, cold, rain, and (especially for taller structures), high winds. Incorporating a planned amount of mechanical compliance ends up being the easier way to deal with this, and forces the engineers behind it to account for failure modes that'll arise over its lifetime.
ratelimitsteve · 8h ago
anyone find a way to get rid of the constantly shifting icons at the bottom of the screen? I'm trying to read and the motion keeps pulling my attention away from the words toward the dancing critters.
foobarbecue · 8h ago
$("#presence").remove()
And yeah, I did that right away. Fun for a moment but extremely distracting.
I use NoScript, which catches all of these sorts of things by default. I only enable first-party JS when there's a clear good reason why the site should need it, and third-party JS basically never beyond NoScript's default whitelist.
vvillena · 8h ago
Reader mode.
trinix912 · 8h ago
Block ###presence with UBlock.
wedn3sday · 6h ago
I absolutely abhor the design of this site. I cannot engage with the content as Im filled with a deep burning hatred of the delivery. Anyone making a personal site: do not do this.
sjrd · 3h ago
Having read the article and read some of the comments here, I think many could learn from dependency management rules in the Scala ecosystem.
Scala uses Maven repositories (where the common practice is to use fixed dependency versions) but with different resolution rules:
* When there are conflicting transitive versions, the highest number prevails (not the closest to the root).
* Artifacts declare the versioning scheme they use (SemVer is common, but there are others)
* When resolving a conflict, the resolution checks whether the chosen version is compatible with the evicted version according to the declared version scheme. If incompatible, an error is reported.
* You can manually override a transitive resolution and bypass the error if you need to.
The above has all the advantages of all the approaches advocated for here:
* Deterministic, time-independent resolution.
* No need for lock files.
* No silent eviction of a version in favor of an incompatible one.
* For compatible evictions, everything works out of the box.
* Security update in a transitive dependency? No problem, declare a dependency on the new version. (We have bots that even automatically send PRs for this.)
* Conflicting dependencies, but you know what you're doing? No problem, force an override.
spooky_deep · 8h ago
> The important point of this algorithm is that it’s fully deterministic.
The algorithm can be deterministic, but fetching the dependencies of a package is not.
It is usually an HTTP call to some endpoint that might flake out or change its mind.
Lock files were invented to make it either deterministic or fail.
Even with Maven, deterministic builds (such as with Bazel) lock the hashes down.
This article is mistaken.
Tainnor · 5h ago
Maven artifacts are immutable, so the whole resolution is deterministic (even if hard to understand), unless you're using snapshot versions (which are mutable) or you use version ranges (which is rare in the Maven world).
beart · 4h ago
Maven artifacts are not immutable. Some maven repositories may prevent overwriting an already published version, but this is not guaranteed. I've personally seen this cause problems where a misconfigured CI job overwrote already published versions.
npm used to allow you to unpublish (and may be overwrite?) published artifacts, but they removed that feature after a few notable events.
Edit: I was not quite correct. It looks like you can still unpublish, but with specific criteria. However, you cannot ever publish a different package using the same version as an already published package.
> Maven artifacts are immutable, so the whole resolution is deterministic
Nope, Maven will grab anything which happens to have a particular filename from `~/.m2`, or failing that it will accept whatever a HTTP server gives it for a particular URL. It can compare downloaded artifacts against a hash; but that's misleading, since those hashes are provided by the same HTTP server as the artifact! (Useful for detecting a corrupt download; useless for knowing anything about the artifact or its provenance, etc.)
If your model is that you trust Maven to never change anything, then sure.
However, I think most people in the reproducible build space would consider Maven an external uncontrolled input.
mystifyingpoi · 5h ago
I never understood this. I can delete anything from Nexus and reupload something else in its place. Is this supposed immutability just a convention that's followed?
xp84 · 2h ago
This is weird to me. (Note: i'll use ruby terms like 'gem' and 'bundle' but the same basic deal applies everywhere)
Generally our practice is to pin everything to major versions, in ruby-speak this means like `gem 'net-sftp', '~> 4.0'` which allows 4.0.0 up to 4.9999.9999 but not 5. Exceptions for non-semver such as `pg` and `rails` which we just pin to exact versions and monitor manually. This little file contains our intentions of which gems to update automatically and for any exceptions, why not.
Then we encourage aggressive performances of `bundle update` which pulls in tons of little security patches and minor bugfixes frequently, but intentionally.
Without the lockfile though, you would not be able to do our approach. Every bundle install would be a bundle update, so any random build might upgrade a gem without anyone even meaning to or realizing it, so, your builds are no longer reproducible.
So we'd fix reproducibility by reverting to pinning everything to X.Y.Z, specifically to make the build deterministic, and then count on someone to go in and update every gem's approved version numbers manually on a weekly or monthly basis. (yeah right, definitely will happen).
bravesoul2 · 1h ago
This works to a point. It would work well on a curated ecosystem. You could get to the point where one package with one version has everything you need. They are called SDKs.
However for NPM you will hit issues where 2 packages need a different React version and if you want to use them both you need to pick. In addition it is better for security. The lock file is a distributed checksum. Not impervious to supply chain attacks but better equipped than trusting the package author not to retrospectively bump (I guess you could have a hash for this tbat included the downloaded source code and claimed deps).
chriswarbo · 2h ago
I would agree with this if the author's examples were using hashes, rather than "version numbers". Specifying a hash lets us check whether any random blob of code is or isn't what we specified; versions can't do this, because any blob of code can claim to have any name, version, etc. it likes. As long as we have a hash, we don't need version numbers (or names, though it's usually helpful to provide them).
Using hashes also makes it easier to distribute, fetch, proxy, etc. since there's no need for trust. In contrast, fetching code based only on (name and) version number requires more centralised repositories with a bunch of security hoops to jump through.
Lockfiles are essential for somewhat reproducible builds.
If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.
tonsky · 8h ago
> Lockfiles are essential for somewhat reproducible builds.
No they are not. Fully reproducible builds have existed without lockfiles for decades
its-summertime · 7h ago
of distros, they usually refer to an upstream by hash
"they are not lockfiles!" is a debatable separate topic, but for a wider disconnected ecosystem of sources, you can't really rely on versions being useful for reproducibility
andix · 7h ago
> they usually refer to an upstream by hash
exactly the same thing as a lockfile
andix · 7h ago
Sure, without package managers.
It's also not about fully reproducible builds, it's about a tradeoff to get modern package manger (npm, cargo, ...) experience and also somewhat reproducible builds.
chriswarbo · 2h ago
> modern package manger (npm, cargo, ...) experience
Lol, the word "modern" has truly lost all meaning. Your list of "modern package managers" seems to coincide with a list of legacy tooling I wrote four years ago! https://news.ycombinator.com/item?id=29459209
pluto_modadic · 6h ago
...source?
show me one "decades old build" of a major project that isn't based on 1) git hashes 2) fixed semver URLs or 3) exact semver in general.
jedberg · 8h ago
The entire article is about why this isn't the case.
andix · 7h ago
It suggests a way more ridiculous fix. As mentioned by other comments in detail (security patches for transient dependencies, multiple references to the same transient dependency).
yawaramin · 3h ago
The article and various comments in this same thread have explained why these are not real issues because the resolution process picks the version 'closest to root'.
freetonik · 8h ago
In the world of Python-based end-user libraries the pinned (non-ranged) versions result in users being unable to use your library in an environment with other libraries. I’d love to lock my library to numpy 2.3.4, but if the developers of another library pin theirs to 2.3.5 then game over.
For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).
Maven, by default, does not check your transitive dependencies for version conflicts. To do that, you need a frustrating plugin that produces much worse error messages than NPM does: https://ourcraft.wordpress.com/2016/08/22/how-to-read-maven-....
How does Maven resolve dependencies when two libraries pull in different versions? It does something insane. https://maven.apache.org/guides/introduction/introduction-to....
Do not pretend, for even half a second, that dependency resolution is not hell in maven (though I do like that packages are namespaced by creators, npm shoulda stolen that).
Lockfiles are great.
Im a big fan of anything Aphex Twin for these type of sessions.
The solution is version ranges, but this then necessitates lockfiles, to avoid the problem of uncontrolled upgrades.
That said, there's an option that uses version ranges, and avoids nondeterminism without lockfiles: https://matklad.github.io/2024/12/24/minimal-version-selecti....
Note: maven technically allows version ranges, but they're rarely used.
Or have I misunderstood?
For example, as a developer I want Spring to stay on 6.3.x and not suddenly jump to 6.4 - as that is likely to break stuff. I do not care whether I get 6.3.1 or 6.3.6, as they are quite unlikely to cause issues. I do not care the slightest what version of libfoobar I get 5 dependencies down the line.
However, I do not want packages to suddenly change versions between different CI runs. Breakage due to minor version bumps are unlikely, but they will happen. That kind of stuff is only going to cause noise when a rebuild of a PR causes it to break with zero lines changed, so you only want version upgrades to happen specifically when you ask for them. On top of that there's the risk of supply chain attacks, where pulling in the absolute latest version of every single package isn't exactly a great idea.
The package spec allows me to define "spring at 6.3.x", the lockfile stores that we are currently using "spring 6.3.5, libfoobar 1.2.3". I ask it to look for upgrades, it resolves to "spring 6.3.6, libfoobar 1.4.0". There's also a "spring 6.4.0" available, but the spec says that we aren't interested so it gets ignored. All tests pass, it gets merged, and we'll stay at those versions until we explicitly ask for another upgrade.
The whole flow exists for things which aren't root dependencies. The major versions of those are trivial to keep track of manually, and you'll only have a handful of them. It's all the minor versions and downstream dependencies which are the issue: tracking them all manually is a nightmare, and picking whatever happens to be the latest version at time-of-build is a massive no-no.
For those times where that's not the case, you can look at the dependency tree to see which is included and why. You can then add a <dependency> override in your pom.xml file specifying the one you want.
It's not an "insane" algorithm. It gives you predictability. If you write something in your pom.xml that overrides whatever dependency your dependency requires, because you can update your pom.xml if you need to.
And because pom.xml is hand-written there are very few merge conflicts (as much as you'd normally find in source code), vs. a lock file where huge chunks change each time you change a dependency, and when it comes to a merge conflict you just have to delete the lot and redo it and hope nothing important has been changed.
Isn't that basically a crappy, hand-rolled equivalent to a lock file?
Never used it myself though, just read about it but never had an actual usecase
> Dependency management - this allows project authors to directly specify the versions of artifacts to be used when they are encountered in transitive dependencies or in dependencies where no version has been specified.
It's just less convenient because you have to manage it yourself.
https://maven.apache.org/enforcer/enforcer-rules/index.html
Maven is dependency heaven.
I had this happen with JUnit after a JDK upgrade. We needed to update to a newer major version of JUnit to match the new JDK, so we updated the test code accordingly. But then things broke. Methods were missing, imports couldn't be resolved, etc. Turned out something else in the dependency tree was still pulling in JUnit 4, and Maven's “nearest-wins” logic just silently went with the older version. No error, no warning. Just weird runtime/classpath issues. This something else turned out to be spring, for some odd reason it was an ancient version of Junit 4 as well.
And yeah, you can eventually sort it out, maybe with mvn dependency:tree and a lot of manual overrides, but it is a mess. And Maven still doesn't give you anything like a lockfile to consistently reproduce builds over time. That's fine if your whole org pins versions very strictly, but it's naive to claim it "just works" in all cases. Certainly because versions often don't get pinned that strictly and it is easy to set up things in such a way that you think you have pinned the version while that isn't the case. Really fun stuff..
The point is, "You don't need lockfiles."
And that much is true.
(Miss you on twitter btw. Come back!)
As an escape hatch, you end up doing a lot of exclusions and overrides, basically creating a lockfile smeared over your pom.
P.S. Sadly, I think enough people have left Twitter that it's never going to be what it was again.
Majority of those people came back after a while. The alternatives get near-zero engagement, so it's just shouting into the wind. For the ones that left over political reasons, receiving near-zero engagement takes all the fun out of posting... so they're back.
There's a very strong argument that manually managing deps > auto updating, regardless of the ergonomics.
P.S. You're, right, but also it's where the greatest remnant remains. :(
(To be generous: it might be that we didn't build our own bar the moment someone who is at least Nazi-tolerant started sniffing around for the opportunity to purchas the deed to the bar. The big criticism might be "we, as a subculture, aren't punk-rock enough.")
Having worked professionally in C, Java, Rust, Ruby, Perl, PHP I strongly prefer lock files. They make it so much nicer to manage dependencies.
vs
"You can use make to ape the job of dependency managers"
wat?
Just because thousands of programmers manage to suffer through your bad system every day does not make it good.
If you force a transitive dependency in Maven, then yes, some other library may get incompatible with it. But in NPM when people declare dependency as, say, ~1.2.3 the also don't know if they will be compatible with a future 1.2.4 version. They just _assume_ the next patch release won't break anything. Yes npm will try to find a version that satisfies all declarations, but library devs couldn't know the new version would be compatible because it wasn't published at that time.
And my point is that it's _exactly_ the same probability that the next patch version is incompatible in both Maven and NPM. That's why NPM users are not afraid to depend on ~x.x or even ^x.x, they basically YOLOing.
Ultimately, these are imperfect solutions to practical problems, and I know that I much prefer the semantic versioning and lockfile approach to whatever the java people are into.
But anyway.. isn't that exactly the purpose of lock files? If you don't trust the semver range, it shouldn't matter because every `npm ci` results in the same package versions.
The person who wrote the range selected a range that they deem likely to work.
I don't use NPM, but in Python it definitely happens that you see e.g.:
Which can save a lot of headaches early on for packages that use ZeroVer[1]1: https://0ver.org/
No comments yet
That's precisely because maven doesn't support version ranges. Maven artifacts are also immutable.
Maven also supports manual override when the insane resolution strategy fails that's the "dependencymanagement" section.
https://maven.apache.org/enforcer/enforcer-rules/versionRang...
When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).
But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.
I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.
I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.
But what if all the packages had automatic ci/cd, and libinsecure 0.2.1 is published, libuseful automatically tests a new version of itself that uses 0.2.1, and if it succeeds it publishes a new version. And consumers of libuseful do the same, and so on.
You also have the option of ignoring it if you want to build the old version for some reason, such as testing the broken version.
[1] https://go.dev/ref/mod#go-mod-file-retract
I think the better model is that your package manager let you do exactly what you want -- override libuseful's dependency on libinsecure when building your app.
I want no security bugs, but as a heuristic, I'd strongly prefer the latest patch version of all libraries, even without perfect guarantees. Code rots, and most versioning schemes are designed with that in mind.
This may sound judgy, but at the heart it's intended to be descriptive: there are two roughly stable states, and both have their problems.
Some guy files a CVE against my library, saying it crashes if you feed it a large, untrusted file.
I decide to put out a new version of the library, fixing the CVE by refusing to load conspicuously large files. The API otherwise remains unchanged.
Is the new release a major, minor, or bugfix release? As I have only an approximate understanding of semantic versioning norms, I could go for any of them to be honest.
Some other library authors are just as confused as me, which is why major.minor.patchlevel is only a hint.
The client who didn't notice a difference would probably call it a bugfix.
The client whose software got ever-so-slightly more reliable probably would call it a minor update.
The client whose software previously was loading large files (luckily) without issue would call it major, because now their software just doesn't work anymore.
You can Google "YAMLException: The incoming YAML document exceeds the limit" - an error introduced in response to CVE-2022-38752 - to see what happens when a library introduces a new input size limit.
What happened in that case is: the updated library bumps their version from 1.31 to 1.32; then a downstream application updates their dependencies, passes all tests, and updates their version from 9.3.8.0 to 9.3.9.0
In binary package managers this kind of workflow seems like an afterthought.
Maven/Java does absolutely insane things, it will just compile and run programs with incompatible version dependencies and then they crash at some point, and pick some arbitrary first version of a dependency it sees. Then you start shading JARs and writing regex rules to change import paths in dependencies and your program crashes with a mysterious error with 1 google result and you spend 8 hours figuring out WTF happened and doing weird surgery on your dependencies dependencies in an XML file with terrible plugins.
This proposed solution is "let's just never use version ranges and hard-code dependency versions". Now a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?
It doesn't even understand lockfiles fully. They don't make your build non-reproducible, they give you both reproducible builds (by not updating the lockfile) and an easy way to update dependencies if and when you want to. They were made for the express purpose of making your build reproducible.
I wish there was a mega article explaining all the concerns, tradeoffs and approaches to dependency management - there are a lot of them.
Just because Java does this doesn't mean every language has to. It's not strongly tied to the dependency management system used. You could have this even with a Java project using lockfiles.
> a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?
Alternatively, just specify the required version in the top-level project's dependency set, as suggested in the article.
2) "Now a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?"
You can't solve both of these simultaneously.
If you want a library's dependences to be updated to versions other than the original library author wanted to use (e.g. because that library is unmaintained) then you're going to get those incompatibilities and crashes.
I think it's reasonable to be able to override dependencies (e.g. if something is unmaintained) but you have to accept there are going to be surprises and be prepared to solve them, which might be a bit painful, but necessary.
But I realized something by attempting to read this article several times first.
If I ever want to write an article and reduce peoples ability to critically engage with the argument in it I should add a focus pulling animation that thwarts concerted focus.
It's like the blog equivalent of public speakers who ramble their audience into a coma.
You should check how comments work on niconico.
Streaming comments on YouTube give it a run for its money, what absolute garbage.
Inverted colours would've been _mostly fine._ Not great, but mostly fine, but instead, the author went out of their way to add this flashlight thing that's borderline unusable?
What the hell is this website?
If you want to do something cute and fun, whatever its your site. But if you actually want people to use your site make it easy to dismiss. We already have annoying ads and this is honestly worse than many ads.
Also, from the bio that I can barely see he writes about "UI Design" and... included this?
As an aside, I have an article in my blog that has GIFs in it, and they're important for the content, but I'm not a frontend developer by any stretch of the imagination so I'm really at wit's end for how to make it so that the GIFs only play on mouse hover or something else. If anybody reading has some tips, I'd love to hear them. I'm using Zola static site generator, and all I've done is make minor HTML and CSS tweaks, so I really have no idea what I'm doing where it concerns frontend presentation.
Obviously, the server gets your IP when you connect but ideally it doesn't share that with all visitors. This isn't as bad as that, still concerning.
I mean, just the fact that the background is yellow is a terrible UX decision. Not to mention that ridiculous "dark mode". No, it's not funny. It's stupid and distracting.
[1] https://tonsky.me/blog/unicode/
tonsky.me##.container
If every dependency was a `=` and cargo allowed multiple versions of SemVer compatible packages.
The first impact will be that your build will fail. Say you are using `regex` and you are interacting with two libraries that take a `regex::Regex`. All of the versions need to align to pass `Regex` between yourself and your dependencies.
The second impact will be that your builds will be slow. People are already annoyed when there are multiple SemVer incompatible versions of their dependencies in their dependency tree, now it can happen to any of your dependencies and you are working across your dependency tree to get everything aligned.
The third impact is if you, as the application developer, need a security fix in a transitive dependency. You now need to work through the entire bubble up process before it becomes available to you.
Ultimately, lockfiles are about giving the top-level application control over their dependency tree balanced with build times and cross-package interoperability. Similarly, SemVer is a tool any library with transitive dependencies [0]
[0] https://matklad.github.io/2024/11/23/semver-is-not-about-you...
* Absence of lockfiles
* Absence of the central registry
* Cryptographically checksummed dependency trees
* Semver-style unification of compatible dependencies
* Ability for the root package to override transitive dependencies
At the cost of
* minver-ish resolution semantics
* deeper critical path in terms of HTTP requests for resolving dependencies
The trick is that, rather than using crates.io as the universe of package versions to resolve against, you look only at the subset of package versions reachable from the root package. See https://matklad.github.io/2024/12/24/minimal-version-selecti...
No, they don't. As the article explains, the resolution process will pick the version that is 'closest to the root' of the project.
> The second impact will be that your builds will be slow....you are working across your dependency tree to get everything aligned.
As mentioned earlier, no you're not. So there's nothing to support the claim that builds will be slower.
> You now need to work through the entire bubble up process before it becomes available to you.
No you don't, because as mentioned earlier, the version that is 'closest to root' will be picked. So you just specify the security fixed version as a direct dependency and you get it immediately.
Some call transforming .java to .class a transpilation, but then a lot of what we call compilation should also be called transpilation.
Well, Java can ALSO be AOT compiled to machine code, more popular nowadays (e.g. GraalVM).
I believe Zig is also considering adopting it.
If there are any dependencies with the same major version the algorithm simply picks the newest one of them all (but not the newest in the package registry), so you don't need a lockfile to track version decisions.
Go's go.sum contains checksums to validate content, but is not required for version selection decisions.
In go 1.17 they were added so that project loading did not require downloading the go.mod of every dependency in the graph.
Go's system may be worth emulating in future designs. It's not perfect (still requires some centralized elements, module identities for versions ≥2 are confusing, etc.) but it does present a way to both not depend strongly on specific centralized authorities without also making any random VCS server on the Internet a potential SPoF for compiling software. On the other hand, it only really works well for module systems that purely deal with source code and not binary artifacts, and it also is going to be the least hazardous when fetching and compiling modules is defined to not allow arbitrary code execution. Those constraints together make this system pretty much uniquely suited to Go for now, which is a bit of a shame, because it has some cool knock-on effects.
(Regarding deterministic MVS resolution: imagine a@1.0 depending on b@1.0, and c@1.0 depending on a@1.1. What if a@1.1 no longer depends on b? You can construct trickier versions of this possibly using loops, but the basic idea is that it might be tricky to give a stable resolution to version constraints when the set of constraints that are applied depends on the set of constraints that are applied. There are possible deterministic ways to resolve this of course, it's just that a lot of these edge cases are pretty hard to reason about and I think Go MVS had a lot of bugs early on.)
The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.
And how will this look like, if your app doesn't have library C mentioned in its dependencies, only libraries A and B? You are prohibited from answering "well, just specify all the transitive dependencies manually" because it's precisely what a lockfile is/does.
- Nearest Definition Wins: When multiple versions of the same dependency appear in the dependency tree, the version closest to your project in the tree will be used.
- First Declaration Wins: If two versions of the same dependency are at the same depth in the tree, the first one declared in the POM will be used.
It's not always the correct solution, but sometimes it is. If I have a dependency that uses libUtil 2.0 and another that uses libUtil 3.0 but neither exposes types from libUtil externally, or I don't use functions that expose libUtil types, I shouldn't have to care about the conflict.
Good luck finding a project of any complexity that manages to adhere to that kind of design sensibility religiously.
(I think the only language I've ever used that provided top-level support for recognizing that complexity was SML/NJ, and it's been so long that I don't remember exactly how it was done... Modules could take parameters so at the top level you could pass to each module what submodule it would be using, and only then could the module emit types originating from the submodule because the passing-in "app code" had visibility on the submodule to comprehend those types. It was... Exactly as un-ergonomic as you think. A real nightmare. "Turn your brain around backwards" kind of software architecting.)
What you're describing with SML functors is essentially dependency injection I think; it's a good thing to have in the toolbox but not a universal solution either. (I do like functors for dependency injection, much more than the inscrutable goo it tends to be in OOP languages anyways)
In theory... None of us should be doing it. Emitting raw underlying structures from a dependency coupled with ranged versioning means part of your API is under-specified; "And this function returns an argument, the type of which is whatever this third-party that we don't directly communicate with says the type is." That's hard to code against in the general case (but it works out often enough in the specific case that I think it's safe to do 95-ish percent of the time).
Alternatively, it's a pointer to an opaque data structure. But then that fact (that it's a pointer) is frozen.
Either way, you can rely on dependencies not just pulling the rug from under you.
(I remember, ages ago, trying to wrap my head around Component Object Model. It took me awhile to grasp it in the abstract because, I finally realized, it was trying to solve a problem I'd never needed to solve before: ABI compatibility across closed-source binaries with different compilation architectures).
The package file (whatever your system) is communication to other humans about what you know about the versions you need.
The lockfile is the communication to other computers about the versions you are using.
What you shouldn't have needed is fully defined versions in your package files (but you do need it, in case some package or another doesn't do a good enough job following semver)
So, this:
(Related: npm/yarn should use a JSON variant (or YAML, regular or simplified) that allows for comments for precisely this reason)People who use a library might use newer versions (via diamond dependencies or because they use latest), but it will result in a combination of dependencies that wasn't tested by the library's authors. Often that's okay because libraries try to maintain backward compatibility.
Old libraries that haven't had a new release in a while are going to specify older dependencies and you just have to deal with that. The authors aren't expected to guess which future versions will work. They don't know about security bugs or broken versions of dependencies that haven't been released yet. There are other mechanisms for communicating about that.
>How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.
Yes, this is a social contract. Not everything in the universe can be locked into code, and with Semantic versioning, we hope that our fellow humans won't unnecessarily break packages in non-major releases. It happens, and people usually apologize and fix, but it's rare.
This has worked successfully if you look at RubyGems which is 6 years older than npm (although Gemfile.lock was introduced in 2010, npm didn't introduce it until 2017).
RubyGems doesn't have the same reputation for dysfunction as Node does. Neither does Rust, Go, PHP, and Haskell. Even more that I probably don't use a daily basis. Node is the only language that I will come back and find a docker container that straight up won't build or a package that requires the entire dependency tree to update because one package pushed a minor-version change that ended up requiring a minor version change to Node, then that new version of Node isn't compatible with some hack that another package did in it's C extension.
In fact, I expect some Node developer to read this article and deploy yet another tool that will break _everything_ in the build process. In other languages I don't even think I've ever really thought about dependency resolution in years.
But it is still useful. It's like a bloom filter. Most of the time you can happily pull minor or patch upgrades with no issue. Occasionally it will break. But that's less work than analysing every upgrade.
This article appears to be talking about lockfiles for libraries - and I agree, for libraries you shouldn't be locking exact versions because it will inevitably pay havoc with other dependencies.
Or maybe I'm missing something about the JavaScript ecosystem here? I mainly understand Python.
A -> L1 -> L2
They are saying that A should not need a lockfile because it should specify a single version of L1 in its dependencies (i.e. using an == version check in Python), which in turn should specify a single version of L2 (again with an == version check).
Obviously if everybody did this, then we wouldn't need lockfiles (which is what TFA says). The main downsides (which many comments here point out) are:
1. Transitive dependency conflicts would abound
2. Security updates are no longer in the hands of the app developers (in my above example, the developer of A1 is dependent on the developer of L1 whenever a security bug happens in L2).
3. When you update a direct dependency, your transitive dependencies may all change, making what you that was a small change into a big change.
(FWIW, I put these in order of importance to me; I find #3 to be a nothingburger, since I've hardly ever updated a direct dependency without it increasing the minimum dependency of at least one of its dependencies).
They would be resolved by just picking the version 'closest to root', as explained in the article.
> Security updates are no longer in the hands of the app developers
It is, the app developers can just put in a direct dependency on the fixed version of L2. As mentioned earlier, this is the version that will be resolved for the project.
> When you update a direct dependency, your transitive dependencies may all change, making what you that was a small change into a big change.
This is the same even if you use a lockfile system. When you update dependencies you are explicitly updating the lockfile as well, so a bunch of transitive dependencies can change.
Which is going to lead to horrible issues when that library isn't compatible with all your other dependencies. What if your app directly depends on both L1 and L2, but L1 is compatible with L3 1.2 ... 1.5 while L2 is compatible with L3 1.4 ... 1.7? A general "stick to latest" policy would have L1: "L3==1.5", L2: "L3==1.7" (which breaks L1 if L2 wins). A general "stick to oldest compatible" policy would have L1: "L3==1.2", L2: "L3==1.4" (which breaks L2 if L1 wins).
The obvious solution would be to use L3 1.4 ... 1.5 - but that will never happen without the app developer manually inspecting the transitive dependencies and hardcoding the solution - in essence reinventing the lock file.
> It is, the app developers can just put in a direct dependency on the fixed version of L2. As mentioned earlier, this is the version that will be resolved for the project.
And how is that going to work out in practice? Is that direct dependency supposed to sit in your root-level spec file forever? Will there be a special section for all the "I don't really care about this, but we need to manually override it for now" dependencies? Are you going to have to manually specify and bump it until the end of time because you are at risk of your tooling pulling in the vulnerable version? Is there going to be tooling which automatically inspects your dependencies and tells you when it is safe to drop?
> This is the same even if you use a lockfile system. When you update dependencies you are explicitly updating the lockfile as well, so a bunch of transitive dependencies can change.
The difference is that in the lockfile world any changes to transitive dependencies are well-reasoned. If every package specifies a compatibility range for its dependencies, the dependency management system can be reasonably sure that any successful resolution will not lead to issues and that you are getting the newest package versions possible.
With a "closest-to-root" approach, all bets are off. A seemingly-trivial change in your direct dependencies can lead to a transitive dependency completely breaking your entire application, or to a horribly outdated library getting pulled in. Moreover, you might not even be aware that this is happening. After all, if you were keeping track of the specific versions of every single transitive dependency, you'd essentially be storing a lockfile - and that's what you were trying to avoid...
Or maybe I misread the article and it did not say that.
[edit]
The author confirmed that they are assuming Maven's rules and added it to the bottom of their post.
I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)
You are wrong; Maven just picks one of lib-x:0.1.4 or lib-x:0.1.5 depending on the ordering of the dependency tree.
Java dependency management is unhinged, antiquated garbage to anyone who has used any other ecosystem.
Maven and Gradle make up the vast majority of all Java projects in the wild today. So, effectively, Maven is Java in terms of dependency management.
Python says "Yes." Every environment manager I've seen, if your version ranges don't overlap for all your dependencies, will end up failing to populate the environment. Known issue; some people's big Python apps just break sometimes and then three or four open source projects have to talk to each other to un-fsck the world.
npm says "No" but in a hilarious way: if lib-a emits objects from lib-x, and lib-b emits objects from lib-x, you'll end up with objects that all your debugging tools will tell you should be the same type, and TypeScript will statically tell you are the same type, but don't `instanceof` the way you'd expect two objects that are the same type should. Conclusion: `instanceof` is sus in a large program; embrace the duck typing (and accept that maybe your a-originated lib-x objects can't be passed to b-functions without explosions because I bet b didn't embrace the duck-typing).
A: You're handling problem X and then unrelated problem Y suddenly arises because you're not locking package versions thoroughly. It's not fun.
B: Now the opposite. You lock all versions of the libs you use. You use renovate or schedule time for updates periodically. You have a thorough test suite that you can automatically exercise when trying the new updates. You can apply the updates and deoy the new version to a test environment to run a final test manually. Things look good. You deploy to production and, quite often, things go smoothly.
A is the blue pill, easy to taste but things are out of your control and will bite you eventually. B is the red pill: you're in control, for the better or worst.
You actually kind-of answer your own question with this bit. Semver not being a guarantee of anything is true, but I'd extend this (and hopefully it's not a stretch): package authors will republish packages with the same version number, but different package contents or dependency specs. Especially newer authors, or authors new to a language or packaging system, or with packages that are very early in their lifecycle.
There are also cases where packages get yanked! While this isn't a universally-available behaviour, many packaging systems acknolwedge that software will ship with unintentional vulnerabilities or serious stability/correctness issues, and give authors the ability to say, "I absolutely have to make sure that nobody can install this specific version again because it could cause problems." In those cases, having flexible subdependency version constraints helps.
It might be helpful to think by analogy here. If a structure is _completely rigid,_ it does have some desirable properties, not the least of which being that you don't have to account for the cascading effects of beams compressing and extending, elements of the structure coming under changing loads, and you can forget about accounting for thermal expansion or contraction and other external factors. Which is great, in a vacuum, but structures exist in environments, and they're subject to wear from usage, heat, cold, rain, and (especially for taller structures), high winds. Incorporating a planned amount of mechanical compliance ends up being the easier way to deal with this, and forces the engineers behind it to account for failure modes that'll arise over its lifetime.
And yeah, I did that right away. Fun for a moment but extremely distracting.
Scala uses Maven repositories (where the common practice is to use fixed dependency versions) but with different resolution rules:
* When there are conflicting transitive versions, the highest number prevails (not the closest to the root).
* Artifacts declare the versioning scheme they use (SemVer is common, but there are others)
* When resolving a conflict, the resolution checks whether the chosen version is compatible with the evicted version according to the declared version scheme. If incompatible, an error is reported.
* You can manually override a transitive resolution and bypass the error if you need to.
The above has all the advantages of all the approaches advocated for here:
* Deterministic, time-independent resolution.
* No need for lock files.
* No silent eviction of a version in favor of an incompatible one.
* For compatible evictions, everything works out of the box.
* Security update in a transitive dependency? No problem, declare a dependency on the new version. (We have bots that even automatically send PRs for this.)
* Conflicting dependencies, but you know what you're doing? No problem, force an override.
The algorithm can be deterministic, but fetching the dependencies of a package is not.
It is usually an HTTP call to some endpoint that might flake out or change its mind.
Lock files were invented to make it either deterministic or fail.
Even with Maven, deterministic builds (such as with Bazel) lock the hashes down.
This article is mistaken.
npm used to allow you to unpublish (and may be overwrite?) published artifacts, but they removed that feature after a few notable events.
Edit: I was not quite correct. It looks like you can still unpublish, but with specific criteria. However, you cannot ever publish a different package using the same version as an already published package.
https://docs.npmjs.com/cli/v8/commands/npm-publish?v=true
https://docs.npmjs.com/policies/unpublish
Nope, Maven will grab anything which happens to have a particular filename from `~/.m2`, or failing that it will accept whatever a HTTP server gives it for a particular URL. It can compare downloaded artifacts against a hash; but that's misleading, since those hashes are provided by the same HTTP server as the artifact! (Useful for detecting a corrupt download; useless for knowing anything about the artifact or its provenance, etc.)
This isn't an academic/theoretical issue; I've run into it myself https://discuss.gradle.org/t/plugins-gradle-org-serving-inco...
However, I think most people in the reproducible build space would consider Maven an external uncontrolled input.
Generally our practice is to pin everything to major versions, in ruby-speak this means like `gem 'net-sftp', '~> 4.0'` which allows 4.0.0 up to 4.9999.9999 but not 5. Exceptions for non-semver such as `pg` and `rails` which we just pin to exact versions and monitor manually. This little file contains our intentions of which gems to update automatically and for any exceptions, why not.
Then we encourage aggressive performances of `bundle update` which pulls in tons of little security patches and minor bugfixes frequently, but intentionally.
Without the lockfile though, you would not be able to do our approach. Every bundle install would be a bundle update, so any random build might upgrade a gem without anyone even meaning to or realizing it, so, your builds are no longer reproducible.
So we'd fix reproducibility by reverting to pinning everything to X.Y.Z, specifically to make the build deterministic, and then count on someone to go in and update every gem's approved version numbers manually on a weekly or monthly basis. (yeah right, definitely will happen).
However for NPM you will hit issues where 2 packages need a different React version and if you want to use them both you need to pick. In addition it is better for security. The lock file is a distributed checksum. Not impervious to supply chain attacks but better equipped than trusting the package author not to retrospectively bump (I guess you could have a hash for this tbat included the downloaded source code and claimed deps).
Using hashes also makes it easier to distribute, fetch, proxy, etc. since there's no need for trust. In contrast, fetching code based only on (name and) version number requires more centralised repositories with a bunch of security hoops to jump through.
Also, on that note, I can plug my own post on the topic: http://www.chriswarbo.net/blog/2024-05-17-lock_files_conside...
If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.
No they are not. Fully reproducible builds have existed without lockfiles for decades
https://src.fedoraproject.org/rpms/conky/blob/rawhide/f/sour...
also of flathub
https://github.com/flathub/com.belmoussaoui.ashpd.demo/blob/...
"they are not lockfiles!" is a debatable separate topic, but for a wider disconnected ecosystem of sources, you can't really rely on versions being useful for reproducibility
exactly the same thing as a lockfile
It's also not about fully reproducible builds, it's about a tradeoff to get modern package manger (npm, cargo, ...) experience and also somewhat reproducible builds.
Lol, the word "modern" has truly lost all meaning. Your list of "modern package managers" seems to coincide with a list of legacy tooling I wrote four years ago! https://news.ycombinator.com/item?id=29459209
show me one "decades old build" of a major project that isn't based on 1) git hashes 2) fixed semver URLs or 3) exact semver in general.
For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).