Sorry for not having much to add but two interesting references for people to check out, if curious enough:
a) Reminds me of Ginger Bill (the Odin language creator, no affiliation) who on a podcast cited the avoidance of dependency hell as one of the main reasons for Odin never ever having an official package manager, see https://www.youtube.com/watch?v=fYUruq352yE&t=11m26s (timestamped to the correct position)
b) another programmer rather seriously worried about the software quality is Jonathan Blow, who's talk "Preventing the Collapse of Civilization" is worth watching in my opinion: https://www.youtube.com/watch?v=ZSRHeXYDLko
kion · 2h ago
IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos.
This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painstakingly included parts of into your size constrained environment (fit it on a floppy). You probably picked apart that library and pulled the bits you needed, integrating them into your builds to be as small as possible.
Today we pile libraries on top of libraries on top of libraries. Its super easy to say `import foolib`, then call `foolib.do_thing()` and just start running. Who knows or cares what all 'foolib' contains.
At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.
In some cases the languages make this worse. Go and Rust, for example, encourage everything for a single package/mod to go in the same file. Adding optional functionality can get ugly when it would require creating new modules, but if you only want to use a tiny part of the module, what do you do?
The only real solution I can think of to deal with this long term is ultra-fine-grained symbols and dependencies. Every function, type, and other top-level language construct needs to declare the set of things it needs to run (other functions, symbols, types, etc). When you depend on that one symbol it can construct, on demand, the exact graph of symbols it needs and dump the rest for any given library. You end up with the minimal set of code for the functionality you need.
Its a terrible idea and I'd hate it, but how else do you address the current setup of effectively building the whole universe of code branching from your dependencies and then dragging it around like a boat anchor of dead code.
SamuelAdams · 1h ago
This idea is already implemented in Dotnet, with Trimming and now ahead of time compilation (AOT). Maybe other languages can learn from dotnet?
which get reinvented all the time, like in dotnet with "trimming" or in JS with "tree-shaking".
C/C++ compiler have been doing that since before dot net was a thing, same for rust which does that since it's 1.0 release (because it's done by LLVM ;) )
The reason it gets reinvented all the time is because while it's often quite straight forward in statically compiled languages it isn't for dynamic languages as finding out what actually is unused is hard (for fine grained code elimination) or at lest unreliable (pruning submodules). Even worse for scripting languages.
Which also brings use to one area where it's not out of the box, if you build .dll/.so in one build process and then use them in another. Here additional tooling is needed to prune the dynamic linked libraries. But luckily it's not a common problem to run into in Rust.
In general most code size problems in Rust aren't caused by too huge LOC of dependencies but by an overuse of monopolization. The problem of tons of LOC in dependencies is one of supply chain trust and review ability more then anything else.
CBLT · 1h ago
Those are done at compile time. Many languages (including Rust, which this story is about) also remove unused symbols at compile time.
The comment you're replying to is talking about not pulling in dependencies at all, before compiling, if they would not be needed.
nicoburns · 2h ago
As far as I'm aware, LTO completely solves this from a binary size perspective. It will optimise out anything unused. You can still get hit from a build time perspective though.
poincaredisk · 1h ago
"completely solves" is a bit of an overstatement. Imagine a curl-like library that allows you to make requests by URL. You may only ever use HTTP urls, but code for all the other schemas (like HTTPS, FTP, Gopher) needs to be compiled in as well.
This is an extreme example, but the same thing happens very often at a smaller scale. Optional functionality can't always be removed statically.
vvanders · 34m ago
That only applies when dynamic dispatch is involved and the linker can't trace the calls. For direct calls and generics(which idiomatic Rust code tends to prefer over dyn traits) LTO will prune extensively.
rafram · 23m ago
let uri = get_uri_from_stdin();
networking_library::make_request(uri);
How is the compiler supposed to prune that?
nicoburns · 40m ago
I guess that depends on the implementation. If you're calling through an API that dynamically selects the protocol than I guess it wouldn't be removable.
Rust does have a feature flagging system for this kind of optional functionality though. It's not perfect, but it would work very well for something like curl protocol backends though.
api · 1h ago
That's a consequence of crufty complicated protocols and standards that require a ton of support for different transports and backward compatibility. It's hard to avoid if you want to interoperate with the whole world.
kccqzy · 9m ago
Everywhere in this thread is debating whether LTO "completely" solves this or not, but why does this even need LTO in the first place? Dead code elimination across translation units in C++ is traditionally accomplished by something like -ffunction-sections, as well as judiciously moving function implementations to the header file (inline).
dathinab · 1h ago
yes, it's not a issue of code size but a issue of supply chain security/reviewability
it's also not always a fair comparison, if you include tokio in LOC counting then you surely would also include V8 LOC when counting for node, or JRE for Java projects (but not JDK) etc.
samus · 2h ago
It's certainly better than in Java where LTO is simply not possible due to reflection. The more interesting question is which code effectively gets compiled so you know what has to be audited. That is, without disassembling the binary. Maybe debug information can help?
pjmlp · 1h ago
Not only it is possible, it has been available for decades on commercial AOT compilers like Aonix, Excelsior JET, PTC, Aicas.
It is also done on the cousin Android, and available as free beer on GraalVM and OpenJ9.
0x696C6961 · 1h ago
In Go, the symbol table contains enough information to figure this out. This is how https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck is able to limit vulnerabilities to those that are actually reachable in your code.
metaltyphoon · 1h ago
Doesn’t Java offer some sort of trimming like C#? I know he won’t remove everything but at least they can trim down a lot of things.
pjmlp · 1h ago
Yes, jlink, code guard, R8/D8 on Android, if you want to stay at the bytecode level, plus all the commercial AOT compilers and the free beer ones, offer similar capabilities at the binary level.
kion · 2h ago
LTO only gets you so far, but IMO its more kicking the can down the road.
The analogy I use is cooking a huge dinner, then throwing out everything but the one side dish you wanted. If you want just the side-dish you should be able to cook just the side-dish.
floating-io · 1h ago
I see it more as having a sizable array of ingredients in the pantry, and using only what you need or want for a given meal.
01HNNWZ0MV43FF · 2h ago
Then another group of armchair programmers will bitch you out for using small dependencies
I just don't listen. Things should be easy. Rust is easy. Don't overthink it
dietr1ch · 2h ago
I don't think libraries are the problem, but we don't have a lot of visibility after we add a new dependency. You either take the time to look into it, or just add it and then forget about the problem (which is kind of the point of having small libraries).
It should be easy to build and deploy profiling-aware builds (PGO/BOLT) and to get good feedback around time/instructions spent per package, as well as a measure of the ratio of each library that's cold or thrown away at build time.
taeric · 1h ago
I agree that I don't like thinking of libraries as the problem. But they do seem to be the easiest area to point at for a lot of modern development hell. Is kind of crazy.
I'll note that it isn't just PGO/BOLT style optimizations. Largely, it is not that at all, oddly.
Instead, the problem is one of stability. In a "foundation that doesn't move and cause you to fall over" sense of the word. Consider if people made a house where every room had a different substructure under it. That, largely, seems to be the general approach we use to building software. The idea being that you can namespace a room away from other rooms and not have any care on what happens there.
This gets equally frustrating when our metrics for determining the safety of something largely discourages inaction on any dependencies. They have to add to it, or people think it is abandoned and not usable.
Note that this isn't unique to software, mind. Hardware can and does go through massive changes over the years. They have obvious limitations that slow down how rapidly they can change, of course.
xlii · 2h ago
> Go and Rust, for example, encourage everything for a single package/mod to go in the same file.
Clarification: Go allows for a very simple multi-file. It’s one feature I really like, because it allows splitting otherwise coherent module into logical parts.
dcow · 2h ago
Further: I’ve never seen rust encourage anything of the sort. Module directory with a mod.rs and any number of files works just fine.
kion · 2h ago
I probably mischaracterized this as its been a while since I did more than trivial Rust. AFAIK its not possible to depend on only a part of a module in Rust though right? (At least without an external build system)
For example, you can't split up a module into foo.rs containing `Foo` and bar.rs containing `Bar`, both in module 'mymod' in such a way that you can `use mymod::Bar and foo.rs is never built/linked.
My point is the granularity of the package/mod encourages course-grained deps, which I argue is a problem.
dathinab · 1h ago
> not possible to depend on only a part of a module in Rust though right
yesn't, you can use feature flags similar to `#if` in C
but it's also not really a needed feature as dead code elimination will prune out all code functions, types, etc. you don't use. Non of it will end up in the produced binary.
eddd-ddde · 1h ago
You'd use feature flags to enable certain parts of the library.
tialaramex · 1h ago
Yeah, likewise Rust is completely fine after you say `mod foo` and have a file named foo.rs, if you also make a foo/ directory and put foo/whatever.rs and foo/something_else.rs that those are all part of the foo module.
Historically Rust wanted that foo.rs to be renamed foo/mod.rs but that's no longer idiomatic although of course it still works if you do that.
dathinab · 1h ago
to extend on this:
in rust crates are semantically one compilation unit (where in C oversimplified it's a .h/.c pair, and practically rustc will try to split it in some more units to speed up build time).
the reason I'm pointing this out is because many sources of "splitting a module across files" come from situations where 1 file is one compilation unit so you needed to have a way to split it (for organization) without splitting it (for compilation) in some sitation
ruraljuror · 48m ago
The actual behavior of go seems much closer to your ideal scenario than what you attribute to it. Although it is more nuanced, so both are true. In go, a module is a collection of packages. When you go get a module, the entire module is pulled onto the host, but when you vendor only the packages you use (and i believe only the symbols used from that package, but am not certain) are vendored to your module as dependencies.
kibwen · 2h ago
> In some cases the languages make this worse. Go and Rust, for example, encourage everything for a single package/mod to go in the same file.
What? I don't know about Go, but this certainly isn't true in Rust. Rust has great support for fine-grained imports via Cargo's ability to split up an API via crate features.
KennyBlanken · 50m ago
What did you expect from someone who thinks that merely including a bunch of library header files but only calling one small function results in hundreds of megabytes of compiled binary code for functions that are never used, and all the functions those functions use?
KennyBlanken · 55m ago
I can't remember the last time I saw someone so conclusively demonstrate they know nothing about the basics of how libraries, compilers, and linkers work.
throwaway462663 · 2h ago
> It's a terrible idea...
It's a terrible idea because you're trying to reinvent section splitting + `--gc-sections` at link time, which rust (which the article is about) already does by default.
kion · 2h ago
The article is about Rust, but I was commenting on dependencies in general.
Things like --gc-sections feels like a band-aid, a very practical and useful band-aid, but a band-aid none the less. You're building a bunch of things you don't need, then selectively throwing away parts (or selectively keeping parts).
IMO it all boils down to the granularity. The granularity of text source files, the granularity of units of distribution for libraries. It all contributes to a problem of large unwieldy dependency growth.
I don't have any great solutions here, its just observations of the general problem from the horrifying things that happen when dependencies grow uncontrolled.
jiggawatts · 1h ago
A consideration that is often overlooked is that the waste accumulates exponentially!
If each layer of “package abstraction” is only 50% utilised, then each layer multiplies the total size by 2x over what is actually required by the end application.
Three layers — packages pulling in packages that pull their own dependencies — already gets you to 88% bloat! (Or just 12% useful code)
An example of this is the new Windows 11 calculator that can take several seconds to start because it loads junk like the Windows 10 Hello for Business account recovery helper library!
Why? Because it has currency conversion, which uses a HTTP library, which has corporate web proxy support, which needs authentication, which needs WH4B account support, which can get locked out, which needs a recovery helper UI…
…in a calculator. That you can’t launch unless you have already logged in successfully and is definitely not the “right place” for account recovery workflows to be kicked off.
But… you see… it’s just easier to package up these things and include them with a single line in the code somewhere.
aeonik · 1h ago
if only we had a system that we could all operate on with a standard set of tools that would take care of shared resource access like this.
zozbot234 · 1h ago
> In the 80s the idea of a library of functionality was something you paid for, and painstakingly included parts of into your size constrained environment (fit it on a floppy). You probably picked apart that library and pulled the bits you needed, integrating them into your builds to be as small as possible.
If anything, the 1980s is when the idea of fully reusable, separately-developed software components first became practical, with Objective-C and the like. In fact it's a significant success story of Rust that this sort of pervasive software componentry has now been widely adopted as part of a systems programming language.
the__alchemist · 20m ago
To address a point near the end of the article, here is my [partial] solution that works as a baseline.
Curate a collection of libraries you use and trust. This will probably involve making a number of your own. Wheel-reinvention, if you will. If done properly, even the upfront time cost will save in the long-run. I am in the minority here, but I roll my own libs whenever possible, and the 3rd party libs I use are often ones I know, have used been for, and vetted that they have a shallow tree of their own.
Is this sustainable? I don't know. But It's the best I've come up with, in order to use what I see as the best programming language available for several domains.
There are a lot of light-weight, excellent libs I will use without hesitation, and have wide suitability. Examples:
Heavier, and periodically experience mutual-version hell, but are are very useful for GUI programs:
- EGUI
- WGPU
- Winit
On a darker note, the rust web ecosystem maybe permanently lost to async and messy dependencies. Embedded is going that way too, but I have more hope there, and am doing my best to have my own tooling.
zaptheimpaler · 2h ago
This is just a modern problem in all software development, regardless of language. We are doing more complex things, we have a much bigger library of existing code to draw from and there are many reasons to use it. Ultimately a dependency is untrusted code, and there's a long road to go in hardening entire systems to make running arbitrary dependencies safe (if its even possible).
In the absence of a technical solution, all others basically involve someone else having to audit and constantly maintain all that code and social/legal systems of trust. If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult.
kdps · 1h ago
I'd argue that the severity varies between languages, despite the core problem being universal. Languages with comprehensive standard libraries have an advantage over those with minimal built-in functionality, where people rely on external dependencies even for the most basic things (e.g. see Java/.NET vs JS/Node). Lightweight is not always better.
RossBencina · 37m ago
> Languages with comprehensive standard libraries have an advantage
I don't see the advantage. Just a different axis of disadvantage. Take python for example. It has a crazy big standard library full of stuff I will never use. Some people want C++ to go in that direction too -- even though developers are fully capable of rolling their own. Similar problem with kitchen-sink libraries like Qt. "batteries included" languages lead to higher maintenance burden for the core team, and hence various costs that all users pay: dollars, slow evolution, design overhead, use of lowest common denominator non-specialised implementations, loss of core mission focus, etc.
zeroxfe · 1h ago
> If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult.
I think Rust really needs to do more of this. I work with both Go and Rust daily at work, Go has its library game down -- the standard library is fantastic. With Rust it's really painful to find the right library and keep up for a lot of simple things (web, tls, x509, base64 encoding, heck even generating random numbers.)
harha_ · 1h ago
Regardless of language, really? I highly doubt that, you don't generally see such problems with C or even C++ because dependencies are more cumbersome to add, especially in a way that's cross-platform.
tialaramex · 34m ago
With C++ it's hilarious because the C++ community is so allergic to proper dependency management and also so desperate for stuff from third party libraries that the committee spends large amounts of its time basically doing dependency management for the community by baking in large features you'd ordinarily take as a dependency into the mandatory standard library.
I'm sure I'll miss some, but IIRC C++ 26 is getting the entire BLAS, two distinct delayed reclamation systems and all of the accompanying infrastructure, new container types, and a very complicated universal system of units.
All of these things are cool, but it's doubtful whether any of them could make sense in a standard library, however for C++ programers that's the easiest way to use them...
It's bedlam in there and of course the same C++ programmers who claim to be "worried" that maybe somebody hid something awful in Rust's crates.io are magically unconcerned that copy-pasting tens of millions of lines of untested code from a third party into absolutely every C++ program to be written in the future could be a bad idea.
zaptheimpaler · 1h ago
Because most dependencies are either manually installed by the user, or are dynamic libraries that are provided and audited by the distro maintainers. The dependencies are there, they're just harder to see - https://wiki.alopex.li/LetsBeRealAboutDependencies
harha_ · 1h ago
Sure, there are various dependencies, but it's nothing like "cargo install crate-name". Cargo makes it so effortless to joink the dumbest dependency for the simplest thing.
crote · 49s ago
On the other hand, C/C++ makes it attractive to reinvent the wheel, or vendor the dependency instead. Rather than a single well-tested implementation in the ecosystem for something like sha256, you end up with every application having its own slightly-different, mostly untested, and essentially unmaintained version.
Applications still need the functionality. The need doesn't magically disappear when installing dependencies is a pain. If a crate has a bug, the entire ecosystem can trivially get the fixed version. If the Stackoverflow snippet a C app is vendoring has a bug, that fix is never getting in the app.
tonyhart7 · 2m ago
so what's the option??? I don't think this is only cargo-rust problem since you can do it too in another language
zaptheimpaler · 1h ago
Sure, I think software that's easy to use is a good thing and Rust dependency management is 100x nicer to work with than C++.
pjmlp · 1h ago
Kind of true, when not using vcpkg/conan.
jongjong · 1h ago
Yes, but a lot of the complexity is unnecessary bloat. Almost every project I've ever seen or worked on was full of unnecessary complexity. People naturally tend to over-complicate things, all the programming books, including software design books focus on unimportant aspects and miss all the important ones. It's incredibly frustrating.
Yet, if someone were to write a book which explained things properly (probably a 3000 word article would suffice to turn anyone into a 10x dev), nobody would buy it. This industry is cooked.
jerf · 9h ago
A true enough statement, but "Rust" is unnecessarily specific. Dependencies are getting scary in general. Supply chain attacks are no longer hypothetical, they're here and have been for a while.
If I were designing a new language I think I'd be very interested in putting some sort of capability system in so I can confine entire library trees safely, and libraries can volunteer somehow what capabilities they need/offer. I think it would need to be a new language if for no other reason than ecosystems will need to be written with the concept in them from the beginning.
For instance, consider an "image loading library". In most modern languages such libraries almost invariably support loading images from a file, directly, for convenience if nothing else. In a language that supported this concept of capabilities it would be necessary to support loading them from a stream, so either the image library would need you to supply it a stream unconditionally, or if the capability support is more rich, you could say "I don't want you to be able to load files" in your manifest or something and the compiler would block the "LoadFromFile(filename)" function at compile time. Multiply that out over an entire ecosystem and I think this would be hard to retrofit. It's hugely backwards incompatible if it is done correctly, it would be a de facto fork of the entire ecosystem.
I honestly don't see any other solution to this in the long term, except to create a world where the vast majority of libraries become untargetable in supply chain attacks because they can't open sockets or read files and are thus useless to attackers, and we can reduce our attack surface to just the libraries that truly need the deep access. And I think if a language came out with this design, you'd be surprised at how few things need the dangerous permissions.
Even a culture of minimizing dependencies is just delaying the inevitable. We've been seeing Go packages getting supply-chain-attacked and it getting into people's real code bases, and that community is about as hostile to large dependency trees as any can be and still function. It's not good enough.
In WUFFS most programs are impossible. Their "Hello, world" doesn't print hello world because it literally can't do that. It doesn't even have a string type, and it has no idea how to do I/O so that's both elements of the task ruled out. It can however, Wrangle Untrusted File Formats Safely which is its sole purpose.
I believe there should be more special purpose languages like this, as opposed to the General Purpose languages most of us learn. If your work needs six, sixteen or sixty WUFFS libraries to load different image formats, that's all fine because categorically they don't do anything outside their box. Yet, they're extremely fast because since they can't do anything bad by definition they don't need those routine "Better not do anything bad" checks you'd write in a language like C or the compiler would add in a language like Rust, and because they vectorize very nicely.
assassinator42 · 2h ago
Java and the .NET Framework had partial trust/capabilities mechanisms decades ago. No one really used them and they were deprecated/removed.
pjmlp · 1h ago
It was more like no one used them correctly.
eikenberry · 1h ago
Wouldn't that mean they were poorly implemented. If no one uses something correctly, seems like that isn't a problem with the people but the thing.
palata · 1h ago
I don't think so. Software is maybe the only "engineering" discipline where it is considered okay to use mainstream tools incorrectly and then blame the tools.
dullcrisp · 3m ago
Do the “mainstream” tools change every five years in other disciplines?
voxgen · 2h ago
I don't think retrofitting existing languages/ecosystems is necessarily a lost cause. Static enforcement requires rewrites, but runtime enforcement gets you most of the benefit at a much lower cost.
As long as all library code is compiled/run from source, a compiler/runtime can replace system calls with wrappers that check caller-specific permissions, and it can refuse to compile or insert runtime panics if the language's escape hatches would be used. It can be as safe as the language is safe, so long as you're ok with panics when the rules are broken.
It'd take some work to document and distribute capability profiles for libraries that don't care to support it, but a similar effort was proven possible with TypeScript.
0cf8612b2e1e · 2h ago
Is there anything in existence which has a version of this idea? It makes a ton of sense to me, but you are right that it would be practically impossible to do in a current language.
martsa1 · 1h ago
Wasm + wasi let you define hard boundaries between components with explicit interfaces, might be loosely along these lines?
kibwen · 1h ago
Yes, but you can't enforce this at the language level if your objective is security (at least not for natively-compiled languages). You need OS-level support for capabilities, which some OSes do provide (SeL4, Fuchsia). But if you're in a VM rather than native code then you can enforce capabilities, which is what Wasm does with WASI.
Austral is a really cool experiment and I love how much effort was put into the spec which you've linked to. It explains the need for capabilities and linear types, and how they interact, really well.
metaltyphoon · 1h ago
.NET Framework, windows only, (non .NET, aka .NET Core)
wofo · 9h ago
I've thought about this (albeit not for that long) and it seems like you'd need a non-trivial revamp of how we communicate with the operating system. For instance, allowing a library to "read from a stream" sounds safe until you realize they might be using the same syscalls as reading from a file!
eikenberry · 1h ago
> I think it would need to be a new language [..]
Languages (plural) ... no single language will work for everyone.
cyberax · 1h ago
TypeScript ecosystem supports this! An environment without e.g. file operations will simply miss classes that are needed for it, and your compilation will fail.
Orangeair · 2h ago
I think that https://blessed.rs does a pretty good job of providing recommendations for things that probably can't be crammed into the standard library, but which you'll almost certainly end up needing at one point or another. I honestly like that system a lot, it makes it so that the only packages you need to worry much about are usually doing something rather specific.
colanderman · 1h ago
Rust at least has a partial remedy to this problem: feature flags. Many libraries use them to gate features which would otherwise pull in extra dependencies. (In fact I believe there is specific support for flags which correspond to dependency names.)
QuadmasterXLII · 1h ago
I wonder how much good a “dependency depth” label on packages would do, at the crates.io level. Like, a package can only depend on a package with a lower declared dependency depth than it, and packages compete to have a low dependency depth as a badge.
palata · 14h ago
Similar feeling here.
Cargo makes it so simple to add tons of dependencies that it is really hard not to do it. But that does not stop here: even if I try to be careful with adding dependencies, a couple dependencies are likely to pull tens of transitive dependencies each.
"Then don't depend on them", you say. Sure, but that means I won't write my project, because I won't write those things from scratch. I could probably audit the dependency (if it wasn't pulling 50 packages itself), but I can't reasonably write it myself.
It is different with C++: I can often find dependencies that don't pull tens of transitive dependencies in C++. Maybe because it's harder to add dependencies, maybe because the ecosystem is more mature, I don't know.
But it feels like the philosophy in Rust is to pull many small packages, so it doesn't seem like it will change. And that's a pity, because I like Rust-the-language better than C++-the-language. It just feels like I trade "it's not memory-safe" for "you have to pull tons of random code from the Internet".
I think it makes a good point that some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded. To some degree that is a plus though as you likely trust the maintainers of your OS distribution to provide stable, supported libraries.
As other commenters have said, perhaps this is an area where the Rust maintainers could provide some kind of extended standard library where they don't guarantee backwards compatibility forever, but do provide guarantees about ongoing fixes for security issues.
palata · 1h ago
> some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded.
Not in my case. I manually compile all the dependencies (either because I need to cross-compile, or because I may need to patch them, etc). So I clearly see all the transitive dependencies I need in C++. And I need a lot less than in Rust, by a long shot.
jampekka · 10h ago
I take bit less unstable dependencies over the total mess of C++ dependencies with CMake, shared libraries, version conflicts etc any time. There's probably also a bit of an illusion about C++ transitive dependencies due to them usually being precompiled (because compiling them is such pain).
ChocolateGod · 9h ago
The whole pkgconfig, cmake, autotools etc ecosystem is insane compared to how Rust and Go do things.
It's part of the reason why software distribution on Linux has been pushed to using containers, removing the point of having shared libraries. I think Google with it's C++ replacement (Carbon) plans on doing it's own system.
palata · 1h ago
> It's part of the reason why software distribution on Linux has been pushed to using containers
My understanding of people distributing their software in containers is that they can't be arsed to learn how to do it properly. They would install their software and ship the entire computer if that was cost effective.
jampekka · 48m ago
What needs to be "learned properly" is sadly a huge pile of incoherent legacy cruft that ideally wouldn't be there at all.
This is not to denigrate the huge and critical effort that makes current computing possible, and that is likely unavoidable in the real world. But software distribution needs to evolve.
skydhash · 2h ago
From my point of view, the issue stems from developers wanting to control distribution. Fine if it's for your own usage, not really if you're planning for others to use it. You will find the most convoluted build system just because they have a pet platform they want to specially support making it hell to do anything on others.
It could be better, but the current solutions (npm, go, python,...) favor only the developers, not the maintainers and packagers.
ChocolateGod · 58m ago
There's examples of maintainers/packagers effectively sabotaging other peoples projects when making packages for distros, whether that's shipping them broken, ancient versions etc.
e.g. Bottles, WebkitGTK (distros liked keeping this one held back even though doing so is a security risk)
IMHO it shouldn't be the responsibility of the OS vendor to package third party applications.
jampekka · 29m ago
Distro maintainers/packagers are who keep the current software stacks running. It's rather amazing how they manage to keep the billion or so lines of separately written code working in unison.
That said, the labor needed to keep the stuff together could be reduced a lot by the more ergonomical and universal packaging and distribution methods like Cargo (and, dare I say, npm). I think some kind of a better bridge between developers and distros could be found here.
antonvs · 9h ago
> Sure, but that means I won't write my project, because I won't write those things from scratch.
You need to think a bit harder about that, to help you decide whether your position is rational.
MeetingsBrowser · 2h ago
This confuses me as well. Is the implied solution to choose a language where you are forced to write those things from scratch?
palata · 56m ago
My point is that if, in the language, everybody is incentivise to use fewer dependencies, then a random library that I would not write myself (because it is an entire project in itself) would have fewer dependencies. Because it is not the case, either I take that library and accept its transitive dependencies, or I don't have a library at all.
In Rust, I'm sometimes actually tempted to wrap a C/C++ library (and its few dependencies) instead of getting the Rust alternative (and its gazillion dependencies).
palata · 55m ago
And you need to think a bit about that (probably not very hard), to help you decide whether I'm irrational or whether you may not have totally understood my point.
perrygeo · 9h ago
> the philosophy in Rust is to pull many small package
I'm not sure it's a philosophy, more a pragmatic consideration for compilation speeds. Anyone who's done a non-trivial amount of Rust knows that moment when the project gets too big and needs to split into separate crates. It's kinda sad that you can't organize code according to proper abstractions, many times I feel forced to refactor for compiler performance.
imtringued · 9h ago
I have been wasting 6 hours yesterday on getting the bullet examples to compile outside of bullet itself with no success. It's more likely that a lot of software simply doesn't get written because C++ and CMake are a pain in the ass.
palata · 52m ago
I find CMake pretty easy, and I only use a few core features from it. Usually the pain comes from completely wrong setups by people who didn't learn the basic. But it's true of everything, I think.
neilv · 9h ago
In the past (not in Rust, but other languages), for important systems, I've instituted policies of minimizing dependencies from these language-specific package repositories, and for the ones you do use, having to copy it to our own repos and audit each update before use.
But that's not practical for all situations. For example, Web frontend developer culture might be the worst environment, to the point you often can't get many things done in feasible time, if you don't adopt the same reckless practices.
I'm also seeing it now with the cargo-culting of opaque self-hosted AI tools and models. For learning and experimenting, I'd spend more time sufficiently compartmentalizing an individual tool than with using it.
This weekend, I'm dusting off my Rust skills, for a small open source employability project (so I can't invest in expensive dependency management on this one). The main thing thing bothering me isn't allocation management, but the sinking feeling when I watch the cast-of-thousands explosion of transitive dependencies for the UI and async libraries that I want to use. It's only a matter of time before one of those is compromised, if not already, and one is all it takes.
pjmlp · 9h ago
Best way is to have CI/CD systems only connected to the official internal repos.
Devs can add whatever they feel like on their workstations but it will be a sad build server if they get pushed without permission.
dsr_ · 9h ago
s/Best way/The only safe way/
Anything else will get abused in the name of expediency and just-this-one-time.
Also, the process for adding a crate/gem/module/library needs to be the same as anything else: license review, code review, subscription to the appropriate mailing list or other announce channel, and assignment of responsibility. All of these except code review can be really, really fast once you have the process going.
All problems are, at least in part, dependency chain management problems.
sunrunner · 2h ago
I agree that some amount of friction when including third party dependencies is a vital thing to push people to consider the value versus cost of dependencies (and license review, code review, channel subscriptions are all incredibily important and almost always overlooked), however how should this work for transitive dependendencies? And the dependencies of _those_ dependencies?
The dependency trees for most interpreted or source-distributed languages are ridiculous, and review of even a few of those seems practically impossible in a lot of development environments.
MeetingsBrowser · 2h ago
> Devs can add whatever they feel like on their workstations
A compromised dev machine is also a problem.
pjmlp · 1h ago
True, hence we can go next level and also deal with limited accounts for developers, and I can tell you most folks on HN would hate to work in such corporate environments.
eddd-ddde · 3h ago
The cool thing about rust is you can implement async yourself. You aren't tied to any specific implementation.
pjmlp · 1h ago
Same in C++, partially true in .NET/C# and F#.
dboreham · 2h ago
Or not use async at all.
wofo · 9h ago
There are some voices trying to address this security risk (e.g. the proponents of this new RFC: https://github.com/rust-lang/rfcs/pull/3810). However, for some reason (probably culture) there isn't much momentum yet to change the status quo.
dathinab · 1h ago
> isn't much momentum yet to change the status quo.
it's complex problem with tons of partial solutions which each have tons of ways to implement them with often their no being a clear winner
i.e. it's the kind of hard to solve by consensus problem
e.g. the idea of a extended standard library is old (around since the beginning of rust) but for years it was believed it's probably the best to make it a separate independent project/library for various reason. One being that the saying "the standard library is the place where code goes to die" has been quite true for multiple ecosystems (most noticeably python)
as a side note ESL wouldn't reduce the LOC count it would increase it as long as you fully measure LOCs and not "skip" over some dependencies
cogman10 · 9h ago
The rust RFC process has, frankly, become somewhat of a CF.
There's literally 1000s of RFCs for rust with only a small handful that are integrated. Having this forest, IMO, makes it hard for any given proposal to really stand out. Further, it makes duplicate effort almost inevitable.
Rust's RFC process is effectively a dead letter box for most.
geodel · 7h ago
I think they can constitute committee for RFC review process(in case there is none today) and based on recommendation multiple domain specific teams/ groups can be created to review RFCs in timely manner.
nemothekid · 9h ago
I feel like leftpad has given package managers a very bad name. I understand the OP's hesitation, but it feels a little ridiculous to me.
tokio is a work-stealing, asynchronous runtime. This is a feature that would be an entire language. Does OP consider it reasonable to audit the entire Go language? or the V8 engine for Node? v8 is ~10x more lines than tokio.
If Cloudflare uses Node, would you expect Cloudflare to audit v8 quarterly?
conradludgate · 2h ago
And for what it's worth, people do audit tokio. I have audited tokio. Many times in fact. Sure, not everyone will, but someone will :)
timewizard · 2h ago
If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default?
This is something I've only ever seen cargo do.
rcxdude · 26m ago
It'll do that if there isn't a single version that meets both requirements. Which is a great thing, because most other languages will just fail the build in that case (well, there are still cases where it won't even work in rust, if types from those sub-dependencies are passed in between the two closer dependencies)
metaltyphoon · 1h ago
> If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default?
No, cargo will resolve using sem ver compatibility and pick the best version. Nuget, for C# does something very similar.
Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight.
I decided that I should leave this project alone and spend my time elsewhere.
constantcrying · 1h ago
I am counting 13 dependencies, the rest are internal ones. Are any of these superfluous or only needed for small edge cases? Serde seems exactly a case where you absolutely should use an external dependency.
Also, repository size seems an extremely irrelevant metric.
rs186 · 1h ago
13 > 12 so over a dozen dependencies. If you look at acorn or babel/parser, they barely have any dependency.
Repository size is directly related to how long it takes to run a build, which is extremely important if I were to contribute to the project.
> Serde seems exactly a case where you absolutely should use an external dependency.
I can't see any reason a parser has a hard dependency on a serialization library.
constantcrying · 1h ago
>13 > 12 so over a dozen dependencies. If you look at acorn or babel/parser, they barely have any dependency.
Which ones are superfluous?
There are good reasons to use dependencies. If someone has solved a problem you need to solve as well it is pointless to duplicate the effort.
>Repository size is directly related to how long it takes to run a build, which is extremely important if I were to contribute to the project.
Totally false. There is zero inherent relation.
>I can't see any reason a parser has a hard dependency on a serialization library.
And because you can't see a reason there is none?
It is totally meaningless to talk about any of this if you can not point out why this is superfluous.
rs186 · 47m ago
I don't think there is any point in debating this, because apparently you are in the camp of "dependencies are ok", with or without a good reason, when a different camp is "avoid dependencies unless you really have to". You just provided an example of why dependencies explode like this.
> And because you can't see a reason there is none?
Somehow every other JS based parser doesn't do fancy serialization, as far as I can tell. You can come up with reasons of why one might need it, but as a user of the parser, I want the footprint to be small, and that's a requirement. In fact, that's one of the reasons I never used swc parser in my serious projects.
MeetingsBrowser · 2h ago
I agree that relying on unknown dependencies is a risk, but this misses the point IMO. Number of dependencies and disk space are kind of arbitrary.
> Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight.
The lightest weight javascript program relies on V8 to run, which has multiple orders of magnitude more dependencies. Most of which you have never heard of.
At least cargo makes it easier to get a clearer picture of what the dependencies are for a program.
pixl97 · 1h ago
Number of dependencies isn't exactly arbitrary...
If you have one huge dep it's easier to keep track you're on the latest update, also it's much less likely you'll fat finger it and import something typosquatting.
Also if you're in enterprise you'll have less 100 page SBOM reports.
MeetingsBrowser · 5m ago
What is more likely to be vulnerable, a 100k LoC project developed by ten people, or ten 10k LoC single maintainer projects.
Keeping track of the latest version is trivial with cargo.
rs186 · 1h ago
No, it has very little to do with v8 or any runtime. Those parsers run on any decent and recent enough runtime, including browsers and Node.js. If you look at the actual code, they use basic APIs in the JavaScript language that you can find in almost any other language.
munificent · 1h ago
> relies on V8 to run, which has multiple orders of magnitude more dependencies.
Actually, this isn't true. (Or at least wasn't a while back.) I used to work with a bunch of ex-V8 folks and they really despised third-party dependencies and didn't trust any code they didn't write. They used a few third-party libs but for them most part, they tried to own everything themselves.
dathinab · 32m ago
they are also Google
.. as in they can afford to rewrite everything
.. can afford to suffer from not invented here syndrome
.. and are under _massive_ threat of people doing supply chain attacks compared to most other projects (as they end up running on nearly any desktop computer and half the phones out there)
this just isn't viable for most projects, not just resource/time investment wise, but also reinventing/writing everything isn't exactly good to reduce bugs if you haven't to reliably access to both resources _and_ expertise. Most companies have to live with having many very average developers, and very tight resource limits.
schmichael · 9h ago
We need a term like “Mature” or similar for dependencies that are done. Mature dependencies have two characteristics:
1. Well defined scope
2. Infrequent changes
Nomad has many of these (msgpack, envparse, cli, etc). These dependencies go years without changing so the dependency management burden rapidly approaches zero. This is an especially useful property for “leaf” dependencies with no dependencies of their own.
I wish libraries could advertise their intent to be Mature. I’d choose a Mature protobuf library over one that constantly tweaked its ergonomics and performance. Continual iterative improvement is often a boon, but sometimes it’s not worth the cost.
procaryote · 2h ago
Java did this sometimes by essentially adding slightly tidied up versions of whatever was the de-facto standard to the standard library. Java 1.3 didn't have regexes but most people were using the same apache commons thing, so java 1.4 added regexes that looked exactly like that. Java's date handling was a pain so people mostly used joda-date; a later java version added something that mostly works like jodadate. Etc.
It is an easy way to get a somewhat OK standard library as the things you add became popular on their own merits at some point.
Once added, the lowest friction path is to just use the standard library; and as it is the standard library you have a slightly better hope someone will care to maintain it. You can still build a better one if needed for your use-case, but the batteries are included for basic usage
delusional · 9h ago
I have a lot of sympathy for this viewpoint, but I also ask that we try to remind ourselves. We are asking for professionalism from hobby projects.
If you want a mature protobuf implementation you should probably buy one. Expecting some guy/gal on the internet to maintain one for your for free seems ill advised.
pclmulqdq · 1h ago
> I have a lot of sympathy for this viewpoint, but I also ask that we try to remind ourselves. We are asking for professionalism from hobby projects.
Nobody is asking for professional quality standards from hobby projects. At best, they are asking for hobby projects to advertise themselves as such, and not as "this is a library for [x] that you can use in your stuff with the expectations of [maintenance/performance/compatibility/etc.]."
Resume-driven development seems to cause people to oversell their hobby projects as software that is ready to have external users.
> If you want a mature protobuf implementation you should probably buy one
No software is ever developed this way. For some reason, libraries are always free. Approximately nobody will buy paid libraries.
procaryote · 2h ago
Isn't that an argument _for_ having a "mature" label? To avoid the hobbyists who have no intention to maintain their thing?
Also there are lots of lovely projects maintained at high levels by hobbyists, and plenty of abandonware that was at some point paid for
schmichael · 5h ago
A great point! All of the libraries I mentioned are created and maintained by corporations. Hobbyists, as always, are free to do as they please without judgement from me. :)
I will say I get great satisfaction from the little envparse library I wrote needing near-0 maintenance. It’s a rare treat to be able to consider any project truly done.
philsnow · 41m ago
> do I even need this crate at all? 35 lines later I had the parts of dotenv I needed.
I'm not saying you copy-pasted those 35 lines from dotenvy, but for the sake of argument let's say you did: now you can't automatically benefit from dotenvy patching some security issue in those lines.
a2128 · 17m ago
To benefit you have to actually trust the current and future maintainers of the package, its dependencies, the dependencies of its dependencies, etc. You can also automatically get breached in a supply chain attack, so it's a tradeoff
XxiXx · 9h ago
I think it's a "cultural" thing. With Go you often find developers/projects proudly mentioning that any or just a few non-std dependencies are used. Coming from Go it really feels strange when you see pages of dependencies scrolling over your screen when you build a Rust project.
api · 58m ago
Go has a fatter standard library and a "fat" runtime with built-in green threads (an asynchronous runtime basically) and garbage collection, so you get more out of the box and thus end up using fewer dependencies.
sophacles · 7h ago
I have yet to come across a go project that doesn't pull in tons of 3rd party code as well. It seems like maybe you're over-stating the "culture" a bit.
hu3 · 1h ago
> I have yet to come across a go project that doesn't pull in tons of 3rd party code as well.
These have Zero dependencies. It's not rare in Go land.
Yeah, while I’ve seen some great libraries that follow the practice of minimizing their dependencies, I’m a bit annoyed with the amount of dependencies that docker will bring along [1]. I’ve been on the lookout for alternatives for my docker needs, but the state of podman, buildah and some others that I checked is similar. They all bring in roughly the same number of dependencies… if anyone knows of a stripped down Go lib that can be used to build from a Dockerfile, pull, and run a container, I would be grateful for any suggestions. Heck docker / moby isn’t even using go.mod proper.
Wow, that's massive. I guess it's inevitable that a popular piece of open-source software for end-users will be compelled to accrue dependencies due to popular demand for features that require them.
I feel Telegraf made a good compromise: out of the box, it comes with a _ton_ of stuff[1] to monitor everything, but they make it possible to build only with pieces that you need via build tags, and even provide a tool to extract said tags from your telegraf config[2]. But lots of supply-chain security stuff assume everything in go.mod is used, so that can results in a lot of noise.
How much maintenance could you possibly need to load secrets from .env into the environment.
shepmaster · 9h ago
I agree with your general point, but for this specific functionality, I’ll point out that setting environment variables of the current process is unsafe. It took us a long time to realize it so the function wasn’t actually marked as unsafe until the Rust 2024 edition.
What this means in practice is that the call to invoke dotenv should also be marked as unsafe so that the invoker can ensure safety by placing it at the right place.
If no one is maintaining the crate, that won’t happen and someone might try to load environment variables at a bad time.
andy_xor_andrew · 2h ago
ok, I'm hooked - how is setting an env var in the current process unsafe? My gut says it's not unsafe in a memory-ownership sense, but rather in a race condition sense?
whatever the issue is, "setting an env var is unsafe" is so interesting to me that I'm now craving a blog post explaining this
Orangeair · 2h ago
It's a long standing bug, setenv and unsetenv are not thread-safe
Ironically a project that hasn't been changed in a while "unmaintained" is a good candidate for bumping to v1, while a project with new breaking commits every day is a bad candidate.
csomar · 9h ago
On the other hand loading .env from the environment is critical (since you are usually passing secrets through .env). I wouldn't want to maintain that myself and not share it with a xxK other projects in case there is a vulnerability.
dathinab · 2m ago
the issue is loading and setting env (which is the default for dot env libraries)
_is fundamentally unsound thanks to unix/posix_
no way around that
hence why set env wasn't marked as unsafe _even through it not being fully save being known since extremely early rust days maybe even 1.0_
it not being unsafe wasn't a oversight but a known to not be fully sound design decision which had been revisited and changed in recently
amelius · 1h ago
I think the main problem is that you should be able to run dependencies inside their own sandbox, and the language focuses only on memory safety within a monolithic program.
dathinab · 39m ago
the problem is if you put library dependencies in their own sandbox you have a different kind of interface (much more limited) for libraries
like e.g. if we look at sandbox boundaries we have:
- some in language permission enforcement (e.g. Java Security Manage) -- this approach turned out to be a very bad idea
- process boundaries, i.e. take the boundary the OS enforces and lock it down more (e.g. by stuff like pledge, cgroups etc.) -- this approach turned out okayish
- VM boundaries (e.g. firecracker VMs) -- tourned out well
- emulation boundaries (e.g. WASM) -- mixed history, can turn out well especially if combined with worker processes which lock themself down
but what that means in practice is that wanting the reliably sand box library dependencies will most likely lead to more or less IPC boundaries between the caller and the libary
what that means is practice it's unsuited for a lot of thing
e.g. for most utility lib it's very unsuited
e.g. for a lot (but not all) data structure libs its unsuited and might be a huge issue
e.g. you can apply it to a web-server, but then you are basically reinventing CGI, AGI which okay but can quite compete with perf.
e.g. but you can't apply it to some fundamental runtime engine (e.g. tokio), worse you now might have one copy of the engine running per sandbox... (but you can apply it to some sub-part of tokio internals)
People have tried this a lot in various ways.
But so far this always died off in the long run.
Would be nice if the latest push based around WASM would have some long term success.
ATechGuy · 32m ago
Thanks! This is a very detailed explanation of why existing sandboxing techniques will not work as expected for dependencies (wrt to functionality or performance).
aliceryhl · 9h ago
I'm quite careful to tightly control the dependencies of Tokio. All dependencies are under control by members of the Tokio team or others that I trust.
demarq · 11h ago
> Many call for adding more to the rust standard library much like Go
This is the way.
dralley · 11h ago
There should be a second stdlib with relaxed stability guarantees. Don't fill the normal stdlib full of cruft that can never be changed again.
It's unfortunate that the response so far hasn't been very positive
tadfisher · 2h ago
That proposal is not exactly this; that seems to propose a "blessed crates" namespace which includes popular open-source libraries. I read this proposal as a Python-style batteries-included stdlib.
wofo · 2h ago
What the OP proposes is not exactly a bigger stdlib, because they mention it should have "relaxed stability guarantees". Or is python allowed to change their stdlib in backwards-incompatible ways?
hmry · 3m ago
Yeah, they do so regularly. Every version removes several old stdlib features (after the last version in which they were not deprecated goes EOL)
infogulch · 9h ago
This is obviously the best solution for Rust. A 'metalibrary' library type would add a lot of value to the ecosystem as a nexus:
- All included crates can be tested for inter-compatibility
- Release all included crates under a single version, simplifying upgrades
- Sample projects as living documentation to demo integrations and upgrades
- Breaking changes can be held until all affected crates are fixed, then bump all at once
- An achievable, valuable, local goal for code review / crev coverage metrics
There could be general "everything and the kitchen sink" metalibraries, metalibraries targeted at particular domains or industries, metalibraries with different standards for stability or code review, etc. It might even be valuable enough to sell support and consulting...
demarq · 10h ago
Yeah, I agree. Something like the Boost lib for C++
jerf · 9h ago
A strong advantage of that approach is that you don't need to be the core Rust team to do it. Anyone who wants to do this can just start doing it now.
shepmaster · 9h ago
I agree. Unfortunately, I think that a lot of the people who ask for a bigger standard library really just want (a) someone else to do the work (b) someone they can trust.
The people working on Rust are a finite (probably overextended!) set of people and you can't just add more work to their plate. "Just" making the standard library bigger is probably a non-starter.
I think it'd be great if some group of people took up the very hard work to curate a set of crates that everyone would use and provide a nice façade to them, completely outside of the Rust team umbrella. Then people can start using this Katamari crate to prove out the usefulness of it.
However, many people wouldn't use it. I wouldn't because I simply don't care and am happy adding my dependencies one-by-one with minimal feature sets. Others wouldn't because it doesn't have the mystical blessing/seal-of-approval of the Rust team.
demarq · 2h ago
lets put a price on it
imtringued · 9h ago
This is only an advantage if the core Rust team is uncooperative, which is sad rather than something to be happy about.
jerf · 6h ago
The "Rust core team" should be working on the "Rust core", not every little thing that someone somewhere thinks should go in a standard library. It is part of the job of a "core team" to say "no".
A lot.
Like, a lot a lot a lot. Browse through any programming language that has an open issue tracker for all the closed proposals sometime. Individually, perhaps a whole bunch of good ideas. The union of them? Not so much.
Alupis · 2h ago
So we reinvent Java's bloated SDK again, with all of the "javax" packages. What's old is new?
int_19h · 1h ago
Well, it turned out that the alternative is even worse, so... let's chalk it down to learning experience.
zanecodes · 9h ago
The non-standard library, if you will.
SkiFire13 · 10h ago
The issue with that is how to get everyone to agree on how that would work, e.g. what the criteria for this extension would be, what's the policy for future changes, who will maintain all of this, etc etc.
morganherlocker · 2h ago
Now instead of seeing millions of lines of inscrutable code in your program bloating binary sizes, you can see it in every program (that doesn't disable stdlib).
lantastic · 2h ago
In every program that uses a particular feature from the stdlib. Given the same feature, I tend to trust stdlib more than some rando project. And if you don't trust the stdlib, why would you trust the compiler?
bigstrat2003 · 9h ago
I think that the bare bones stdlib is a huge mistake in Rust. I would love to see that rectified. Unfortunately, approximately 5 other people share that view. The Rust community as a whole is very opposed to adding functionality to std.
pjmlp · 9h ago
Indeed, yes sometimes this brings cruft into the mix.
However I rather have cruft that works everywhere the toolchain is fully implemented, instead of playing whack-a-mole with third party libraries when only some platforms are supported.
constantcrying · 1h ago
That is a serious burden on the maintainers, it creates all kinds of different problems, especially if the functionality of the libraries assumes a certain execution environment. Rust doesn't just target x86 desktops.
echelon · 10h ago
No way. I'd much prefer we have a constellation of core companion libraries like Google's Guava.
We do not need to saddle Rust with garbage that will feel dated like Python's standard library. Cargo does the job just fine. We just need some high quality optional batteries.
Embedded projects are unlikely to need standard library bloat. No_std should be top of mind for everyone.
Something that might make additional libraries feel more first class: if cargo finally got namespaces and if the Rust project took on "@rust/" as the org name to launch officially sanctioned and maintained packages.
procaryote · 1h ago
Python's standard library is the main reason python is usable.
Python packaging is somehow a 30 year train crash that keeps going, but the standard library is good enough that I can do most things without dependencies or with very small number of them.
jaas · 9h ago
I don't think an additional standard library layer, whatever you call it, has to have the same tight controls on backwards compatibility and evolution that the actual standard library has. IMO the goal of creating it should be to improve supply chain security, not to provide an extremely stable API, which might be more of a priority at lower levels but chokes off the kind of evolution that will be needed.
I think what you're suggesting is a great idea for a new standard library layer, you're just not using that label. A set of packages in a Rust namespace, maintained by the same community of folks but under policies that comply with best practices for security and some additional support to meet those best practices. The crates shouldn't be required, so no_std should work just as it would prior to such a collection.
No comments yet
bigstrat2003 · 9h ago
> We do not need to saddle Rust with garbage that will feel dated like Python's standard library.
Python's standard library is a strength, not a weakness. Rust should be so lucky. It's wonderful to have basic functionality which is guaranteed to be there no matter what. Many people work in environments where they can't just YOLO download packages from the Internet, so they have to make do with whatever is in the stdlib or what they can write themselves.
echelon · 9h ago
> Python's standard library is a strength, not a weakness. Rust should be so lucky.
Rust is luckier. It has the correct approach. You can find every battery you need in crates.io.
Python has had monstrosities like urllib, urllib2, http, etc. All pretty much ignored in favor of the external requests library and its kin. The standard library also has inconsistencies in calling conventions and naming conventions and it has to support those *FOREVER*.
The core language should be pristine. Rust is doing it right. Everything else you need is within grasp.
wolvesechoes · 5h ago
"Rust is doing it right."
Standard response every time there is some criticism of Rust.
echelon · 3h ago
bigstrat2003's argument is approximately "Python is batteries included"
My counter argument is that the "batteries included" approach tends to atrophy and become dead weight.
Your counter seems to be "that's not an argument, that's just Rust hype."
Am I interpreting you correctly? Because I think my argument is salient and correct. I don't want to be stuck with dated APIs from 20 years of cruft in the standard library.
The Python standard library is where modules go to die. It has two test frameworks nobody uses anymore, and how many XML libraries? Seven? (The correct answer is "four", I think. And that's four too many.) The Python standard library has so much junk inside, and it can't be safely removed or cleaned up.
A standard library should be data structure/collections, filesystem/os libraries, and maybe network libraries. That's it. Everything else changes with too much regularity to be packed in.
Daishiman · 44m ago
Your critique doesn't match the reality of Python users.
There is a single datetime library. It covers 98% of use cases. If you want the final 2% with all the bells and whistles you can download it if you wish.
There is a single JSON library. It's fast enough for almost anything you want. If you want faster libraries with different usability tradeoffs you can use one but I have never felt compelled to do so.
Same thing with CSV, filesystem access, DB api, etc. They're not the best libraries at the time of any script you're writing, but the reality is that you never really need the best, most ergonomic library ever to get you through a task.
Because of this, many big complex packages like Django have hardly any external dependencies.
If anything you're not the one getting stuck with date APIs; it's the Python core devs. Maintainers of other packages are always free to choose other dependencies, but they almost invariably find that the Python stdlib is good enough for everything.
pjmlp · 9h ago
Python's garbage works everywhere there is a full CPython implementation, I see that as an advantage.
echelon · 9h ago
I develop for Linux, Mac, and Windows. Multiple architectures and OSes. I rarely see platform issues with Rust. It's typically only stuff at the edge, like CUDA libraries, that trip up cross-platform builds.
Rust, as a systems language, is quite good at working on a variety of systems.
pjmlp · 8h ago
Starts already that Rust won't support architectures not available on LLVM, but on GCC, otherwise having a Rust frontend project for GCC wouldn't be a thing.
And the systems language remark, I am still looking forward when sorting ABI issues for binary libraries is finally something that doesn't need to go through solutions designed for C and C++.
dezgeg · 30m ago
What architectures that are missing from LLVM are commercially relevant today and not on well-earned retirement?
As a fellow rust developer, I love our dependencies but I put a lot of effort into pruning the ones I want to use. If I see a crate using too many I might contribute to it or find a replacement.
If you want to use dependencies, I wouldn't be surprised when you realise they also want to use dependencies. But you can put your money/time in the right places. Invest in the dependencies that do things well.
gxt · 9h ago
You can audit your dependencies for crates with security vulnerabilities reported to the RustSec Advisory Database, also block unmaintained crates, and enforce your license requirements using SPDX expressions with cargo-audit and cargo-deny.
You can ensure that third-party Rust dependencies have been audited by a trusted entity with cargo-vet.
And you should have taken a look at where those 3M locs come from, it's usually from Microsoft's windows-rs crates that are transitively included in your dependencies through default features and build targets of crates built to run on windows.
Everyone is in such a rush to get their project out the door, no one has time to generate a key and properly code sign releases and begin developing a more secure chain. Now we have JS package "whatever code" ecosystem but for Rust. As if we haven't watched NPM get hacked many times over the last decade or so.
mcflubbins · 9h ago
> Everyone is in such a rush to get their project out the door
This is the cause of so many issues.
And its not like we're at war or trying to cure the next pandemic, we're writing CRUD apps and trying to convince people to click on adds for crap they don't need.
1vuio0pswjnm7 · 8h ago
"Not thinking about package management careful makes me sloppy."
Isn't the point of a memory safe language to allow programmers to be sloppy without repercussions, i.e., to not think about managing memory and even to not understand how memory works.
Would managing dependencies be any different. Does Rust allow programmers to avoid thinking carefully about selecting dependencies.
1vuio0pswjnm7 · 4h ago
If careful programmers who can manage memory should use the same language as careless ones who cannot, then does this mean both should also automatically use third party libraries by default.
Are there systems languages that provide memory management but do not default to using third party libraries. If yes, then do these languages make it easier for programmers to avoid dependencies.
timewizard · 2h ago
> to be sloppy without repercussions
It's the difference between a wet mess and a dry one. Rust creates dry messes. It's still a mess.
sophacles · 8h ago
> Isn't the point of a memory safe language to allow programmers to be sloppy without repercussions, i.e., to not think about managing memory and even to not understand how memory works
No. The point is even the best programmers of unsafe languages regularly introduce both simple and subtle bugs into codebases while being careful about handling memory correctly, and therefore we should use languages that don't even allow those bugs for most every use case. Using these languages still allows crap programmers to waste GBs of correctly allocated and handled memory, and good programmers to write tight, resouce-sipping code.
Dependencies are orthogonal to this.
empath75 · 2h ago
No, the point is to stop you from being sloppy. The code won't compile if you're sloppy with memory management.
You can be _relatively_ sure that you're not introducing memory unsafety by adding a dependency, but you can't be sure that it isn't malware unless you audit it.
James_K · 28m ago
When I am compiling Rust applications, I must admit I'm always rather bemused at the number of dependencies pulled. Even what I'd have thought to be simple tools reach easily about 200 dependent packages. It's nightmarish. One way this becomes particularly apparent is if you're trying to create a reproducible package for Guix or Nix. You end up having to manually specify a package for every different Rust library because of how those system require reproducible builds. The process of writing Guix package for software has been extremely illuminating for me, as to just how deeply nested certain technologies are vs. others. I'd be willing to bet it's a good metric for what sticks around. If you've got 200 dependencies, I don't think your software is gonna last the test of time. It seems a recipe for endless churn.
thrance · 9h ago
I had the same concerns when I started using Rust, but then I eventually embraced it, for better or worse. Cargo makes it so your build almost never breaks (it's happened maybe twice for the 8 years I've been doing Rust). Plus there are still way less vulnerabilities with Rust projects than non-Rust projects, in spite of the crazy number of dependencies.
If I was to design a Rust 2.0, I'd make it so dependencies need permissions to access IO, or unsafe code, etc.
srikanth767 · 9h ago
True
csomar · 9h ago
> when checking a rust security advisory mentioning that dotenv is unmaintained
This is a problem with all languages and actually an area where Rust shines (due to editions). Your pulled in packages will compile as they previously did. This is not true for garbage collected languages (pun intended).
> Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust .... How could I ever audit all of that code?
Again, another area where Rust shines. You can audit and most importantly modify the code. This is not that easy if you were using Nodejs where the runtimes are behind node/v8 or whatever. You compile these things (including TLS) yourself and have full control over them. That's why Tokio is huge.
lolinder · 9h ago
> This is not true for garbage collected languages
JavaScript is backwards compatible going back effectively forever, as is Java. Rust's unique system is having a way to make breaking changes to the language without breaking old code, not that they prioritize supporting old code indefinitely.
The libraries are a different story—you're likely to have things break under you that rely on older versions of libraries when you update—but I don't see Rust actually having solved that.
> You can audit and most importantly modify the code. This is not that easy if you were using Nodejs where the runtimes are behind node/v8 or whatever.
Node and V8 are open source, which makes the code just as auditable and modifiable as the 3.6 million lines of Rust. Which is to say, both are equally unapproachable.
csomar · 9h ago
> The libraries are a different story—you're likely to have things break under you that rely on older versions of libraries when you update—but I don't see Rust actually having solved that.
No language can fix that. However, I've lost count of the times my Python/JavaScript interpretation fails because of something in one of the dependencies. Usually, it's not a JS/Python problem but rather has to do with a Node/Python version update. It always boils down to the "core" issue which is the runtime. That's why I like that Rust give me a "fixed" runtime that I download/compile/package with my program.
> Node and V8 are open source, which makes the code just as auditable and modifiable as the 3.6 million lines of Rust. Which is to say, both are equally unapproachable.
I've recently patched a weird bug under Tokio/Otel and can't imagine doing that with Node/V8 without it being a major hassle. It is relatively straightforward in Rust though requires maintaining your own fork of only the dependency/branch in question.
a) Reminds me of Ginger Bill (the Odin language creator, no affiliation) who on a podcast cited the avoidance of dependency hell as one of the main reasons for Odin never ever having an official package manager, see https://www.youtube.com/watch?v=fYUruq352yE&t=11m26s (timestamped to the correct position)
b) another programmer rather seriously worried about the software quality is Jonathan Blow, who's talk "Preventing the Collapse of Civilization" is worth watching in my opinion: https://www.youtube.com/watch?v=ZSRHeXYDLko
This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painstakingly included parts of into your size constrained environment (fit it on a floppy). You probably picked apart that library and pulled the bits you needed, integrating them into your builds to be as small as possible.
Today we pile libraries on top of libraries on top of libraries. Its super easy to say `import foolib`, then call `foolib.do_thing()` and just start running. Who knows or cares what all 'foolib' contains.
At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.
In some cases the languages make this worse. Go and Rust, for example, encourage everything for a single package/mod to go in the same file. Adding optional functionality can get ugly when it would require creating new modules, but if you only want to use a tiny part of the module, what do you do?
The only real solution I can think of to deal with this long term is ultra-fine-grained symbols and dependencies. Every function, type, and other top-level language construct needs to declare the set of things it needs to run (other functions, symbols, types, etc). When you depend on that one symbol it can construct, on demand, the exact graph of symbols it needs and dump the rest for any given library. You end up with the minimal set of code for the functionality you need.
Its a terrible idea and I'd hate it, but how else do you address the current setup of effectively building the whole universe of code branching from your dependencies and then dragging it around like a boat anchor of dead code.
https://learn.microsoft.com/en-us/dotnet/core/deploying/trim...
https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...
which get reinvented all the time, like in dotnet with "trimming" or in JS with "tree-shaking".
C/C++ compiler have been doing that since before dot net was a thing, same for rust which does that since it's 1.0 release (because it's done by LLVM ;) )
The reason it gets reinvented all the time is because while it's often quite straight forward in statically compiled languages it isn't for dynamic languages as finding out what actually is unused is hard (for fine grained code elimination) or at lest unreliable (pruning submodules). Even worse for scripting languages.
Which also brings use to one area where it's not out of the box, if you build .dll/.so in one build process and then use them in another. Here additional tooling is needed to prune the dynamic linked libraries. But luckily it's not a common problem to run into in Rust.
In general most code size problems in Rust aren't caused by too huge LOC of dependencies but by an overuse of monopolization. The problem of tons of LOC in dependencies is one of supply chain trust and review ability more then anything else.
The comment you're replying to is talking about not pulling in dependencies at all, before compiling, if they would not be needed.
This is an extreme example, but the same thing happens very often at a smaller scale. Optional functionality can't always be removed statically.
Rust does have a feature flagging system for this kind of optional functionality though. It's not perfect, but it would work very well for something like curl protocol backends though.
it's also not always a fair comparison, if you include tokio in LOC counting then you surely would also include V8 LOC when counting for node, or JRE for Java projects (but not JDK) etc.
It is also done on the cousin Android, and available as free beer on GraalVM and OpenJ9.
The analogy I use is cooking a huge dinner, then throwing out everything but the one side dish you wanted. If you want just the side-dish you should be able to cook just the side-dish.
I just don't listen. Things should be easy. Rust is easy. Don't overthink it
It should be easy to build and deploy profiling-aware builds (PGO/BOLT) and to get good feedback around time/instructions spent per package, as well as a measure of the ratio of each library that's cold or thrown away at build time.
I'll note that it isn't just PGO/BOLT style optimizations. Largely, it is not that at all, oddly.
Instead, the problem is one of stability. In a "foundation that doesn't move and cause you to fall over" sense of the word. Consider if people made a house where every room had a different substructure under it. That, largely, seems to be the general approach we use to building software. The idea being that you can namespace a room away from other rooms and not have any care on what happens there.
This gets equally frustrating when our metrics for determining the safety of something largely discourages inaction on any dependencies. They have to add to it, or people think it is abandoned and not usable.
Note that this isn't unique to software, mind. Hardware can and does go through massive changes over the years. They have obvious limitations that slow down how rapidly they can change, of course.
Clarification: Go allows for a very simple multi-file. It’s one feature I really like, because it allows splitting otherwise coherent module into logical parts.
For example, you can't split up a module into foo.rs containing `Foo` and bar.rs containing `Bar`, both in module 'mymod' in such a way that you can `use mymod::Bar and foo.rs is never built/linked.
My point is the granularity of the package/mod encourages course-grained deps, which I argue is a problem.
yesn't, you can use feature flags similar to `#if` in C
but it's also not really a needed feature as dead code elimination will prune out all code functions, types, etc. you don't use. Non of it will end up in the produced binary.
Historically Rust wanted that foo.rs to be renamed foo/mod.rs but that's no longer idiomatic although of course it still works if you do that.
in rust crates are semantically one compilation unit (where in C oversimplified it's a .h/.c pair, and practically rustc will try to split it in some more units to speed up build time).
the reason I'm pointing this out is because many sources of "splitting a module across files" come from situations where 1 file is one compilation unit so you needed to have a way to split it (for organization) without splitting it (for compilation) in some sitation
What? I don't know about Go, but this certainly isn't true in Rust. Rust has great support for fine-grained imports via Cargo's ability to split up an API via crate features.
It's a terrible idea because you're trying to reinvent section splitting + `--gc-sections` at link time, which rust (which the article is about) already does by default.
Things like --gc-sections feels like a band-aid, a very practical and useful band-aid, but a band-aid none the less. You're building a bunch of things you don't need, then selectively throwing away parts (or selectively keeping parts).
IMO it all boils down to the granularity. The granularity of text source files, the granularity of units of distribution for libraries. It all contributes to a problem of large unwieldy dependency growth.
I don't have any great solutions here, its just observations of the general problem from the horrifying things that happen when dependencies grow uncontrolled.
If each layer of “package abstraction” is only 50% utilised, then each layer multiplies the total size by 2x over what is actually required by the end application.
Three layers — packages pulling in packages that pull their own dependencies — already gets you to 88% bloat! (Or just 12% useful code)
An example of this is the new Windows 11 calculator that can take several seconds to start because it loads junk like the Windows 10 Hello for Business account recovery helper library!
Why? Because it has currency conversion, which uses a HTTP library, which has corporate web proxy support, which needs authentication, which needs WH4B account support, which can get locked out, which needs a recovery helper UI…
…in a calculator. That you can’t launch unless you have already logged in successfully and is definitely not the “right place” for account recovery workflows to be kicked off.
But… you see… it’s just easier to package up these things and include them with a single line in the code somewhere.
If anything, the 1980s is when the idea of fully reusable, separately-developed software components first became practical, with Objective-C and the like. In fact it's a significant success story of Rust that this sort of pervasive software componentry has now been widely adopted as part of a systems programming language.
Curate a collection of libraries you use and trust. This will probably involve making a number of your own. Wheel-reinvention, if you will. If done properly, even the upfront time cost will save in the long-run. I am in the minority here, but I roll my own libs whenever possible, and the 3rd party libs I use are often ones I know, have used been for, and vetted that they have a shallow tree of their own.
Is this sustainable? I don't know. But It's the best I've come up with, in order to use what I see as the best programming language available for several domains.
There are a lot of light-weight, excellent libs I will use without hesitation, and have wide suitability. Examples:
Heavier, and periodically experience mutual-version hell, but are are very useful for GUI programs: On a darker note, the rust web ecosystem maybe permanently lost to async and messy dependencies. Embedded is going that way too, but I have more hope there, and am doing my best to have my own tooling.In the absence of a technical solution, all others basically involve someone else having to audit and constantly maintain all that code and social/legal systems of trust. If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult.
I don't see the advantage. Just a different axis of disadvantage. Take python for example. It has a crazy big standard library full of stuff I will never use. Some people want C++ to go in that direction too -- even though developers are fully capable of rolling their own. Similar problem with kitchen-sink libraries like Qt. "batteries included" languages lead to higher maintenance burden for the core team, and hence various costs that all users pay: dollars, slow evolution, design overhead, use of lowest common denominator non-specialised implementations, loss of core mission focus, etc.
I think Rust really needs to do more of this. I work with both Go and Rust daily at work, Go has its library game down -- the standard library is fantastic. With Rust it's really painful to find the right library and keep up for a lot of simple things (web, tls, x509, base64 encoding, heck even generating random numbers.)
I'm sure I'll miss some, but IIRC C++ 26 is getting the entire BLAS, two distinct delayed reclamation systems and all of the accompanying infrastructure, new container types, and a very complicated universal system of units.
All of these things are cool, but it's doubtful whether any of them could make sense in a standard library, however for C++ programers that's the easiest way to use them...
It's bedlam in there and of course the same C++ programmers who claim to be "worried" that maybe somebody hid something awful in Rust's crates.io are magically unconcerned that copy-pasting tens of millions of lines of untested code from a third party into absolutely every C++ program to be written in the future could be a bad idea.
Applications still need the functionality. The need doesn't magically disappear when installing dependencies is a pain. If a crate has a bug, the entire ecosystem can trivially get the fixed version. If the Stackoverflow snippet a C app is vendoring has a bug, that fix is never getting in the app.
Yet, if someone were to write a book which explained things properly (probably a 3000 word article would suffice to turn anyone into a 10x dev), nobody would buy it. This industry is cooked.
If I were designing a new language I think I'd be very interested in putting some sort of capability system in so I can confine entire library trees safely, and libraries can volunteer somehow what capabilities they need/offer. I think it would need to be a new language if for no other reason than ecosystems will need to be written with the concept in them from the beginning.
For instance, consider an "image loading library". In most modern languages such libraries almost invariably support loading images from a file, directly, for convenience if nothing else. In a language that supported this concept of capabilities it would be necessary to support loading them from a stream, so either the image library would need you to supply it a stream unconditionally, or if the capability support is more rich, you could say "I don't want you to be able to load files" in your manifest or something and the compiler would block the "LoadFromFile(filename)" function at compile time. Multiply that out over an entire ecosystem and I think this would be hard to retrofit. It's hugely backwards incompatible if it is done correctly, it would be a de facto fork of the entire ecosystem.
I honestly don't see any other solution to this in the long term, except to create a world where the vast majority of libraries become untargetable in supply chain attacks because they can't open sockets or read files and are thus useless to attackers, and we can reduce our attack surface to just the libraries that truly need the deep access. And I think if a language came out with this design, you'd be surprised at how few things need the dangerous permissions.
Even a culture of minimizing dependencies is just delaying the inevitable. We've been seeing Go packages getting supply-chain-attacked and it getting into people's real code bases, and that community is about as hostile to large dependency trees as any can be and still function. It's not good enough.
In your particular example of image loading, you want WUFFS. https://github.com/google/wuffs
In WUFFS most programs are impossible. Their "Hello, world" doesn't print hello world because it literally can't do that. It doesn't even have a string type, and it has no idea how to do I/O so that's both elements of the task ruled out. It can however, Wrangle Untrusted File Formats Safely which is its sole purpose.
I believe there should be more special purpose languages like this, as opposed to the General Purpose languages most of us learn. If your work needs six, sixteen or sixty WUFFS libraries to load different image formats, that's all fine because categorically they don't do anything outside their box. Yet, they're extremely fast because since they can't do anything bad by definition they don't need those routine "Better not do anything bad" checks you'd write in a language like C or the compiler would add in a language like Rust, and because they vectorize very nicely.
As long as all library code is compiled/run from source, a compiler/runtime can replace system calls with wrappers that check caller-specific permissions, and it can refuse to compile or insert runtime panics if the language's escape hatches would be used. It can be as safe as the language is safe, so long as you're ok with panics when the rules are broken.
It'd take some work to document and distribute capability profiles for libraries that don't care to support it, but a similar effort was proven possible with TypeScript.
Languages (plural) ... no single language will work for everyone.
Cargo makes it so simple to add tons of dependencies that it is really hard not to do it. But that does not stop here: even if I try to be careful with adding dependencies, a couple dependencies are likely to pull tens of transitive dependencies each.
"Then don't depend on them", you say. Sure, but that means I won't write my project, because I won't write those things from scratch. I could probably audit the dependency (if it wasn't pulling 50 packages itself), but I can't reasonably write it myself.
It is different with C++: I can often find dependencies that don't pull tens of transitive dependencies in C++. Maybe because it's harder to add dependencies, maybe because the ecosystem is more mature, I don't know.
But it feels like the philosophy in Rust is to pull many small packages, so it doesn't seem like it will change. And that's a pity, because I like Rust-the-language better than C++-the-language. It just feels like I trade "it's not memory-safe" for "you have to pull tons of random code from the Internet".
I think it makes a good point that some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded. To some degree that is a plus though as you likely trust the maintainers of your OS distribution to provide stable, supported libraries.
As other commenters have said, perhaps this is an area where the Rust maintainers could provide some kind of extended standard library where they don't guarantee backwards compatibility forever, but do provide guarantees about ongoing fixes for security issues.
Not in my case. I manually compile all the dependencies (either because I need to cross-compile, or because I may need to patch them, etc). So I clearly see all the transitive dependencies I need in C++. And I need a lot less than in Rust, by a long shot.
It's part of the reason why software distribution on Linux has been pushed to using containers, removing the point of having shared libraries. I think Google with it's C++ replacement (Carbon) plans on doing it's own system.
My understanding of people distributing their software in containers is that they can't be arsed to learn how to do it properly. They would install their software and ship the entire computer if that was cost effective.
This is not to denigrate the huge and critical effort that makes current computing possible, and that is likely unavoidable in the real world. But software distribution needs to evolve.
It could be better, but the current solutions (npm, go, python,...) favor only the developers, not the maintainers and packagers.
e.g. Bottles, WebkitGTK (distros liked keeping this one held back even though doing so is a security risk)
IMHO it shouldn't be the responsibility of the OS vendor to package third party applications.
That said, the labor needed to keep the stuff together could be reduced a lot by the more ergonomical and universal packaging and distribution methods like Cargo (and, dare I say, npm). I think some kind of a better bridge between developers and distros could be found here.
You need to think a bit harder about that, to help you decide whether your position is rational.
In Rust, I'm sometimes actually tempted to wrap a C/C++ library (and its few dependencies) instead of getting the Rust alternative (and its gazillion dependencies).
I'm not sure it's a philosophy, more a pragmatic consideration for compilation speeds. Anyone who's done a non-trivial amount of Rust knows that moment when the project gets too big and needs to split into separate crates. It's kinda sad that you can't organize code according to proper abstractions, many times I feel forced to refactor for compiler performance.
But that's not practical for all situations. For example, Web frontend developer culture might be the worst environment, to the point you often can't get many things done in feasible time, if you don't adopt the same reckless practices.
I'm also seeing it now with the cargo-culting of opaque self-hosted AI tools and models. For learning and experimenting, I'd spend more time sufficiently compartmentalizing an individual tool than with using it.
This weekend, I'm dusting off my Rust skills, for a small open source employability project (so I can't invest in expensive dependency management on this one). The main thing thing bothering me isn't allocation management, but the sinking feeling when I watch the cast-of-thousands explosion of transitive dependencies for the UI and async libraries that I want to use. It's only a matter of time before one of those is compromised, if not already, and one is all it takes.
Devs can add whatever they feel like on their workstations but it will be a sad build server if they get pushed without permission.
Anything else will get abused in the name of expediency and just-this-one-time.
Also, the process for adding a crate/gem/module/library needs to be the same as anything else: license review, code review, subscription to the appropriate mailing list or other announce channel, and assignment of responsibility. All of these except code review can be really, really fast once you have the process going.
All problems are, at least in part, dependency chain management problems.
The dependency trees for most interpreted or source-distributed languages are ridiculous, and review of even a few of those seems practically impossible in a lot of development environments.
A compromised dev machine is also a problem.
it's complex problem with tons of partial solutions which each have tons of ways to implement them with often their no being a clear winner
i.e. it's the kind of hard to solve by consensus problem
e.g. the idea of a extended standard library is old (around since the beginning of rust) but for years it was believed it's probably the best to make it a separate independent project/library for various reason. One being that the saying "the standard library is the place where code goes to die" has been quite true for multiple ecosystems (most noticeably python)
as a side note ESL wouldn't reduce the LOC count it would increase it as long as you fully measure LOCs and not "skip" over some dependencies
There's literally 1000s of RFCs for rust with only a small handful that are integrated. Having this forest, IMO, makes it hard for any given proposal to really stand out. Further, it makes duplicate effort almost inevitable.
Rust's RFC process is effectively a dead letter box for most.
tokio is a work-stealing, asynchronous runtime. This is a feature that would be an entire language. Does OP consider it reasonable to audit the entire Go language? or the V8 engine for Node? v8 is ~10x more lines than tokio.
If Cloudflare uses Node, would you expect Cloudflare to audit v8 quarterly?
This is something I've only ever seen cargo do.
No, cargo will resolve using sem ver compatibility and pick the best version. Nuget, for C# does something very similar.
Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight.
I decided that I should leave this project alone and spend my time elsewhere.
Also, repository size seems an extremely irrelevant metric.
Repository size is directly related to how long it takes to run a build, which is extremely important if I were to contribute to the project.
> Serde seems exactly a case where you absolutely should use an external dependency.
I can't see any reason a parser has a hard dependency on a serialization library.
Which ones are superfluous?
There are good reasons to use dependencies. If someone has solved a problem you need to solve as well it is pointless to duplicate the effort.
>Repository size is directly related to how long it takes to run a build, which is extremely important if I were to contribute to the project.
Totally false. There is zero inherent relation.
>I can't see any reason a parser has a hard dependency on a serialization library.
And because you can't see a reason there is none?
It is totally meaningless to talk about any of this if you can not point out why this is superfluous.
> And because you can't see a reason there is none?
Somehow every other JS based parser doesn't do fancy serialization, as far as I can tell. You can come up with reasons of why one might need it, but as a user of the parser, I want the footprint to be small, and that's a requirement. In fact, that's one of the reasons I never used swc parser in my serious projects.
> Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight.
The lightest weight javascript program relies on V8 to run, which has multiple orders of magnitude more dependencies. Most of which you have never heard of.
At least cargo makes it easier to get a clearer picture of what the dependencies are for a program.
If you have one huge dep it's easier to keep track you're on the latest update, also it's much less likely you'll fat finger it and import something typosquatting.
Also if you're in enterprise you'll have less 100 page SBOM reports.
Keeping track of the latest version is trivial with cargo.
Actually, this isn't true. (Or at least wasn't a while back.) I used to work with a bunch of ex-V8 folks and they really despised third-party dependencies and didn't trust any code they didn't write. They used a few third-party libs but for them most part, they tried to own everything themselves.
.. as in they can afford to rewrite everything
.. can afford to suffer from not invented here syndrome
.. and are under _massive_ threat of people doing supply chain attacks compared to most other projects (as they end up running on nearly any desktop computer and half the phones out there)
this just isn't viable for most projects, not just resource/time investment wise, but also reinventing/writing everything isn't exactly good to reduce bugs if you haven't to reliably access to both resources _and_ expertise. Most companies have to live with having many very average developers, and very tight resource limits.
1. Well defined scope
2. Infrequent changes
Nomad has many of these (msgpack, envparse, cli, etc). These dependencies go years without changing so the dependency management burden rapidly approaches zero. This is an especially useful property for “leaf” dependencies with no dependencies of their own.
I wish libraries could advertise their intent to be Mature. I’d choose a Mature protobuf library over one that constantly tweaked its ergonomics and performance. Continual iterative improvement is often a boon, but sometimes it’s not worth the cost.
It is an easy way to get a somewhat OK standard library as the things you add became popular on their own merits at some point.
Once added, the lowest friction path is to just use the standard library; and as it is the standard library you have a slightly better hope someone will care to maintain it. You can still build a better one if needed for your use-case, but the batteries are included for basic usage
If you want a mature protobuf implementation you should probably buy one. Expecting some guy/gal on the internet to maintain one for your for free seems ill advised.
Nobody is asking for professional quality standards from hobby projects. At best, they are asking for hobby projects to advertise themselves as such, and not as "this is a library for [x] that you can use in your stuff with the expectations of [maintenance/performance/compatibility/etc.]."
Resume-driven development seems to cause people to oversell their hobby projects as software that is ready to have external users.
> If you want a mature protobuf implementation you should probably buy one
No software is ever developed this way. For some reason, libraries are always free. Approximately nobody will buy paid libraries.
Also there are lots of lovely projects maintained at high levels by hobbyists, and plenty of abandonware that was at some point paid for
I will say I get great satisfaction from the little envparse library I wrote needing near-0 maintenance. It’s a rare treat to be able to consider any project truly done.
I'm not saying you copy-pasted those 35 lines from dotenvy, but for the sake of argument let's say you did: now you can't automatically benefit from dotenvy patching some security issue in those lines.
These have Zero dependencies. It's not rare in Go land.
- https://github.com/go-chi/chi 19k stars
- https://github.com/julienschmidt/httprouter 16k stars
- https://github.com/gorilla/mux 21k stars
- https://github.com/spf13/pflag 2.6k stars
- https://github.com/google/uuid 5.6k starts
Many others have just a few dependencies.
[1] https://github.com/moby/moby/blob/master/vendor.mod
I feel Telegraf made a good compromise: out of the box, it comes with a _ton_ of stuff[1] to monitor everything, but they make it possible to build only with pieces that you need via build tags, and even provide a tool to extract said tags from your telegraf config[2]. But lots of supply-chain security stuff assume everything in go.mod is used, so that can results in a lot of noise.
[1] https://github.com/influxdata/telegraf/blob/master/go.mod [2] https://github.com/influxdata/telegraf/tree/master/tools/cus...
How much maintenance could you possibly need to load secrets from .env into the environment.
What this means in practice is that the call to invoke dotenv should also be marked as unsafe so that the invoker can ensure safety by placing it at the right place.
If no one is maintaining the crate, that won’t happen and someone might try to load environment variables at a bad time.
whatever the issue is, "setting an env var is unsafe" is so interesting to me that I'm now craving a blog post explaining this
https://www.evanjones.ca/setenv-is-not-thread-safe.html
non
small "completed" well tested libraries being flagged as security issues due to being unmaintained seem to be starting to become an issue
> Achtung! This is a v0.* version! Expect bugs and issues all around. Submitting pull requests and issues is highly encouraged!
https://github.com/dotenv-rs/dotenv
ZeroVer https://0ver.org/
_is fundamentally unsound thanks to unix/posix_
no way around that
hence why set env wasn't marked as unsafe _even through it not being fully save being known since extremely early rust days maybe even 1.0_
it not being unsafe wasn't a oversight but a known to not be fully sound design decision which had been revisited and changed in recently
like e.g. if we look at sandbox boundaries we have:
- some in language permission enforcement (e.g. Java Security Manage) -- this approach turned out to be a very bad idea
- process boundaries, i.e. take the boundary the OS enforces and lock it down more (e.g. by stuff like pledge, cgroups etc.) -- this approach turned out okayish
- VM boundaries (e.g. firecracker VMs) -- tourned out well
- emulation boundaries (e.g. WASM) -- mixed history, can turn out well especially if combined with worker processes which lock themself down
but what that means in practice is that wanting the reliably sand box library dependencies will most likely lead to more or less IPC boundaries between the caller and the libary
what that means is practice it's unsuited for a lot of thing
e.g. for most utility lib it's very unsuited
e.g. for a lot (but not all) data structure libs its unsuited and might be a huge issue
e.g. you can apply it to a web-server, but then you are basically reinventing CGI, AGI which okay but can quite compete with perf.
e.g. but you can't apply it to some fundamental runtime engine (e.g. tokio), worse you now might have one copy of the engine running per sandbox... (but you can apply it to some sub-part of tokio internals)
People have tried this a lot in various ways.
But so far this always died off in the long run.
Would be nice if the latest push based around WASM would have some long term success.
This is the way.
It's unfortunate that the response so far hasn't been very positive
The people working on Rust are a finite (probably overextended!) set of people and you can't just add more work to their plate. "Just" making the standard library bigger is probably a non-starter.
I think it'd be great if some group of people took up the very hard work to curate a set of crates that everyone would use and provide a nice façade to them, completely outside of the Rust team umbrella. Then people can start using this Katamari crate to prove out the usefulness of it.
However, many people wouldn't use it. I wouldn't because I simply don't care and am happy adding my dependencies one-by-one with minimal feature sets. Others wouldn't because it doesn't have the mystical blessing/seal-of-approval of the Rust team.
A lot.
Like, a lot a lot a lot. Browse through any programming language that has an open issue tracker for all the closed proposals sometime. Individually, perhaps a whole bunch of good ideas. The union of them? Not so much.
However I rather have cruft that works everywhere the toolchain is fully implemented, instead of playing whack-a-mole with third party libraries when only some platforms are supported.
We do not need to saddle Rust with garbage that will feel dated like Python's standard library. Cargo does the job just fine. We just need some high quality optional batteries.
Embedded projects are unlikely to need standard library bloat. No_std should be top of mind for everyone.
Something that might make additional libraries feel more first class: if cargo finally got namespaces and if the Rust project took on "@rust/" as the org name to launch officially sanctioned and maintained packages.
Python packaging is somehow a 30 year train crash that keeps going, but the standard library is good enough that I can do most things without dependencies or with very small number of them.
I think what you're suggesting is a great idea for a new standard library layer, you're just not using that label. A set of packages in a Rust namespace, maintained by the same community of folks but under policies that comply with best practices for security and some additional support to meet those best practices. The crates shouldn't be required, so no_std should work just as it would prior to such a collection.
No comments yet
Python's standard library is a strength, not a weakness. Rust should be so lucky. It's wonderful to have basic functionality which is guaranteed to be there no matter what. Many people work in environments where they can't just YOLO download packages from the Internet, so they have to make do with whatever is in the stdlib or what they can write themselves.
Rust is luckier. It has the correct approach. You can find every battery you need in crates.io.
Python has had monstrosities like urllib, urllib2, http, etc. All pretty much ignored in favor of the external requests library and its kin. The standard library also has inconsistencies in calling conventions and naming conventions and it has to support those *FOREVER*.
The core language should be pristine. Rust is doing it right. Everything else you need is within grasp.
Standard response every time there is some criticism of Rust.
My counter argument is that the "batteries included" approach tends to atrophy and become dead weight.
Your counter seems to be "that's not an argument, that's just Rust hype."
Am I interpreting you correctly? Because I think my argument is salient and correct. I don't want to be stuck with dated APIs from 20 years of cruft in the standard library.
The Python standard library is where modules go to die. It has two test frameworks nobody uses anymore, and how many XML libraries? Seven? (The correct answer is "four", I think. And that's four too many.) The Python standard library has so much junk inside, and it can't be safely removed or cleaned up.
A standard library should be data structure/collections, filesystem/os libraries, and maybe network libraries. That's it. Everything else changes with too much regularity to be packed in.
There is a single datetime library. It covers 98% of use cases. If you want the final 2% with all the bells and whistles you can download it if you wish. There is a single JSON library. It's fast enough for almost anything you want. If you want faster libraries with different usability tradeoffs you can use one but I have never felt compelled to do so.
Same thing with CSV, filesystem access, DB api, etc. They're not the best libraries at the time of any script you're writing, but the reality is that you never really need the best, most ergonomic library ever to get you through a task.
Because of this, many big complex packages like Django have hardly any external dependencies.
If anything you're not the one getting stuck with date APIs; it's the Python core devs. Maintainers of other packages are always free to choose other dependencies, but they almost invariably find that the Python stdlib is good enough for everything.
Rust, as a systems language, is quite good at working on a variety of systems.
And the systems language remark, I am still looking forward when sorting ABI issues for binary libraries is finally something that doesn't need to go through solutions designed for C and C++.
If you want to use dependencies, I wouldn't be surprised when you realise they also want to use dependencies. But you can put your money/time in the right places. Invest in the dependencies that do things well.
You can ensure that third-party Rust dependencies have been audited by a trusted entity with cargo-vet.
And you should have taken a look at where those 3M locs come from, it's usually from Microsoft's windows-rs crates that are transitively included in your dependencies through default features and build targets of crates built to run on windows.
This is the cause of so many issues.
And its not like we're at war or trying to cure the next pandemic, we're writing CRUD apps and trying to convince people to click on adds for crap they don't need.
Isn't the point of a memory safe language to allow programmers to be sloppy without repercussions, i.e., to not think about managing memory and even to not understand how memory works.
Would managing dependencies be any different. Does Rust allow programmers to avoid thinking carefully about selecting dependencies.
Are there systems languages that provide memory management but do not default to using third party libraries. If yes, then do these languages make it easier for programmers to avoid dependencies.
It's the difference between a wet mess and a dry one. Rust creates dry messes. It's still a mess.
No. The point is even the best programmers of unsafe languages regularly introduce both simple and subtle bugs into codebases while being careful about handling memory correctly, and therefore we should use languages that don't even allow those bugs for most every use case. Using these languages still allows crap programmers to waste GBs of correctly allocated and handled memory, and good programmers to write tight, resouce-sipping code.
Dependencies are orthogonal to this.
You can be _relatively_ sure that you're not introducing memory unsafety by adding a dependency, but you can't be sure that it isn't malware unless you audit it.
If I was to design a Rust 2.0, I'd make it so dependencies need permissions to access IO, or unsafe code, etc.
This is a problem with all languages and actually an area where Rust shines (due to editions). Your pulled in packages will compile as they previously did. This is not true for garbage collected languages (pun intended).
> Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust .... How could I ever audit all of that code?
Again, another area where Rust shines. You can audit and most importantly modify the code. This is not that easy if you were using Nodejs where the runtimes are behind node/v8 or whatever. You compile these things (including TLS) yourself and have full control over them. That's why Tokio is huge.
JavaScript is backwards compatible going back effectively forever, as is Java. Rust's unique system is having a way to make breaking changes to the language without breaking old code, not that they prioritize supporting old code indefinitely.
The libraries are a different story—you're likely to have things break under you that rely on older versions of libraries when you update—but I don't see Rust actually having solved that.
> You can audit and most importantly modify the code. This is not that easy if you were using Nodejs where the runtimes are behind node/v8 or whatever.
Node and V8 are open source, which makes the code just as auditable and modifiable as the 3.6 million lines of Rust. Which is to say, both are equally unapproachable.
No language can fix that. However, I've lost count of the times my Python/JavaScript interpretation fails because of something in one of the dependencies. Usually, it's not a JS/Python problem but rather has to do with a Node/Python version update. It always boils down to the "core" issue which is the runtime. That's why I like that Rust give me a "fixed" runtime that I download/compile/package with my program.
> Node and V8 are open source, which makes the code just as auditable and modifiable as the 3.6 million lines of Rust. Which is to say, both are equally unapproachable.
I've recently patched a weird bug under Tokio/Otel and can't imagine doing that with Node/V8 without it being a major hassle. It is relatively straightforward in Rust though requires maintaining your own fork of only the dependency/branch in question.