I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext.
Somehow journald manages to provide a database which is 40x slower to query than running grep on an uncompressed text file.
I'm all in favour of storing logs in an indexed structured format but journald ain't it.
haiku2077 · 52m ago
I've worked as a sysadmin/devops for a double digit number of years.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
thyristan · 24m ago
And made a number of new ones appear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
vinceguidry · 12m ago
I used to be fine with it, until I started writing my own software tools, then looking at how to actually do stuff with systemd and I'm like, nope nope nope nope. At some point when I have space for a new project I'm going to migrate from Arch to Obarun, which uses a modern update on the excellent daemontools to provide init. When systemd works it's mostly fine, when it breaks, path of least resistance is often to reinstall the OS. Which is fine if you're just working on it. Painful if it's your personal machine, and those are what's most likely to break systemd.
bcoates · 44m ago
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example).
You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
nosioptar · 32m ago
I use journald whenever I feel my blood pressure getting too low.
It's slow, truncates lines, and doesn't work well at all with less. It's almost like Pottering created it so that pulseaudio wouldn't be his worst program anymore.
tylerjl · 18m ago
One convenience of journald is that it exposes a single place to plug in log collection for observability tooling
opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup
Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler
thyristan · 38m ago
Well, everyone and noone uses journald. The usual way to log is journald -> (r)syslogd -> remote log destination(s). I've never actually seen any other way.
The preferred journald way of "fetch your logs periodically" wouldn't pass any audit and doesn't work with any kind of log processing software that people use.
gerdesj · 31m ago
Pipe it into lnav eg:
journalctl -b | lnav
or use -f instead of -b for follow instead of everything since boot. Now you have a colourised journal and the power of lnav.
inetknght · 41m ago
For debugging my desktop:
journalctl --follow --tail --no-trunc -b 0
For anything else: export to a syslog server, which basically any tool that matters will support in some fashion.
sethops1 · 42m ago
I use it everyday?
NewJazz · 42m ago
Cloudwatch fucking sucks.
Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector)
Even ChromeOS uses it, even on devices that still use Upstart.
plantain · 2m ago
journald is such a disaster. How is it possible to be so much slower than even gzipped plain text.
shrubble · 40m ago
My view is pragmatic: if you are managing the server and I am a user, then use systemd if you want to.
If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
hungryhobbit · 27m ago
FreeBSD is many great things, but ... a pragmatic choice?
spicyusername · 1h ago
I remember when I first upgraded a system to a version using it. This was probably a RHEL 6 -> 7 update back in 2014 or so.
It made so many common things I had to do so as a sysadmin super straightforward.
freedomben · 1h ago
Same, that was my first experience as well. It was a breath of fresh air in many ways, and after my initial learning of the new syntax, I quickly became a convert.
That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
Propelloni · 38m ago
And some things got super hard to do, e.g. sequence. I guess it is sorted now, but either way, I wouldn't want to go back.
artooro · 33m ago
I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.
haiku2077 · 15m ago
+1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
iambvk · 1h ago
I am not a sysadmin -- so I keep forgetting the command to see logs cause I only need it once every 3 months.
So, systemd change is a pain for me.
slashdev · 46m ago
AI is so helpful for doing those kinds of tasks in the terminal that you only do a couple times a year and can never remember the exact incantation
hundchenkatze · 11m ago
You only need to look those things up once, put it in a snippets.txt file, and then search in the file when you need it.
fsflover · 13m ago
> The unix philosophy cries out: is this the end of Linux (or, as many are calling it, GNU plus Linux)?
I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence:
Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!
rootnod3 · 1h ago
I wouldn’t call it a success it all. Almost managed to do was moving or forcing other software to get locked in. GDM, Podman, Pipewire…just to name a few.
pram · 39m ago
I personally love the init/service/unit-file portion of systemd and have few complaints. It has a lot of powerful features for security, cron, etc that are very simple to use. I also really like journald.
The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
bryanlarsen · 10m ago
Hard disagree. Name resolution feels like a solved problem, so why fix it if it isn't broken?
But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better.
I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved.
Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
thyristan · 19m ago
And those ancillary managers are usually half-baked or outright broken and dangerous, in that they only implement half of the stuff they are trying to replace. E.g. https://github.com/systemd/systemd/issues/25676
jmclnx · 7m ago
systemd has been a success for 2 groups, Windows admins and Fortune 500 Companies.
For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though.
For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.
dontlaugh · 1h ago
These were pretty much my feelings back when I envied launchd and was frustrated with some upstart behaviour. Systemd turned out better than both.
pessimizer · 28m ago
No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now.
Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
Analemma_ · 13m ago
> Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
jmogly · 1h ago
Agree wholeheartedly.
up2isomorphism · 1h ago
There are numerous distributions not using systemd. So adding “complete, utter, unmitigated “ in front of success seems far-fetched.
freedomben · 1h ago
The whole post does have a somewhat hyperbolic tone to it. Personally, I agree with the content and analysis, but I could see the language could feel a bit off-putting, especially given the past flame wars over systemd
jhickok · 1h ago
It's a lighthearted post, so probably not worth tone policing.
freedomben · 48m ago
Agreed, I didn't mean to tone police, just acknowledging it might be off-putting to some people who got wrapped up in the battles back in the day. I wouldn't suggest the author make any changes
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
It's slow, truncates lines, and doesn't work well at all with less. It's almost like Pottering created it so that pulseaudio wouldn't be his worst program anymore.
opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup
Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler
The preferred journald way of "fetch your logs periodically" wouldn't pass any audit and doesn't work with any kind of log processing software that people use.
journalctl -b | lnav
or use -f instead of -b for follow instead of everything since boot. Now you have a colourised journal and the power of lnav.
Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector)
Even ChromeOS uses it, even on devices that still use Upstart.
If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
It made so many common things I had to do so as a sysadmin super straightforward.
That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
So, systemd change is a pain for me.
I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence:
https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd
https://news.ycombinator.com/item?id=42918448
https://news.ycombinator.com/item?id=42889792
(I reposted my comment from previous submission)
The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better.
I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved.
Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though.
For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.
They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now.
Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.