I still remember when it was the norm for features like this to be entirely implemented in the EC and behaved sanely in any OS, even one that doesn't know about ACPI. For example, on old Thinkpads you can adjust the screen brightness, audio volume, and turn on/off the keyboard backlight in DOS. No drivers needed at all. The outliers were the stupid designs that would overheat and shut down if you left the laptop sitting in the BIOS setup screen for too long, because the fans would stay off. Several decades later, ECs have gotten much bigger flash and RAM, and BIOSes have bloated several times, yet modern laptops need an OS and a massively complex driver loaded just to do something trivial. I recently discovered on a new Thinkpad that the screen brightness was not adjustable in the bootloader, or even early in the OS boot, and that GPU drivers were necessary for that function. A regression which by any other name would just be by design. IMHO this is the increasing fragility leading to the closing of the PC ecosystem that MS ultimately wants.
pzs · 17h ago
This trend of overengineering is apparent now in cars, too. An innocent failure, like a headlight going out can turn into a major systemic issue, like the engine refusing to start through a chain reaction inside an inadequately tested software control system.
I wonder if this is a one-way street, that is, if a realization will come at some point that simple solutions to simple problems can be more robust...
masklinn · 15h ago
> This trend of overengineering
I'd dispute it being over-engineering: media keys tend to control a mix of hardware and software (OS) features (looking at asus keyboards on the internet I see audio volume, mic mute, fan speed / perf governor, display off, brightness, projection mode, touchpad off, sleep, and airplane mode).
Given this, an OS driver is a requirement, and the OS further needs to access the hardware for obvious reasons.
This means you can either implement everything uniformly in driver (just bouncing from the interrupt to a hardware operation in the case of hardware features), or you can mix the implementation between firmware and driver.
Unless you have a very good justification to do so (which I'd very much dispute the existence of for gaming-oriented ASUS laptops) the latter seems like the over-engineering.
jclulow · 16h ago
I think in many respects these problems are actually _under_-engineering. It's possible to treat software as an artefact with a measurable level of quality, and to use frankly not especially ambitious tools (programming languages with memory safety and rich type systems, unit and integration tests, etc) to build them. It's also possible to have a strong sense of user experience and taste as far as what makes a product, not just a pile of parts.
But you have to take software seriously as something that can improve a system, not just a cost centre to be minimised where possible, and an embarrassing source of problems that will ultimately end up in the newspaper or worse.
userbinator · 17h ago
Some of the "proudly-open" laptops have open-source EC firmware. I don't have one and haven't looked deeply enough to know, but maybe they have these features sanely implemented there.
On the other hand, I'm not as optimistic about open-source BIOSes like Coreboot, whose only reason for existence seems to be "it's open-source!" --- that project has been around since the last century, yet still lacks any actual GUI/TUI for configuration, like any other BIOS has had since the late 80s.
mjg59 · 14h ago
The UI is a payload issue, not a Coreboot issue - various vendors ship Coreboot based firmware with a configuration interface, usually based on the Tiano payload. But for my EC issues I simply took the approach of reverse engineering the EC firmware, binary patching it, flashing that back, and getting on with life. Skill issue.
StopDisinfo910 · 12h ago
> I simply took the approach of reverse engineering the EC firmware, binary patching it, flashing that back, and getting on with life. Skill issue.
There is no simply here.
You can’t list a litany of niche skills before implying that’s just life and it’s everyone fault they don’t have the time and knowledge to just, you know, casually reverse engineer and patch a binary.
sevg · 9h ago
It was a sarcastic joke ;)
StopDisinfo910 · 6h ago
Hard to tell in writing. Still not convinced.
mjg59 · 14h ago
Yeah, no drivers required, merely the ability for userland code to smash hardware by hand. That's a terrible situation. You want to adjust screen brightness on a Thinkpad right now? Write to the appropriate control registers in the GPU to send commands over the relevant eDP sidechannel and you can do that, except any modern operating system will prevent you from doing that because you're going to be racing against anything else that's trying to use the same index/value interface, and now you tried to adjust the brightness but actually changed some other value and now your screen is displaying garbage. You know how your brightness hotkeys used to work? They generated system management interrupts, which caused the CPU to STOP EXECUTING THE OS and EXECUTE OPAQUE CODE IN A HIDDEN AREA OF RAM, and THAT FUCKING SUCKED. Now they simply tell the OS that someone hit the key, and it's up to the OS to hook that up to actually doing something as a result. Is the user experience worse? Eh yeah in some ways, but if anyone cared enough we could make it better. Is the past a better place? Fuck no. In the past HP managed to screw this up such that if the hotkey interrupt ended up being serviced on CPU 1 it would restore the CPU registers on CPU 0 instead, and you'd crash immediately (Windows handled all of these on CPU 0 so it worked by accident).
Simple interfaces were fine when we didn't have pre-emptive multitasking, and didn't have SMP. Life is better now. Having mediated interfaces where we can ensure that anything accessing the same hardware is doing so in a controlled manner is a good thing. And, counter to your claim, this is a space where vendors do stuff in spite of Microsoft in ways that make it harder for Microsoft to provide a well-defined user experience - this is literally vendors trying to differentiate within the space Microsoft allows, not something that allows Microsoft more control.
userbinator · 14h ago
Yeah, no drivers required, merely the ability for userland code to smash hardware by hand.
Brightness control has always been in ring 0, or -1 when it was still exclusively SMM.
I don't know what you're on about. What "FUCKING SUCKED" about not having to worry about what the OS does, if it does anything at all? The OS can poke the EC if it wants to adjust the screen brightness, fan control, or whatever else. Otherwise the EC takes care of everything.
Windows handled all of these on CPU 0 so it worked by accident
...and if it was handled by SMM, Windows doesn't need to care at all!
Then again, I'm not surprised at your misdirection and half-truths given you've been essentially shilling Treacherous Computing and Restricted Boot among other hostilities.
Edit: and as if destiny itself wanted to show why needing an OS and accompanying mess of drivers to do something simple is a bad idea, especially for Asus products in particular, this shows up right on cue: https://news.ycombinator.com/item?id=43951588
mjg59 · 13h ago
For most of the past 20 years, brightness control has been ring 0 mechanism, ring 3 policy. ACPI provides a mechanism for the OS to interface with the EC in a way that aligns with any SMM that's going on, and where we are now is just fundamentally better than the halcyon DOS days you're pining for. It also means we don't need to involve another computer in here, which is what the EC is.
> Otherwise the EC takes care of everything.
See, that's just not possible. Modern displays have the backlight integrated into them, and the control is over something that's roughly i2c over eDP. But the same i2c channel needs to be accessible to the OS, and if the OS and the EC try to access that at the same time then things will break. So you need some kind of locking, and that's utterly impossible if hitting a key just jams you into SMM and smashes some bits over there - you might do that in the period between the OS writing an index and writing a value.
> ...and if it was handled by SMM, Windows doesn't need to care at all!
The behaviour of the SMM handler is defined by the platform vendor, and if the vendor ends up only testing against one OS then they may embody assumptions about that OS. In this case they assumed that SMM could only be triggered when running on CPU 0, which was true in the case of Windows and not in the case of Linux. This isn't about Windows needing to care, it's about SMM being a mechanism for vendors to accidentally assume that what Windows does is universal.
> Then again, I'm not surprised at your misdirection and half-truths given you've been essentially shilling Treacherous Computing and Restricted Boot among other hostilities.
I am arguing that it is better that this code lives in visible space, executed in the context of the OS, and subject to security boundaries that the OS imposes. You appear to be arguing for opaque code running on a separate microcontroller, mediated by code running on the main CPU, but which stops the entire OS from running while it executes, and which is hidden from the OS. Which of these sounds better for the user?
wizzwizz4 · 6h ago
> Modern displays have the backlight integrated into them, and the control is over something that's roughly i2c over eDP.
Maybe that's a bad idea? You've clearly outlined some problems it causes.
mjg59 · 13h ago
And in response to your edit: do you think putting this code in hardware is more or less secure than having it in the OS because if it were in hardware all operating systems would be fucked while having it at the OS level means that only Windows is fucked
5- · 15h ago
interestingly enough, when the 51nb guys set about recreating a classic thinkpad with newer innards -- i have an x2100 which is x200 with 10th gen intel motherboard -- they've chosen to replicate this aspect faithfully.
brightness control, thinklight, etc. all work without any os support. this (and many other nice touches) was an unexpected surprise from an entirely hobbyist hack.
mjg59 · 14h ago
Eh other than the Thinkvantage button not working, and various other failings. We ended up with various reverse engineered patches to enable those again.
5- · 13h ago
i have no idea whether i'm using your patches -- thanks anyway!
this has been my daily driver for years and i'm looking forward to future upgrades (there seems to be a meteor lake motherboard in the works).
matheusmoreira · 11h ago
At least he managed to make it work in the end... Dumped my laptop's ACPI tables in order to implement a driver and when I finally managed to decompile the WMI stuff I discovered it was stub code. Didn't find a way to intercept ACPI calls from the manufacturer's software either.
nosrepa · 10h ago
Thinkpads didn't have keyboard backlights until the 2010s.
yencabulator · 7h ago
They did sometimes have an "overhead" ThinkLight, toggled by a magic Fn+PageUp combo just like some backlights.
I think putting in key backlights made the previous spill-proof tray design not viable, so it's not a pure win and I could see why they tried to hang on to the old design.
Old Thinkpads could survive repeated spilling of drinks on their keyboard, the only laptop I've seen to cope with that. There was a tray underneath the keys with a drainage hole all the way through the laptop, that way the liquid didn't reach any zappy bits.
Rediscover · 37m ago
> ...survive repeated spilling of drinks on their keyboard...
Panasonic's "Let's Note" series, too. Their machines from ~2002 were quite impressive to me.
Tsiklon · 29m ago
They still make these! You can pick up a recent model with the circular track pad relatively easily on eBay! 10th Gen Intel at least
EvanAnderson · 9h ago
I fondly remember the "ThinkLight"[0] from my ThinkPad T22. I don't know that I prefer it to a backlit keyboard but it was nice.
Windows makes it much easier for userland policy agents to plug into WMI than to call raw ACPI methods, so WMI became much more popular as Microsoft made it more difficult for vendors to ship random kernel drivers. I didn't write the first WMI integration for Linux (that was Carlos Corbacho back in 2008) but I did write various glue drivers - initially for HP because there was one single hotkey on my laptop that fired a WMI event and which Linux couldn't receive, and later the entire Sony hotkey driver which involved a ludicrous number of layers of indirection. WMI is /somewhat/ self describing (there's a method to call to get a compressed binary of a blob that contains method signatures), but unfortunately there's no requirement that anything they call is documented in any way, so it's often still a matter of reading decompiled ACPI bytecode.
I have taken psychic damage so you don't have to.
matheusmoreira · 10h ago
Would you mind elaborating on the reverse engineering and development process?
I dumped my laptop's DSDT tables and decompiled all the WMI methods. It turned out to be stub code.
diegoperini · 14h ago
> Linux separates things such that I was looking at C files in drivers/platform/x86 and header files in include/linux/platform_data/x86. And the ACPI code lives other places as well. It’s all very orderly, but at times it felt like navigating a grocery store that arranges products in alphabetical order. Logical, but not exactly cozy.
So beautifully put!
mjg59 · 14h ago
The ACPI code largely lives separately in Linux because it was contributed by Intel and is (as far as possible) intended to be dual-licensed GPL/BSD to ensure non-Linux OSes benefit from core improvements.
ac29 · 8h ago
> Battery life is also much better. The fan itself is obviously eating less power. I think the CPU also enters a somewhat lower power state in whisper mode, or it’s willing to sleep a little deeper. It’s not clear everything that changes, and it varies by machine, but that’s the beauty of it.
By shifting to a more power efficient mode, the fan noise should be lowered or eliminated (the fan in my laptop is default off and it rarely comes on at all).
dodomodo · 16h ago
Nice artical! However the supposed rationale for having ACPI WMI is wrong, the real reason is that it allows firmware to (almost) directly expose higher level WMI objects, which is in my opinion clearer and easier to use than the sysfs style used in linux. The problem is that it stopes making any sense when all the supporting infrastructure is not available.
mjg59 · 14h ago
It's easier to use in Windows, but in some cases vendors expose WMI interfaces that allow you to overwrite arbitrary RAM and so Linux doesn't give that to you because it's kind of a huge security violation.
unangst · 9h ago
Thanks for sharing both the tech challenges and solutions so openly.
I still remember when it was the norm for features like this to be entirely implemented in the EC and behaved sanely in any OS, even one that doesn't know about ACPI. For example, on old Thinkpads you can adjust the screen brightness, audio volume, and turn on/off the keyboard backlight in DOS. No drivers needed at all. The outliers were the stupid designs that would overheat and shut down if you left the laptop sitting in the BIOS setup screen for too long, because the fans would stay off. Several decades later, ECs have gotten much bigger flash and RAM, and BIOSes have bloated several times, yet modern laptops need an OS and a massively complex driver loaded just to do something trivial. I recently discovered on a new Thinkpad that the screen brightness was not adjustable in the bootloader, or even early in the OS boot, and that GPU drivers were necessary for that function. A regression which by any other name would just be by design. IMHO this is the increasing fragility leading to the closing of the PC ecosystem that MS ultimately wants.
I wonder if this is a one-way street, that is, if a realization will come at some point that simple solutions to simple problems can be more robust...
I'd dispute it being over-engineering: media keys tend to control a mix of hardware and software (OS) features (looking at asus keyboards on the internet I see audio volume, mic mute, fan speed / perf governor, display off, brightness, projection mode, touchpad off, sleep, and airplane mode).
Given this, an OS driver is a requirement, and the OS further needs to access the hardware for obvious reasons.
This means you can either implement everything uniformly in driver (just bouncing from the interrupt to a hardware operation in the case of hardware features), or you can mix the implementation between firmware and driver.
Unless you have a very good justification to do so (which I'd very much dispute the existence of for gaming-oriented ASUS laptops) the latter seems like the over-engineering.
But you have to take software seriously as something that can improve a system, not just a cost centre to be minimised where possible, and an embarrassing source of problems that will ultimately end up in the newspaper or worse.
On the other hand, I'm not as optimistic about open-source BIOSes like Coreboot, whose only reason for existence seems to be "it's open-source!" --- that project has been around since the last century, yet still lacks any actual GUI/TUI for configuration, like any other BIOS has had since the late 80s.
There is no simply here.
You can’t list a litany of niche skills before implying that’s just life and it’s everyone fault they don’t have the time and knowledge to just, you know, casually reverse engineer and patch a binary.
Simple interfaces were fine when we didn't have pre-emptive multitasking, and didn't have SMP. Life is better now. Having mediated interfaces where we can ensure that anything accessing the same hardware is doing so in a controlled manner is a good thing. And, counter to your claim, this is a space where vendors do stuff in spite of Microsoft in ways that make it harder for Microsoft to provide a well-defined user experience - this is literally vendors trying to differentiate within the space Microsoft allows, not something that allows Microsoft more control.
Brightness control has always been in ring 0, or -1 when it was still exclusively SMM.
I don't know what you're on about. What "FUCKING SUCKED" about not having to worry about what the OS does, if it does anything at all? The OS can poke the EC if it wants to adjust the screen brightness, fan control, or whatever else. Otherwise the EC takes care of everything.
Windows handled all of these on CPU 0 so it worked by accident
...and if it was handled by SMM, Windows doesn't need to care at all!
Then again, I'm not surprised at your misdirection and half-truths given you've been essentially shilling Treacherous Computing and Restricted Boot among other hostilities.
Edit: and as if destiny itself wanted to show why needing an OS and accompanying mess of drivers to do something simple is a bad idea, especially for Asus products in particular, this shows up right on cue: https://news.ycombinator.com/item?id=43951588
> Otherwise the EC takes care of everything.
See, that's just not possible. Modern displays have the backlight integrated into them, and the control is over something that's roughly i2c over eDP. But the same i2c channel needs to be accessible to the OS, and if the OS and the EC try to access that at the same time then things will break. So you need some kind of locking, and that's utterly impossible if hitting a key just jams you into SMM and smashes some bits over there - you might do that in the period between the OS writing an index and writing a value.
> ...and if it was handled by SMM, Windows doesn't need to care at all!
The behaviour of the SMM handler is defined by the platform vendor, and if the vendor ends up only testing against one OS then they may embody assumptions about that OS. In this case they assumed that SMM could only be triggered when running on CPU 0, which was true in the case of Windows and not in the case of Linux. This isn't about Windows needing to care, it's about SMM being a mechanism for vendors to accidentally assume that what Windows does is universal.
> Then again, I'm not surprised at your misdirection and half-truths given you've been essentially shilling Treacherous Computing and Restricted Boot among other hostilities.
I am arguing that it is better that this code lives in visible space, executed in the context of the OS, and subject to security boundaries that the OS imposes. You appear to be arguing for opaque code running on a separate microcontroller, mediated by code running on the main CPU, but which stops the entire OS from running while it executes, and which is hidden from the OS. Which of these sounds better for the user?
Maybe that's a bad idea? You've clearly outlined some problems it causes.
brightness control, thinklight, etc. all work without any os support. this (and many other nice touches) was an unexpected surprise from an entirely hobbyist hack.
this has been my daily driver for years and i'm looking forward to future upgrades (there seems to be a meteor lake motherboard in the works).
I think putting in key backlights made the previous spill-proof tray design not viable, so it's not a pure win and I could see why they tried to hang on to the old design.
Old Thinkpads could survive repeated spilling of drinks on their keyboard, the only laptop I've seen to cope with that. There was a tray underneath the keys with a drainage hole all the way through the laptop, that way the liquid didn't reach any zappy bits.
Panasonic's "Let's Note" series, too. Their machines from ~2002 were quite impressive to me.
[0] https://www.thinkwiki.org/wiki/ThinkLight
I have taken psychic damage so you don't have to.
I dumped my laptop's DSDT tables and decompiled all the WMI methods. It turned out to be stub code.
So beautifully put!
You can get 90% of the way here on an Intel CPU by setting the EPB. There are a number of ways to do this on Linux: https://wiki.archlinux.org/title/CPU_frequency_scaling#Intel...
By shifting to a more power efficient mode, the fan noise should be lowered or eliminated (the fan in my laptop is default off and it rarely comes on at all).