Car companies are in a billion-dollar software war

I worked for a $ ~billions revenue software storage vendor who had the exact same issue (excessive logging wearing out under-spec'd flash drives).

Everyone has spent a mountain of money on this problem but spent it all assiduously avoiding addressing the root causes.

Car companies realized early on they could outsource component development and production to 3rd parties and they could make them bid each other to further lower the prices.

So their platforms were optimized to be able to swap component vendors very easily (to achieve lowest costs).

Of course the vendors are not 100% interchangeable and building a platform to accommodate everyone has to make sacrifices.Aka target the least common denominator across all vendors.

Each car has dozens to 100+ ecus, written in different languages, by different teams, different requirements, and different companies. Some are proprietary. Ford can’t just tell Bosch, hey your abs module needs to now integrate with our api, multiplied by 100+ companies. The legacy car makers need to revisit everything, and move most of it in-house.

With that kind of adversarial relationship, you are never getting anything above the barest minimum of competence.

It's puzzling to see this push for general computing on devices that need to far outlast the typical release cycle of GC devices. There is nothing good that can come out of installing Android in your TV, fridge, let alone a - for fuck's sake! - a car.

If your consumer hardware needs to last for decades, then the core functionality and automation should be provided by sturdy embedded computers that are self-contained and do not require any kind of network access or regular updates, while the general computing functions functions should be provided by the user's own device or a replaceable/upgradable computer with a standardized interface.

My own car is too old for Android Auto, but I sometimes drive a car that's from 2017 or so, and Android Auto works just fine on it, it's a pleasure to use (with the caveat that the phone has to be plugged in the USB port, wireless came later). So to me it seems like it always worked well.

I guess I'm in the minority, then, but as a data point: I own a VW ID.4 and I'd pay significantly more to get software that isn't such a burning dumpster tire fire.

And no, the excuses provided in this thread don't cut it.

To be clear: it doesn't even annoy me anymore that the infotainment is slow and crappy, I've gotten used to it and I just never use it. But I when I want to close both windows and I press two buttons simultaneously, I would like both windows to go up, not one up and one down, as it sometimes happens.

The crappiness of the software in this car is mind-boggling and it cannot be excused: most of it is incompetent and sloppy programming.

I would pay more for a car where the software department is somewhat competent and knows what they're doing.

Now of course tesla/musk are destroying themselves through various idiotic actions. Sales are dropping through the roof. But the technical quality of the software ecosystem (car, web, app) is still better than all the incumbents. Think about Rivian getting a billion dollars from VW for their much better ECU and and software integration, for example.

I feel like Rivian is almost as good as tesla. Tesla still has all that, even as the company is in awful shape sales wise. Lucid seems to be better than the legacy auto, but I haven't looked into it as closely.

You don’t know that vertical integration will guarantee that you’re more competitive, and the investment you need to make before you see a return is beyond 5 years. That’s not an easy bet to make. It looks obvious in retrospect, but it’s really not.

It requires quite a bit of in-housing that many of these teams aren’t yet well-versed in, so as you vertically integrate you’re also disrupting your internal structure while adding new people. It’s a lot to take on. Meanwhile, there are other long term plans underway already.

Unless the top of the company comes in and starts chopping every head that gets in the way of the new paradigm then it just ends up in locked up meetings for years of people that don't want to change.

Electronics integration isn't the problem, the people currently there are.

https://www.hwe.design/product-development-process/developme...

For components that have many components or complex requirements, or are part of more complicated systems, this takes longer. Cars have a design cycle that's many years long - 5-6 years would be a decent ballpark. That's due to the complexity of the product, complexity of the supply chains and tooling, requirements, and scale.

I will never want to listen to the radio. I would love to remove radio as an option. I would love to have no fallback as an option. But no, the car just f-n loves the radio and will not stop trying to force it on me.

Oh yeah, and the radio is buggy and could get stuck if I tune into the wrong station. https://www.bbc.com/news/technology-60333765.

This car definitely tries too hard to be smarter than it is. There's all sorts of exceptions that keep the doors from auto-locking when I walk away, and I would turn all of them off, but I can't. Walk away too fast? doesn't lock. Open the rear? won't auto lock. Car just doesn't feel like it? doesn't auto-lock.

And god forbid you hit the unlock button when the passenger has already unlocked it. Anxious beeps from the car for several solid seconds. That is not an error condition!

Performance and reliability have been great though. They just need to stop trying to be smart. They're not.

I even adore the scroll wheel and wish it could be in any car I own in future. Yeah it takes slightly longer to do certain actions in CarPlay, but I can do it so much more safely than I could in the Civic I had before. The infotainment boots basically instantly; as you mentioned CarPlay starts itself, and the patronising-but-mandated “don’t use this in motion” warning dismisses itself. In the Civic I would be half way down the road already by the time it booted, blindly prodding at the screen to try to dismiss that warning so I could pause the podcast that started playing itself because I plugged my phone in.

And, while my 2022 car predates the stupid auto-re-enabling ADAS requirement in Europe, the 2024+ models have single button deactivation. I dunno how, cause it’s supposed to require a minimum of two presses legally, but it sure makes me wanna stick with Mazda.

However that makes the upcoming 6E that much more disappointing. They’ve partnered with a Chinese manufacturer, I assume because they don’t have an EV platform of their own ready yet. Looks fantastic from the outside, but the inside is a sea of touch screens with barely a physical control in sight.

When looking at who is doing it right, I wouldn’t put Mazda on a pedestal. They simply are behind the competition.

So I only use Google maps with Android Auto now, but cannot put the turn by turn display on. Also, who knows what telemetry Mazda is sending home on me without me knowing or wanting them to. Probably selling it to data brokers.

* Bed size is just five feet

* Towing capacity is just 1000 lbs

* Not AWD

None of these can be retrofitted after the sale.

Where I live, it'd struggle to be called a "truck" with these limitations.

As an aside a lot of people like to levy criticism on the infotainment screens which I think is very well deserved, but then people text and drive, watch YouTube videos, and do all sorts of crazy things too.

Instead of levying criticism on these distractions (let’s include billboard too) we should instead focus on just reducing car usage since we won’t stop people from being distracted.

The safest car is the one in your garage.

That’s…terrible

It does cost time/money to integrate, like any feature

Tracking, phoning home (with related privacy issues), etc:

* https://arstechnica.com/cars/2024/09/flaw-in-kia-web-portal-...

* https://arstechnica.com/information-technology/2019/10/five-...

> support wireless CarPlay and android auto

Removing LTE doesn't cost me real-time traffic updates because (preferred maps app) is running on my phone which already has LTE. Streaming media? The media is being played from my phone or streamed via my phone, which already has LTE. I'm not sure what "remote controls" are in this context? Letting me set the A/C fan to high from Internet (almost certainly via a browser or app running on... wait for it... my phone)?

We've already paid for the LTE modems and app integration on the phone side of things, don't need to pay for it a second time on the car side or have to deal with the vehicle manufacturer's terrible implementations of navigation apps and media streaming services or yet another vendor collecting telemetry about me and reselling it to whoever wants to pay.

If you're gonna build that crap in at least go back to a standard-sized replacable module.

Everything makes it beep. Beeps for “you will die now” are similar to “you put me in gear”.

There’s one exception: For many reasons, it turns off one-pedal driving. When it does that and is unexpectedly accelerating into cross traffic, it does not beep (until the collision alarm sounds, presumably, ask me if it kills me…)

I disable the "land assist" every time (which often tries to steer me into wildlife or other cars and was clearly not built for use on a single track country roads with hedges and random verges), but this was the first time in 3 years that the "front assist" caused problems.

If that's "high quality", I dread to think what low quality would be.

It states that consumer reports, (a for profit company providing independent reviews, and not a regulatory body) said the Model 3 stopping distance was not good. Allegedly due to a “bad ABS calibration”. Tesla released an OTA SW update.

Why wasn’t the bad calibration and degraded performance caught by regulators testing automobile safety standards?

The article also posits that this ability to make OTA updates expands the (IMO very very bad) SWE perspective that “it’s OK to ship unfinished and buggy products” into safety critical systems.

That is a piece of paper.

> Software for steering or braking systems is of high quality.

There's literally no way for me to know that before I trust my life with it.

So move to one of the 2 or 3 cities in the US that have Waymo?

Is this then logic that gets airlines to buy from The Boeing "Are door plugs supposed to stay in?" Company?

Now sure, if you’re looking for 500k+ jobs, embedded isn’t the area to be in, unfortunately. But I prefer low-stress, fun-environment embedded jobs, and don’t mind trading off salary for that. Different strokes.

Alot of embedded stuff is outsourced and doesn't want to waste the computing power for stuff like stack canaries. I recall the following from making a tool for a dlink? router?

//Reads a file name foo ReadFilePath() { // Get file name // TICKET 21321: Fixed crash by increasing buffer size char FilePath[100]; ReadFileName(&FilePath); }

It sticks out to me, since the crash was clearly from a buffer overflow, and they had this documented in the source code that increasing the buffer size fixes it. What they didn't realize was that the bug would still happen and you could get a buffer overflow from this and do whatever you wanted. This is the level of programmer you're dealing with who's writing embedded software in an overseas sweatshop. And the talent isn't even there domestically since they're severely underpaid compared to someone writing simple javascript.

This is an industry that is about as far from the light of science & enlightenment as it is possible to get, ensnared as deeply in the entangling anti-human anti-science Intellectual Property qualgmire-hell as can be got. Oh sure plenty of science goes it! It's fantastically interesting & technical! But aside from some Application Notes write-ups trying desperately to help move the practice along, move it out of jank, knowledge goes in, but it doesn't ever come out! There's such a lack of peershios with which to practice science, to report your findings to, to replicate works on.

The software world talks about its patterns and practices. The biggest industries on the planet are building software like wild AND are mad into open source. But... computer engineering is the shadowland, where no talk nor victories that happen there are allowed to be shared, where nothing escapes confinement. What a fucking plagued awful land of people unable to ever do the right thing, unable to bring their work out of the dark & into real civilization.

I did some initial requirements work on a system to monitor continuous-web papermaking machinery; the line had to be stopped, physically and completely, within 100ms if anything went wrong, because an uncontained web of paper can literally cut people in half. They wanted, in order to be able to hire, to use one of the embedded flavors of a well-known consumer-grade OS, and I had to prove to them that there was no way to make any of them safe, at any cost. And they knew their hardware, because they had built it themselves.

The absolute last resort is a watchdog timer that hits the reset button if N milliseconds go by without the software telling it it's okay. This is what you have to implement if you are dealing with buggy and undocumented hardware -- as, all too often, you are. Sometimes you can get some doco for $ and an NDA, but then in order to get the real doco it is much more $$$ and a much tighter NDA, and the existence of that option is not even divulged until after things have already gone very far south.

If it were only a matter of reading the top-level doco for this or that chip, there would be no issue.

Maybe things just really suck for embedded in the states? But since my last year of university I’ve been inundated with recruiters for embedded positions, and I’ve never had a problem finding work. ~75th percentile in salary alone for software engineers in my area, ~55th-60th for Canada. I make more than every JS developer I know who graduated with me, except for the ones who moved to Seattle, Vancouver, or the Bay.

[1] https://wikipedia.org/wiki/Firewall_(engine)

For what it's worth, I work in this industry and the general rule of thumb is that every increase in validation from QM (standard quality) up to the various levels of safety critical code has up to 10x the cost per line of code of the previous level.

Sure someone in that situation could also "just cut brakes or put in a pipe bomb" but car theft is a lot more common than assassination, at least where I live.

See [1] from 2023, where popping the headlight gives access to the bus. Lack of internal security then gives a way to steal the car.

The threat just isn't the same as the one you are modeling.

Security will come eventually, if only to prevent bad publicity.

[1]: https://arstechnica.com/information-technology/2023/04/crook...

ETA: Just as the sibling says...

Is this not just Android Automotive? A lot of Volvos use it, it’s a lower-level OS type thing that sits below Android Auto or CarPlay.

When is this actually useful? In the ~12 years I’ve been driving, I’ve never needed to know the fuel level of a car when I’m not in it. I guess maybe if I’m planning a road trip and need to know if I’m going to have to stop for gas before I leave? But I’ll figure that out when I get in to leave and I’m probably not leaving with <10 minutes of margin.

> locating my car

Again, never once have I not known where my car was. I think my phone keeps track of where I park too already? But I’ve never needed that feature. I guess if it’s stolen and the thieves don’t know how to disable this, it could potentially be useful for insurance/police.

> telling me about open windows, giving me the option to close them remotely

This could be useful. I’ve never left windows open by accident before, but I have left them open on purpose - if there were an automatic notification when this happens, I’d probably just eventually turn it off to reduce the irritation from false positives, and then not be notified if I ever left them open by accident.

> remote door un/locking

I had a Lincoln that had this feature, while I was working as a reverse engineer/pentester. Took me ~45 minutes to be able to send an unlock request to the car, unauthenticated, and have it open the doors, over the internet. Pretty sure that’s never been fixed (at least, it hadn’t been when I got rid of the car - model year 2016, which was identical to the 2013s, and I got rid of it in 2022). Needless to say, not a fan of that kind of “feature.”

I could see charging ETA being useful if multiple people are using the same car and for whatever reason can’t communicate that sort of thing with each other, and don’t have a feel for how long charging takes. (I’ve never owned an EV, but I imagine that you plug it in when you get home, and then it’s ready for you in the morning, so I don’t really know what the use case for knowing the ETA is in that case. Maybe if you’ve been driving around all day and need to make a long drive in the evening? I still assume you’d know how long it’ll take to charge when you plug it in though. And if you're at a fast charger, don’t they have a screen that gives you the ETA when you plug it in? I’ve only used one before, but it did that, and it was accurate to within 30 seconds, so I’m not too sure how useful it would be to have the ETA on your phone in that case either.)

I would also guess (completely un-informedly) that because the simple (and probably correct) answer is very difficult, a lot of companies are trying to avoid it by doing things that are more complicated but also easier. And because they are more complicated, it is not immediately obvious why they won't work....but they won't. Which is resulting in the repeated failures.

And so did traditional manufacturers, they just had the benefit of being able to phase it in if they so chose. Or they could have done a hard cutover, either way, the failure is on them for ignoring the benefits of the Software Defined Vehicle discussed in the article.

> It's unclear what getting one of the android automotive developers would do for you.

Do they do vehicle control systems or just infotainment?

> they needed to use Unreal Engine to show you what your own car looks like and spin it around in. 3-D deserves some kind of salesmanship Nobel prize.

I mean that's exactly the kind of thing that makes Tesla fanboys rave endlessly about their car. It just needs to be decoupled from the actual software system, like any UI.

OTOH, ABS and ESP systems can achieve similar or even better results with less complexity because motor torque control is inherently low-latency, which can also complement brake deployment (hydraulics is not as well behaved as e-motor).

You do get rid of emissions control and tiny little sensors / flap actuators sprinkled all around the engine bay, so yeah, probably overall still a simplification win, but I doubt you can get very far without "massive amounts of [Mechatronics] engineering".