Reads very similar to some blackouts we had in Australia. Weakly connected grids with vast geographical distances leading to oscillations that took down the grid.
You can also read numerous stories of how Australia's lithium ion grid storage systems have prevented blackouts in many cases. https://www.teslarati.com/tesla-big-battery-south-australia-... The fact is that the batteries responsiveness is the fastest of any system at correcting gaps like this. 50/60hz is nothing for a lithium ion battery nor are brief periods of multi-gigawatt draw/dumping as needed.
> nor are brief periods of multi-gigawatt draw/dumping as needed.
Actually this is typically an issue for grid batteries.
Spinning generators can easily briefly go to 10x the rated current for a second or so to smooth out big anomalies.
Stationary batteries inverters can't do 10x current spikes ever - the max they can get to is more like 1.2x for a few seconds.
That means you end up needing a lot of batteries to provide the same spinning reserve as one regular power station.
AnotherGoodName · 1h ago
Collectively Australia's battery storage systems will be able to beat any single power plant for peak output in Australia once fully built out based on pure numbers. But for these sorts of grid oscillations the more important thing is the localization of generation. Which obviously favours the batteries over large centralized power stations in any case.
ajross · 38m ago
That... doesn't sound correct. Inverters are the cheap part, you can literally wire as many as you want in parallel. Batteries have immense power availability, with most chemistries you can trivially deliver the entire capacity in half an hour or so (more like 5 minutes with lithium cells).
Basically I'm dubious. I'm sure there are grids somewhere that have misprovisioned their inverter capacity, but I don't buy that battery facilities are inherently unable to buffer spikes. Is there a cite I can read?
AnotherGoodName · 28m ago
Agreed. The relatively small battery substation linked above can output 2GW of equivalent inertia generation (a measure to align batteries to inertial power systems) when needed. That's an entire power station they can match for short periods of time. Link: https://www.energymagazine.com.au/sa-approves-world-first-ba...
Australia's largest power plant has 2.9GW of inertial generation assuming all generators are running at 100%. As in the small battery substation alone comes close to the countries largest power station. I'm not sure where the idea that lithium ion can't dump power quickly comes from. They are absolutely phenomenal at it. Australia's building dozens of these substations too since they are so cheap and reduce overall power costs. It's a win from all points of view.
probablypower · 27m ago
You can google "system inertia" as a starting point.
rcxdude · 8m ago
This is what batteries can provide very well.
eldaisfish · 5m ago
it is technically correct, but so are you.
More inverters in parallel will achieve the same end goal - fast frequency response.
giantg2 · 1h ago
Seems like pumped hydro offers a nice compromise.
floatrock · 2m ago
Most economically-suitable locations for pumped hydro have been built out already.
You can always use a ton more concrete and force new locations, but the best locations have already been utilized and scaling law of batteries has brought them to the point where they're more competitive than new hydro for this kind of use.
That equivalent inertia can only be done for short periods but that's exactly what grids need in stability - there's generally no lack of total generation, just a need to jump in and smooth out spikes.
You can't build a dam for that price, nor could you do it in under 100 days from contract signing as that battery was built. Batteries are definitely the answer here. The 'more spinning mass' answers don't make sense since Australia literally solved the above problem in a much cheaper way already.
giantg2 · 1h ago
You don't need to build a dam, you just need the pipes and pumps for an existing dam (or elevated natural basin).
diggan · 1h ago
Seems the money shot starts at page 131:
> The ultimate cause of the peninsular electrical zero on April 28th was a phenomenon of overvoltages in the form of a "chain reaction" in which high voltages cause generation disconnections, which in turn causes new increases in voltage and thus new disconnections, and so on.
> 1. The system showed insufficient dynamic voltage control capabilities sufficient to
maintain stable voltage
> 2. A series of rhythmic oscillations significantly conditioned the system, modifying
its configuration and increasing the difficulties for voltage stabilization.
If I understand it correctly (and like software, typical), it was a positive feedback-loop. Since there wasn't enough voltage control, some other station had to be added but got overloaded instead, also turning off, and then on to the next station.
Late addition: It was very helpful for me to read through the "ANNEX X. BRIEF BASICS OF THE ELECTRIC SYSTEM" (page 168) before trying to read the report itself, as it explains a lot of things that the rest of the report (rightly) assumes you already know.
leymed · 34m ago
I think your interpretation is correct. The voltage control is done at the high level of the grid, meaning the control covers bigger generation stations and major substations. Even if it’s small generator, rotating machinery, you won’t have strict voltage control other than its own AVR. The problem I see here is that we embed smaller individual generations at the lower level, where they pump the generated power to the grid at the medium voltage level. When you have majority of your generation at this level, you won’t have strict control over voltage and even frequency, I assume. I’m still digesting the report, but what I am after is whether they really neglected it and if it is not possible to do voltage control with 50% generation coming from renewable and through medium voltage level, aka lower level.
fuoqi · 2h ago
>Given the timing of this drop in solar generation, in the middle of the production ramp-up and without any meteorological phenomena to explain it, the most plausible explanation is that it is due to market reasons (prices)
>These changes in production can be significant (if the price signals from the markets are sufficiently strong) and affect the energy flows in the networks and the stresses in the nodes
>Regarding the correlation between changes in generation and voltage: if the generation operating at power factor falls (with the existing regulation, the renewable generation), there is a decrease in the reactive energy absorbed by these installations (since it is reduced proportionally to the reduction in generation). Additionally, as the energy transmitted by the networks decreases due to this reduction in generation, the capacitive effects of the electrical
circuits increase as they become more discharged, which causes an increase in reactive energy. Both effects (higher reactive power production by the circuits and lower reactive power absorption) push the voltages upwards.
As expected, renewables played the leading role in destabilization of the grid which led to the collapse. Of course, these instabilities could be worked around on the grid level using various (not cheap) measures, so you may argue that it's not renewables' "fault", but the fact still stands: above the certain threshold of renewable generation the current grid architecture becomes increasingly unstable.
What I'm reading from that quote is that the issue wasn't renewables as such, but an issue of power generation reacting too quickly and too intensely to price fluctuations. "Renewables" only matter insofar as they're the sort of generation that, under the current regulatory regime, get to react to those pricing changes.
baq · 1h ago
The report goes to great lengths to avoid certain words or phrases. The market failed here, it didn’t price in risk of grid collapse correctly.
But the quote literally spells out it was market forces, not some instability in solar generation?
Your other comment probably got flagged because it started with a huge straw man and had multiple unwarranted jabs in it.
fuoqi · 2h ago
Temporary negative prices have been caused by the renewable generation which exceeded the grid demand at the time, which then evolved into the nasty feedback loop caused by the reaction of renewable generation to those conditions. You simply do not get such situation with traditional generation, it's the direct consequence of the intermittent nature of renewables and its high ratio in the total generation.
Also, have you read after the market part? Please watch this video https://www.youtube.com/watch?v=7G4ipM2qjfw if the last quote is gibberish to you. It discusses somewhat different issues, but the point still stands.
pkilgore · 1h ago
Where is the market for someone to get paid to pump water into a reservoir and let it fall down later for $$$?
plorg · 10m ago
If you read the report there was a significant amount of solar being produced at low prices and being pumped for storage. Further, the pumped hydro is the first load to be disconnected to balance demand on their system.
pdpi · 25m ago
That market exists, but the window of time here is like twenty minutes. Pumps have inertia and take time to spin up, you can't HFT load and generation.
eldaisfish · 16m ago
you are correct, but your analysis is not popular here. You will soon be presented with several reasons as to why renewable energy is not the problem and how batteries are the one true solution to these problems.
The reality is that electricity is complex and that renewable energy presents a new set of problems, problems to which we do not yet have complete solutions.
shakow · 1h ago
True, but the market moves fast because renewables (or, more precisely, wind & solar) move fast.
There is not much fast trading to be done on a nuke/gas/coal/hydro powerplant ramping up or down, but there is a lot of instability (and thus market volatility) to be found in fast varying solar/wind conditions.
baq · 1h ago
Should’ve said ‘not enough spinning mass’ and it’d be perfectly fine for the politically correct and mean the same thing. This was highlighted as a risk for years and it finally materialized.
philipkglass · 1h ago
According to the operator report linked in another comment by leymed [1], the problem was not a lack of spinning mass (inertia) but voltage instability. From page 16 of the PDF:
The incident was NOT caused by a lack of system inertia. Rather, it was triggered by a voltage issue and the cascading disconnection of renewable generation plants, as previously indicated. Higher inertia would have only resulted in a slightly slower frequency decline. However, due to the massive generation loss caused by voltage instability, the system would still have been unrecoverable.
Obviously I’m as good of a grid operator as I was a stealth bomber expert on the weekend, but superficially that just doesn’t seem right. Maybe I’m underestimating how much spinning mass would be required, but that still qualifies as ‘not enough was present’.
rcxdude · 4m ago
You very much are underestimating it. Spinning mass helps even out very short term fluctuations in supply vs demand. Like on the timescale of tens of seconds, even when the whole grid is spinning mass. Even 10x the inertia in the grid would have maybe bought a few extra minutes, because the problem by the point the grid was collapsing was there were not enough plants online to provide the demand.
(Spinning mass on its own doesn't do much to deal with the voltage fluctuations. It's entirely something that's reactive to grid frequency, which is the most 'global' indicator of supply vs demand in a grid, since it can't fluctuate locally. But voltage and current can vary wildly in different parts of the grid, and required separate management)
eldaisfish · 15m ago
please can you explain what doesn't seem right?
wavefunction · 2h ago
You quoted
>the most plausible explanation is that it is due to market reasons (prices)
Seems to be market conditions or manipulations or inefficiencies in the market.
No comments yet
tofflos · 2h ago
It's a difficult read.
Cybersecurity and digital systems was not the issue but gets thirteen pages of proposed measures. I feel this could have been left out.
Electric System Operation was the issue and gets seven pages of proposed measures.
Individual generators monitor Voltage, Frequency, and reactive power (≈ how much current is out of phase with voltage) to make decisions about injecting more or less power into the network. This is just historically how they've always been doing it.
Due to interactions between different generators, there can be instabilities causing voltage or frequency or reactive power to deviate outside of spec. A simple example might be two generators where one surges while the other drops back, then vice versa. The measurement (by the network operator) of these effects is poor for Spain - shown by the simple example that they have large oscillations that they couldn't explain.
There's path dependent healing and correction of problems by different generators, which overall leads to network stability. However the network operator here is not actually resolving cause and effect, and does not have the insight to manage their stability properly.
In this case you can see them trying a few things to inject changes that they hope will bring stability - e.g. tying many connections hoping that adding generators together into one network will resolve to a stable outcome.
Are there countries that have a better design for their electricity network control systems?
Disclaimer: I don't design electricity networks nor electricity markets. And the above is ignoring loads (loads are mostly less problematic for control than generation).
rcarmo · 2h ago
Page 130 is where the actual human readable summary is. Although the previous pages were pretty detailed in explaining the cumulative instabilities.
Sadly, some news outlets are probably only going to look at the recommendations and read "cybersecurity" and (even though they are common sense recommendations) assume there might be more to say about the matter.
Nextgrid · 9m ago
There's been a shit-ton of misinformation about cyberattacks within the first hour of the outage, and the public were unfortunately very receptive to it, so I guess they're trying to preempt those concerns?
decimalenough · 1h ago
Don't worry, some news outlets will summarize this as "renewables = bad" regardless of what the report actually says.
Ed: Do I need a /s tag here or something? My point was that we shouldn't worry too much about about the presentation of the report, its actual contents will be spun to suit any narrative regardless.
baq · 1h ago
Flame bait journalism is one of the things you can count on in any circumstance. If you replace that ‘renewables = bad’ with ‘politicians = idiots’ OTOH… sometimes the elected representatives should listen to unelected physicists and engineers. Grid stuff is one of those things.
londons_explore · 1h ago
It doesn't look like this report really identifies the root causes...
I would like to see: "We have simulated the complete 200 and 400 kV grid of the iberian peninsula and western europe, and can reproduce the situation that occurred. Any one of the following changes would have prevented the issue, and we suggest implementing them all for redundancy. This simulation will be re-run every day from now on to identify future cases similar incidents could occur"
baq · 1h ago
The engineers knew exactly what’s going to happen, the report is politically redacted. Very unfortunate, but completely expected.
decimalenough · 1h ago
> Non-confidential version of the report
of the committee for the analysis
of the circumstances surrounding
the electricity crisis of the
April 28, 2025
Now I'm curious about what's in the confidential version of the report.
londons_explore · 1h ago
Based on the redacted bits, it is mostly company names and locations.
gred · 1h ago
Why so many pages of "Recommendation: implement multi-factor authentication" and other IT security irrelevancies? Did they need to pad out the number of pages?
diggan · 1h ago
> In the systems with network traffic evaluation probes, no records consistent with unauthorized activity have been observed, such as lateral movements, network traces or file movements for vulnerability exploitation or privilege escalation, among others.
> However, as is common in networks and information systems in any sector, other risks have been identified, such as vulnerabilities, deficiencies or inadequate configurations of security measures, which may expose networks and systems to potential risks, for which a series of measures are proposed.
JanneVee · 1h ago
When skimming through the report I got to think of the oscillation problem in RIP routing protocol. Although it isn't the same thing, but it shows the complexity of the problem to anyone who thinks there is a single solution to it.
https://en.wikipedia.org/wiki/2016_South_Australian_blackout
Completely solved with lithium based grid storage at key locations btw. This grid storage has also been massively profitable for it's owners https://en.wikipedia.org/wiki/Hornsdale_Power_Reserve#Revenu...
Australia currently has 4 of the 5 largest battery storage systems under construction as a result of this profit opportunity; https://en.wikipedia.org/wiki/Battery_energy_storage_system#...
You can also read numerous stories of how Australia's lithium ion grid storage systems have prevented blackouts in many cases. https://www.teslarati.com/tesla-big-battery-south-australia-... The fact is that the batteries responsiveness is the fastest of any system at correcting gaps like this. 50/60hz is nothing for a lithium ion battery nor are brief periods of multi-gigawatt draw/dumping as needed.
There's even articles that if Europe investing in battery storage systems like Australia they'd have avoided this. https://reneweconomy.com.au/no-batteries-no-flexibility-spai...
Actually this is typically an issue for grid batteries.
Spinning generators can easily briefly go to 10x the rated current for a second or so to smooth out big anomalies.
Stationary batteries inverters can't do 10x current spikes ever - the max they can get to is more like 1.2x for a few seconds.
That means you end up needing a lot of batteries to provide the same spinning reserve as one regular power station.
Basically I'm dubious. I'm sure there are grids somewhere that have misprovisioned their inverter capacity, but I don't buy that battery facilities are inherently unable to buffer spikes. Is there a cite I can read?
Australia's largest power plant has 2.9GW of inertial generation assuming all generators are running at 100%. As in the small battery substation alone comes close to the countries largest power station. I'm not sure where the idea that lithium ion can't dump power quickly comes from. They are absolutely phenomenal at it. Australia's building dozens of these substations too since they are so cheap and reduce overall power costs. It's a win from all points of view.
More inverters in parallel will achieve the same end goal - fast frequency response.
You can always use a ton more concrete and force new locations, but the best locations have already been utilized and scaling law of batteries has brought them to the point where they're more competitive than new hydro for this kind of use.
That equivalent inertia can only be done for short periods but that's exactly what grids need in stability - there's generally no lack of total generation, just a need to jump in and smooth out spikes.
You can't build a dam for that price, nor could you do it in under 100 days from contract signing as that battery was built. Batteries are definitely the answer here. The 'more spinning mass' answers don't make sense since Australia literally solved the above problem in a much cheaper way already.
> The ultimate cause of the peninsular electrical zero on April 28th was a phenomenon of overvoltages in the form of a "chain reaction" in which high voltages cause generation disconnections, which in turn causes new increases in voltage and thus new disconnections, and so on.
> 1. The system showed insufficient dynamic voltage control capabilities sufficient to maintain stable voltage
> 2. A series of rhythmic oscillations significantly conditioned the system, modifying its configuration and increasing the difficulties for voltage stabilization.
If I understand it correctly (and like software, typical), it was a positive feedback-loop. Since there wasn't enough voltage control, some other station had to be added but got overloaded instead, also turning off, and then on to the next station.
Late addition: It was very helpful for me to read through the "ANNEX X. BRIEF BASICS OF THE ELECTRIC SYSTEM" (page 168) before trying to read the report itself, as it explains a lot of things that the rest of the report (rightly) assumes you already know.
>These changes in production can be significant (if the price signals from the markets are sufficiently strong) and affect the energy flows in the networks and the stresses in the nodes
>Regarding the correlation between changes in generation and voltage: if the generation operating at power factor falls (with the existing regulation, the renewable generation), there is a decrease in the reactive energy absorbed by these installations (since it is reduced proportionally to the reduction in generation). Additionally, as the energy transmitted by the networks decreases due to this reduction in generation, the capacitive effects of the electrical circuits increase as they become more discharged, which causes an increase in reactive energy. Both effects (higher reactive power production by the circuits and lower reactive power absorption) push the voltages upwards.
As expected, renewables played the leading role in destabilization of the grid which led to the collapse. Of course, these instabilities could be worked around on the grid level using various (not cheap) measures, so you may argue that it's not renewables' "fault", but the fact still stands: above the certain threshold of renewable generation the current grid architecture becomes increasingly unstable.
I wrote exactly about it in this comment https://news.ycombinator.com/item?id=43831523 and its child comment, but the fans of renewables just flagged it.
Your other comment probably got flagged because it started with a huge straw man and had multiple unwarranted jabs in it.
Also, have you read after the market part? Please watch this video https://www.youtube.com/watch?v=7G4ipM2qjfw if the last quote is gibberish to you. It discusses somewhat different issues, but the point still stands.
The reality is that electricity is complex and that renewable energy presents a new set of problems, problems to which we do not yet have complete solutions.
There is not much fast trading to be done on a nuke/gas/coal/hydro powerplant ramping up or down, but there is a lot of instability (and thus market volatility) to be found in fast varying solar/wind conditions.
The incident was NOT caused by a lack of system inertia. Rather, it was triggered by a voltage issue and the cascading disconnection of renewable generation plants, as previously indicated. Higher inertia would have only resulted in a slightly slower frequency decline. However, due to the massive generation loss caused by voltage instability, the system would still have been unrecoverable.
[1] https://news.ycombinator.com/item?id=44360052
(Spinning mass on its own doesn't do much to deal with the voltage fluctuations. It's entirely something that's reactive to grid frequency, which is the most 'global' indicator of supply vs demand in a grid, since it can't fluctuate locally. But voltage and current can vary wildly in different parts of the grid, and required separate management)
>the most plausible explanation is that it is due to market reasons (prices)
Seems to be market conditions or manipulations or inefficiencies in the market.
No comments yet
Cybersecurity and digital systems was not the issue but gets thirteen pages of proposed measures. I feel this could have been left out.
Electric System Operation was the issue and gets seven pages of proposed measures.
https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_Sp...
Due to interactions between different generators, there can be instabilities causing voltage or frequency or reactive power to deviate outside of spec. A simple example might be two generators where one surges while the other drops back, then vice versa. The measurement (by the network operator) of these effects is poor for Spain - shown by the simple example that they have large oscillations that they couldn't explain.
There's path dependent healing and correction of problems by different generators, which overall leads to network stability. However the network operator here is not actually resolving cause and effect, and does not have the insight to manage their stability properly.
In this case you can see them trying a few things to inject changes that they hope will bring stability - e.g. tying many connections hoping that adding generators together into one network will resolve to a stable outcome.
Are there countries that have a better design for their electricity network control systems?
Disclaimer: I don't design electricity networks nor electricity markets. And the above is ignoring loads (loads are mostly less problematic for control than generation).
Sadly, some news outlets are probably only going to look at the recommendations and read "cybersecurity" and (even though they are common sense recommendations) assume there might be more to say about the matter.
Oh wait, they already did: https://www.telegraph.co.uk/business/2025/06/18/renewable-en...
Ed: Do I need a /s tag here or something? My point was that we shouldn't worry too much about about the presentation of the report, its actual contents will be spun to suit any narrative regardless.
I would like to see: "We have simulated the complete 200 and 400 kV grid of the iberian peninsula and western europe, and can reproduce the situation that occurred. Any one of the following changes would have prevented the issue, and we suggest implementing them all for redundancy. This simulation will be re-run every day from now on to identify future cases similar incidents could occur"
Now I'm curious about what's in the confidential version of the report.
> However, as is common in networks and information systems in any sector, other risks have been identified, such as vulnerabilities, deficiencies or inadequate configurations of security measures, which may expose networks and systems to potential risks, for which a series of measures are proposed.