Too much hardware was allowed to disconnect right at the edge of normal conditions.
See full article...
See full article...
Final analysis of 2025 Iberian blackout: Policies left Spain at risk
- Thread starter JournalBot
- Start date
Upvote
46
(46
/
0)
Err..
One of the conclusions from the report is that Spain runs it's 400 kV grid at 380-435 kV while most/all other ENTSO-E members run it at 360-420 kV.
And Spain had a lot of 400 kV stuff set to trip out at 440 kV, 435 kV or even less.
Thus little, none or negative margin to the normal operation voltage range.
That played a role in the quick trip cascade once voltages rose due to reactive power.
One of the conclusions from the report is that Spain runs it's 400 kV grid at 380-435 kV while most/all other ENTSO-E members run it at 360-420 kV.
And Spain had a lot of 400 kV stuff set to trip out at 440 kV, 435 kV or even less.
Thus little, none or negative margin to the normal operation voltage range.
That played a role in the quick trip cascade once voltages rose due to reactive power.
Upvote
61
(64
/
-3)
The ENTSO-E report also says that the classic power plants were not performing up to spec when it came to handling reactive power.
Upvote
47
(48
/
-1)
Very likely that the small home inverter manufacturers prioritized protecting the inverters instead of implementing best practices for the grid. After all, if the grid fails but the inverters survive, no warranty claims from unhappy customers. If the reverse happens, the grid stays up but inverters die, mad customers and warranty claims paid out. Might even be the case for the larger commercial units as well.
Upvote
78
(79
/
-1)
I should acknowledge that the ENSO-e report is over 400 pages long, and does not include anything like an executive summary. Even with this being a long report, I have necessarily skipped a lot of what it covered, including a complete analysis of the restart process for both Spain and Portugal.
Upvote
125
(125
/
0)
It's not returned to the grid at "some point". It's being returned all the time, at grid frequency.
Basically, reactive power is energy ping-ponging back and forth (at grid frequency) between the coils and capacitors (reactances) in the grid.
Attempt at clarification.
Ideally, all generators connected to the transmission grid are required to absorb up to a certain amount of reactive power, under command of the transmission grid operator.
In modern solar/wind farms that's done automatically.
In older power plants, that's often done by phoning the power plant operators.
But at the time of the blackout, the wind and solar farms in Spain were not operating like that.
They were just required to absorb a certain reactive power as a fraction of the (real) active power they were providing at the moment.
In short there was a lot of underused reactive power absorption capacity in Spain's wind/solar farms and those that stayed online didn't even try to pick up the slack once the others tripped out.
EDIT: missing "reactive"
Last edited:
Upvote
90
(90
/
0)
Page 23-24. It's as close to a summary of what went wrong as it can fit into two pages.
Upvote
34
(34
/
0)
Not suggesting you’re incorrect (this seems likely) but is there also concern that inverters allowing a larger swing in grid voltage could cause arcing/fires? I know that wasn’t an outcome from the one manufacturer that didn’t have so many inverters fall off the grid, but not sure if that’s also a legitimate concern.
Upvote
10
(10
/
0)
Good to get some closure on this. I keep getting disappointed with publications, including Ars, for not following up on their stories, unless the outcomes are newsworthy on their own.
For example, Beth just stopped writing about the South Carolina measles outbreak once it petered out, even though her last article is titled "South Carolina tops Texas measles outbreak record—with no end in sight."
For example, Beth just stopped writing about the South Carolina measles outbreak once it petered out, even though her last article is titled "South Carolina tops Texas measles outbreak record—with no end in sight."
Last edited:
Upvote
11
(26
/
-15)
Love the call out to Practical Engineering in the article. 
I wonder if some of these systems need a certain amount of variability baked in for making certain decisions.
Like say your system/product needs to disconnect when a measure goes beyond a set value. Normally not a problem. But when you have hundreds, thousands, tens of thousands, or more, of those devices all operating on the same rule? Ouch.
Maybe we can identify certain measures where it is useful to require manufacturers and operators to add a random offset for each instance of a device. The aim being to avoid everything acting at exactly the same time.
One of the curses of our ability to measure things getting ever more and more reliably accurate.
I wonder if some of these systems need a certain amount of variability baked in for making certain decisions.
Like say your system/product needs to disconnect when a measure goes beyond a set value. Normally not a problem. But when you have hundreds, thousands, tens of thousands, or more, of those devices all operating on the same rule? Ouch.
Maybe we can identify certain measures where it is useful to require manufacturers and operators to add a random offset for each instance of a device. The aim being to avoid everything acting at exactly the same time.
One of the curses of our ability to measure things getting ever more and more reliably accurate.
Upvote
33
(33
/
0)
Love the call out to Practical Engineering in the article.
I wonder if some of these systems need a certain amount of variability baked in for making certain decisions.
Like say your system/product needs to disconnect when a measure goes beyond a set value. Normally not a problem. But when you have hundreds, thousands, tens of thousands, or more, of those devices all operating on the same rule? Ouch.
Maybe we can identify certain measures where it is useful to require manufacturers and operators to add a random offset for each instance of a device. The aim being to avoid everything acting at exactly the same time.
One of the curses of our ability to measure things getting ever more and more reliably accurate.
In some cases yes.
But when it comes to things like trip-thresholds no.
The price you'd be paying for the randomness is that some nodes will have a lower trip margin than needed. You're basically placing yourself closer to a trip cascade.
You want to have as much of a trip margin as feasible.
Upvote
16
(16
/
0)
YetAnotherAnonymousAppellation
Ars Praefectus
One thing I've been wondering about for years is what effect the massive number of constant-power devices attached to the grid have on stability. Switching power supplies are in pretty much every line-operated electronic device these days (even LED light bulbs). Because they present a constant-power load to the grid, they are a negative resistance. Negative resistance is really good at promoting oscillation.
Any comments from those who are knowledgeable out there?
Any comments from those who are knowledgeable out there?
Upvote
12
(13
/
-1)
I seem to have been misconstrued. Animal dung was a proxy for an old (and inefficient) way of doing things. New tech is always going to have a learning curve. We will learn how to integrate massive proportions of renewables into the mix with an eventual goal of 100% renewables.
Upvote
15
(15
/
0)
Yeah, for things I'm trying to learn, I tend to read at least four sources to make sure I'm not getting biased by any single approach to explaining it, and get exposed to at least some of the subtleties. In this case, I decided to watch a couple of the videos as well, since an AC grid is a dynamic thing. I thought the PE video was impressively clear, and then I saw a few people saying "I work on power systems and this is the best I've seen" in the comments.Love the call out to Practical Engineering in the article.
So, seemed worthwhile to share, especially since I had to breeze over reactive power pretty quickly in the piece.
Upvote
67
(67
/
0)
What I meant in that the old way wasn't operating to spec thus the old it also contributed to this SNAFU.
So we also have to learn to properly operate animal dung.
Upvote
15
(15
/
0)
So is the problem that the common grid standard is 380kV and so equipment suppliers create equipment with a standard safety factor of 10% (meaning +-38kV) and so when running at 400kV nominal they they might be setup to trip at 435 or 440kV which provides a smaller cushion factor?
Upvote
3
(3
/
0)
I think there would be some variability built into the system as those small domestic units are operating on local domestic lines around 230v. Now that voltage can vary depending on local loading and how far you are from the local transformer etc? So while it's a nominal 230v, it's +/- 5%. So while I might be seeing 235v, the guy down the other end of the road might be getting 225v. Both are totally "in spec". But how our grid tied inverter behaves would be linked to those numbers, and if the voltage spiked or dropped, one unit would drop out before the other.
I wonder if a contributing factor was the delay in the tripped inverters kicking in again? At that point the larger generators would have reacted and the voltage started swinging back to normal, when a heap of uncontrolled generation kicked in again. Did that delay reinforce the oscillation? Would it be better to have the domestic system kick in and out faster so they don't work in sync with the larger power grid oscillation?
This is just speculation of course, I'm not a power grid surgeon.
Upvote
5
(5
/
0)
Fun fact: Australia's nominal voltage is 230v, +10%/-6%. 216-253 volts. Centred on 234.6 volts. The preferred operating range is +6%/-2% (225-244 volts).
This has absolutely nothing to do with the fact that Australia's nominal voltage used to be 240 volts, plus or minus 6% (prior to the year 2000).
Typically, home solar inverters here drop out around 250 volts, and in fact the distributors are known to deliberately push the voltage up to that level when they want less uncontrolled solar on the grid.
I doubt that tripped inverters kicking in again would be a significant factor; usually, when an inverter trips because of high voltage, there's a low voltage mark that the line has to reach before it'll kick back in, and that's usually significantly below the high voltage cutoff mark (to prevent that sort of kick in-out-in-out oscillation.) Note that this is uninformed speculation on my part, not researched or supported by documentation that I've verified.
Upvote
18
(19
/
-1)
The grid nominal voltage is 400 kV, which means the grid will aim at voltages at around 400 kV.
But the tolerances are not symmetrical. E.g. the requirements for a generator connected to the 400 kV Portuguese network are:
- 400 kV -15% to -10%: operate during 60 minutes
- 400 kV -10% to +5%: normal operation
- 400 kV +5% to +10%: operate during 20 minutes
Basically the generators are required to be more tolerant to under voltage than to over voltage before tripping out.
That said you basically got it right: since in practice the allowed range in the Spanish side is higher than other countries, it looks like they ended up with quite a bit of things where the threshold provides little to no cushion.
Upvote
25
(25
/
0)
Oscillations in the grid...
It is not spelled out explicitly in the article (but it is on page 11 of the report), that the things that are oscillating (varying) in the grid are power, voltage and frequency. Technically, the power and voltage are already oscillating at ~50Hz, so what is being measured is the RMS (i.e. average) values of these. The variation in frequency is above or below the nominal value of 50Hz.
These three things are all interrelated, but while the grid can successfully accomodate a variation in voltage of a few percent, the frequency must be maintained within a much narrower range for the generators to be able to connect and supply power.
It is not spelled out explicitly in the article (but it is on page 11 of the report), that the things that are oscillating (varying) in the grid are power, voltage and frequency. Technically, the power and voltage are already oscillating at ~50Hz, so what is being measured is the RMS (i.e. average) values of these. The variation in frequency is above or below the nominal value of 50Hz.
These three things are all interrelated, but while the grid can successfully accomodate a variation in voltage of a few percent, the frequency must be maintained within a much narrower range for the generators to be able to connect and supply power.
Upvote
16
(16
/
0)
Resistance
Wise, Aged Ars Veteran
The restart process could probably be an entire article on its own... please.
Upvote
42
(42
/
0)
I second that notion. Black starts of a grid are something that absolutely fascinate me: managing the load while the generators spin up to prevent overload, progressively bringing it all online within the bounds of what can be handled without having too much or too little load at any given stage... it's a dance that I'd love to see detailed.
Upvote
40
(40
/
0)
Resistance
Wise, Aged Ars Veteran
I've listened to a lot of experts from a wide variety of fields talk about challenging aspects of their work, things that intimidate them, what keeps them up at night, and there isn't much that is taken as seriously as a black start by people who work in power generation and distribution.
Upvote
31
(31
/
0)
The last time this event had an article here I unsuccessfully tried to make the point that to have a reliable grid with a high percentage of power input provided by inverters that it was going to require running those inverters in more than just “grid following mode”. Since it was made clear in the final report that this grid following mode was what was actually being done and further stated that things will have to change to increase grid reliability!
Unfortunately I said that grid forming inverters were not a thing yet (which was a bad simplification). Maybe a more appropriate terminology would have been to say a remotely dynamically controlled grid forming inverter but that is also not the generally used terminology. ABB uses terminology like Dynamic real power control, Dynamic reactive power control, Generator emulating control mode, Grid stabilization features including synthetic inertia and active damping, Voltage and frequency dynamic envelope / regulation functions, Synthetic inertia with Synchronous machine emulation, providing system voltage stability with dynamic system response and so on. I was placing these capabilities under the dynamic grid forming terminology which is an over simplification and each capability probably requires an explanation.
I stated that this grid forming inverter capabilities were a work in progress. I even got someone saying that they have a grid forming inverter under their desk which obviously shows that my point was completely missed. Most of the BESS inverters have most of the ABB capabilities as noted. The reason I stated that this is a work in progress is because those capabilities are LOCAL to the inverter. What the final report made clear was that to increase overall grid reliability, was that a DYNAMIC NONLOCAL REALTIME GRID CONTROL SYSTEM has to provide REALTIME control inputs to the large inverters LOCAL inner control loops to deal with dynamic oscillatory grid events.
This realtime grid control system is a very complex problem and seems to require a lot of interface standardization between a large number of different vendors that are doing a lot of different LOCAL CONTROL FUNCTIONS with different grid technologies. The days of just making a phone call to someone at a rotating generator to change the power factor are long gone.
The London blackout event all happened within a few hundred milliseconds due to an internal oscillation that triggered a software fault parameter excursion in the control system at a wind farm. That final report was not covered here to close the loop from the original article, so this article is a good step in the right direction.
Unfortunately I said that grid forming inverters were not a thing yet (which was a bad simplification). Maybe a more appropriate terminology would have been to say a remotely dynamically controlled grid forming inverter but that is also not the generally used terminology. ABB uses terminology like Dynamic real power control, Dynamic reactive power control, Generator emulating control mode, Grid stabilization features including synthetic inertia and active damping, Voltage and frequency dynamic envelope / regulation functions, Synthetic inertia with Synchronous machine emulation, providing system voltage stability with dynamic system response and so on. I was placing these capabilities under the dynamic grid forming terminology which is an over simplification and each capability probably requires an explanation.
I stated that this grid forming inverter capabilities were a work in progress. I even got someone saying that they have a grid forming inverter under their desk which obviously shows that my point was completely missed. Most of the BESS inverters have most of the ABB capabilities as noted. The reason I stated that this is a work in progress is because those capabilities are LOCAL to the inverter. What the final report made clear was that to increase overall grid reliability, was that a DYNAMIC NONLOCAL REALTIME GRID CONTROL SYSTEM has to provide REALTIME control inputs to the large inverters LOCAL inner control loops to deal with dynamic oscillatory grid events.
This realtime grid control system is a very complex problem and seems to require a lot of interface standardization between a large number of different vendors that are doing a lot of different LOCAL CONTROL FUNCTIONS with different grid technologies. The days of just making a phone call to someone at a rotating generator to change the power factor are long gone.
The London blackout event all happened within a few hundred milliseconds due to an internal oscillation that triggered a software fault parameter excursion in the control system at a wind farm. That final report was not covered here to close the loop from the original article, so this article is a good step in the right direction.
Upvote
12
(12
/
0)
Upvote
6
(6
/
0)
I’ll preface this with an explicit declaration that I am way out of my element in understanding any of this beyond some general simplicity. Or, even more to the point, I may be ignorantly understanding any of this believing I even have a basic understanding of it.
But if there continues to be a broad movement to moving generation to smaller “contributors” (ie rooftop solar or even businesses with their own generation) that also feed into a broader grid, isn’t there a necessity to improve the ability for those contributors to better participate in the reactive side of the equation?
As someone previously mentioned, there is a significant disincentive to individualize the risk and cost of absorbing that when it’s probably better to socialize the risk. Based on the report, generation policy needs to incorporate the socializing of that risk and that needs to include covering the cost of warranty or repairs for those individuals if the grid operators exceed whatever thresholds are defined.
But if there continues to be a broad movement to moving generation to smaller “contributors” (ie rooftop solar or even businesses with their own generation) that also feed into a broader grid, isn’t there a necessity to improve the ability for those contributors to better participate in the reactive side of the equation?
As someone previously mentioned, there is a significant disincentive to individualize the risk and cost of absorbing that when it’s probably better to socialize the risk. Based on the report, generation policy needs to incorporate the socializing of that risk and that needs to include covering the cost of warranty or repairs for those individuals if the grid operators exceed whatever thresholds are defined.
Upvote
10
(10
/
0)
Both.
Legal standards defining what small scale solar needs to be able to do in terms of remote control and monitoring, and electrical regulation to ensure that they follow those standards.
To add to the fun, it’s likely that existing equipment will need to be retrofitted to comply with said standards; it won’t be as effective if half the solar installations don’t comply because they’ve been grandfathered in.
All of which adds to the fun of general internet of things equipment, since, as we all know, the “S” in IoT stands for security…
Upvote
18
(18
/
0)
It always amazes me how Ars readers and writers can communicate through comments in normal, positive way. Not only that, but explanations and attached content both in article and comments are really worthy of read on their own. That's what keeps me on this site. A rare beacon of light in this ever darker world.
Great article, great community!
Great article, great community!
Upvote
61
(62
/
-1)
Amen fellow Arsian. It's exactly why I got a subscription for a while.
Upvote
24
(24
/
0)
You are making a lot of assumptions.
“Very likely that the small home inverter manufacturers prioritized protecting the inverters instead of implementing best practices for the grid.’ That’s not something manufacturers get to decide. That behavior is set and enforced by grid operators.
Nor is a grid power source disconnecting necessarily a bad thing. Any generator operating outside its regulated limits can damage itself or the grid, so shutting down when conditions get out of bounds is often the correct and expected response.
Upvote
9
(9
/
0)
Yeah, i get that, but that might have been part of the issue. The domestic supply tripped out on high voltage. So the voltage drops, then other parts of the grid ramped up a minute later (gas turbine and hydro have response times in minutes). Than as they were recovering the domestic solar started kicking in again.
See how you could get out of control oscillation.
Upvote
1
(1
/
0)
Yes, black start procedures strike fear into the heart of any grid operator. I'd happily read an Ars feature on it, but I'm not sure I'm quite willing to read the report in full.
Random aside - here in NZ one of the key protection systems for the grid is Automatic Under-Frequency Load Shedding, aka AUFLS. As in things are awful if AUFLS happens. I'm sure someone had a chuckle when they came up with that backronym...
Upvote
24
(24
/
0)
It appears that the 2016 South Australian Blackout has not become prescribed reading for grid operators.
If are interested in how grids can move to 100% renewables, South Australia is a good case study.
The quick fix was the Hornsdale Big Battery to provide frequency regulation. Politicians were told by Electrical Engineers that the intertia provided by big spinning turbines needed to be replaced by something else, but were ignored until the state wide blackout demonstrated the risk.
If are interested in how grids can move to 100% renewables, South Australia is a good case study.
Upvote
22
(22
/
0)
I don't know what the rest of this week looks like in terms of news that should be covered, but I will try to skim that this weekend and see if there's anything there I can build a story around. While scrolling past those sections, I think I did see that they did a sort of "bring up islands then connect them" approach, and a few islands failed to come up, but I don't know if some level of failure was expected or things failed in interesting ways that I can understand, etc.
Upvote
38
(38
/
0)