The Spanish Grid Drop-out - recently released information.

...

Perhaps for systems that have large solar or wind arrays they could use a number of large rotating masses to smooth over these burps? Vacuum and magnetic bearings...

I imagine a series of rotating masses so if any single or several fail (earthquake, etc.) the system wouldn't collapse.

As you say, there is little inertia in these solar systems unlike water or fuel generated power.

John :-#)#

Fascinating. Thanks. That ties in with my earlier post wondering if older inverters with too strict rocof cutoffs were at least partially responsible for collapse.

Any hints at the precipating cause?

Maybe some modest local event triggered a fundamentally unstable system.

If every solar inverter was networked and controlled in voltage/current/phase angle, by some intelligent system controller, one might not be so dependant on rotating mass.

As solar and wind get to be dominant, micromanagement of power sources and loads will be necessary to ensure uptime.

(That's a big part of the green agenda: control everything.)

This is largely unnecessary - if the control signal that was being sent out by the central controller to micromanage each power source was derived from a function of the frequency, phase, voltage etc., then rather than trying to distribute the result of this calculation to millions of devices with low latency, it is better to distribute just the formula (once every few years or as necessary), and run it on a microcontroller in the inverters several times every mains cycle. They already have more than enough processing power.

I believe that there are some new regulatuions in at least one Austrlian state, driven by the (fossil-fuel-stoked) fear of "too much solar destabilising the grid", which require new home solar inverters to stop exporting power, unless they receive continuous "permission to export" signals from our overlords, the network operators. In other words, rather than exporting power in the case of communications failure, it goes into the state of "export no power" in case of communications failure, because otherwise people might unplug their internet to export more scary solar power if exporting power was allowed when the internet connection fails. This is a fairly new requirement, so not many compliant devices are installed now, but once a few gigawatts of these inverters are running, it will be interesting to see what happens when there is a major internet outage on a hot summer day, and all of those gigawatts suddenly go away. Hopefully they thought of that but I doubt it.

The rapid control algorithms should be distributed, and the only low-latency communication signals they should rely upon are frequency and voltage.

The sun disappears every night, and less sunlight gets through when the sky clouds over. Extra solar panel area and grid scale storage makes this easier to deal with.

Only in climate change denial propaganda, which is paid for by the fossil fuel extraction industry, which wants to continue to control everything.

Paradoxically, because solar panel can be distributed much more easily than oil wells and oil refineries, solar power lends itself a lot more easily to the kind of local control and negotiated interaction between larger areas that the anarcho-syndicalists prefer.

John Larkin would probably see that as disadvantage if he understood what it meant, he prefers to be hoodwinked by the people who have always manipulated his opinions.

It is cruder than that. They've just stopped paying any realistic kind of feed-in tariff to people with roof-top solar, and as a result 40% of new roof-top solar in Australia is now being installed with Tesla Powerwall or similar battery. It more than doubles the cost of the installation, but reduces the pay-back time for the whole installation to about seven years, and save you negotiating with your power supplier about their derisory feed-in tariffs.

It's more that they haven't yet got enough poles and wires in the right places to move the roof-top power to where it could be used.

The fundamental problem is that the network got privatised about twenty years ago, and the ownership structure that got set up was designed to make it easy to sell shares in the old hardware - it is totally unsuitable for any kind of distributed system.

Sounds sensible, but the current ownership structure wasn't designed with that in mind.

"1. Large generators in the South of Spain started to trip at 12:32:57 CET. Over a period of 20 seconds a total of 2.2GW was lost – this is well beyond largest infeed so not secured against."

This is a pretty clear statement. It doesn't say anything about why the large generators - type unspecified - lost 2.2GW of generating capacity over a twenty second period, and I haven't seen anything any more specific anywhere.

Too modest to have been noticed. Power generation systems are quite busy

- people are connecting any disconnecting stuff all the time - so it would have taken a rather improbable modest event, or collection of modest events to to trigger this hypothetical mode of instability.

No, they say:

"What happens if my solar inverter loses internet connectivity?

If your solar inverter loses internet connectivity, the excess energy you export to the grid will automatically be reduced. This ensures it can be safely managed." ( from here:

So if all of the inverters lose internet, which is entirely likely at some point bearing in mind our telcos, we can expect a blackout too, all so that "it can be safely managed." The blackout will no doubt help the telcos to get back online promptly. Fun times ahead.

I see a problem if the network is hacked. Same as routers are hacked today.

... (distributed algorithms)

What they say isn't all that interesting. What they do is discourage people from trying to sell their excess power back to the grid.

I haven't lost my internet recently - the last time it happened it was not due to anything the telcos had done - the mains supply to my apartment block had to be cut off for hours while they replaced the local distribution transformer, which sits just outside our front gate, and it was entirely local. A few years back it dropped out for couple of hours due a problem with my telco, but it only affected people served by that telco, and was confined to a single suburb.

The chance of all the inverters losing internet connectivity at once doesn't seem to be all that high.

And it isn't now?

I think the chance of at least one major telco going offline in the next decade is pretty high. It's happened to mobile networks and payment systems several times.

For problems to occur, it isn't necessary that the inverters all lose internet - the other end of the connection could also fail. If the other end of the connection goes to just a few datacentres, and if for some reason they get misconfigured, or just encounter a DNS problem, it might cause an unnecessary blackout.

There are enough unavoidable causes of power failures, we needn't create new ones through legislated fragility in otherwise resilient equipment.

They could at least build in a long random time delay between loss of internet connection and the inverter shutting down (so the population of inverters will gradually shut down over a 5+ hour period).

The whole point about ARPANET - and today's system is just a development of that - is that it was resilient. People have managed to take down bits of it anyway, but that's been just stupidity.

But it is very unlikely to be quite so clumsily implemented.

There doesn't seem to be any legislated fragility on offer - just people trying to implem4ent a distributed system without paying enough attention to what they are doing.

Everybody and his brother has bright ideas about what might have prevented the Spanish shut-down, uninhibited by any detailed low level information about what actually went wrong.

Design is about developing a detailed understand of what is actually going on - and in this case what actually went wrong, and only then trying to change or improve the system to make it less likely to happen again.

Ethernet relies on short random delays before you try to resend a message. The wait times are chosen to match the delays around the network. It might well be sensible to for the feed-in inverters on a distributed grid to react to large phase inversions in a way that doesn't lead to all of them dropping out at once.

It might be a whole lot more sensible for them to react by acting in a way that tended to reduce the phase excursion. The amount they could do would be constrained by the amount of power they could contribute, and the amount of stored energy that they could call on to sustain any corrective input.

Let's see some design ideas that might prevent the problem from developing in the first place, rather than getting fixated on the kind of mistake that might have created the problem.

A naive question: why do we need to get these signals from the grid at all? Why can't we broadcast a synchronisation message on something like LW radio that is picked up by every generator large or small? Then the network operator can monitor what's happening and adjust the signal as appropriate.

No need for internet connectivity means no problems with network delays, only the speed of RF from one end of the country to the other. You would of course have multiple transmitter sites - they would cost in terms of power to run, but compared with grid power it's tiny. (before anyone says you cannae get the transmitters any more, yes you can - Nautel will sell you a new one)

Or is the problem that we actually do need slight desynchronisation - some parts of the network become overloaded and need to 'slow down' compared with other parts? (and they do that by phase differences rather than voltage sag) In which case the frequency differences follow the network topology and the power flows.

Theo

That is not unlike the failure in the UK Aug 2019 where an apparently inconsequential minor power station dropping off due to a lightning strike started a cascade failure that spread until they shed enough load to get a balance again. Wide area power cut resulted.

Green energy systems often react badly to frequency deviations. Whilst there is no reason why this should be the case it is frequently shown to happen. Some BESS systems *are* configured to maintain grid frequency but by no means all. They have the advantage of fast response but for that to be true they must not also be already running at full capacity.

Problem in the UK is that daytime load is such that everything that can is running close to the limits during the daytime with an evening peak that stresses the N-S interconnectors even in summer. I noticed last week one evening at peaktime that the supergrid power cables were visibly sagging as a result of the current flowing through them. I'd never really noticed that before but I expect it happens all weekdays.

They are intrinsically dangerous if they store enough energy to really matter. We had such a steel reinforced lead flywheel and motor generator configuration on big radio telescopes storing just enough energy to stow them in the event of a storm taking out 3 phase mains power. The dishes can only reliably survive storms if they are pointed at the zenith. (sometimes not even then)

Working out how far it would travel if it ever broke free from its very substantial bearings was used as an exam question. It was installed pointing so that it would not hit any property if it did.

Magnetic levitation vacuum pumps were all the rage when I was in Japan. That was until one day the entire world moved an inch to the left. Every last one of them crashed with shattered titanium blades everywhere and no vacuum/moist summer air in the chambers. Hell of a mess. After that we went back to conventional bearings in all earthquake countries.

The advantage of gas turbines or diesel generators is that when the rotor starts to slow it automatically increases the gas supply to try and maintain frequency. The stored energy in the rotor is significant but it it have the ability to output a bit extra or a bit less in response to changing load that makes them so handy for stability.

The other alternative is to have loads of last resort that can be shed at any time to compensate for loss of generating capacity, but losing

Green energy systems draw power from wind and sun. Both are variable. They have have to a maximum capacity way above their average load. They won't ever all be running at full capacity.

A battery energy storage system has exactly the same advantage. The point about rotating lumps of metal is that they store energy. So does a battery, and it has the advantage that it is less sensitive to earthquakes.

It probably wasn't a single 2.2GW source, but a badly configured collection of smaller sources. The rest of the net does seem to have been equally badly configured, but less tightly coupled.

We still haven't got a clue why the system fell over, but a common duff algorithm is a minimal explanation.

A central (international!) controller would want to know what every contributor was pushing into the grid, and probably see wind flow and clouds moving around. One local transmission line could fail and take down half of Europe. Again.

A solar panel with an algorithm can't now about potential system overloads. Solar and wind will have to be shed sometimes to protect the entire system. Loads shed too. Renewable-heavy grids are fragile.