As someone that used to help manage data-centers I think this is fine. I had to load test the generators quarterly. It's good to ensure the transfer switches and generators are working and nothing beats a real world load test. The data-centers have contracts with diesel fuel providers to keep the main tanks full. They can run on diesel any time the infrastructure load is high and write off the cost in their taxes check with your tax lawyer. There may even be other tax advantages being the need for generators would be compelled by the state, perhaps a tax lawyer could find ways to make a generator tech refresh get a bigger write-down, write off better noise abatement walls as usage will increase. If a company was running with scissors and did not buy enough generator capacity to run everything then they will have to get their act together now vs. later.
Shank · 3h ago
This is true, but how long will firm load shed events last, and how many of them will happen? In California, when CAISO has events that lead to firm load shedding, they're predictable, rolling blackouts and everyone knows how long they'll last, and they're assigned on a customer-specific basis. You know your power will be cut, and you know it will be a certain amount of time, and you know roughly where you are in the schedule.
I could see operators of datacenters in Texas wondering about this. Also, it's underrated how much critical infrastructure is dependent on datacenters running. Like, are you going to pull someone's EHR system down that serves a local hospital, while keeping the local hospital on a critical circuit?
Bender · 2h ago
You know your power will be cut, and you know it will be a certain amount of time, and you know roughly where you are in the schedule.
Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity.
Some data-centers are indeed on circuits deemed to be critical but I could see regulations changing this so that they are "business critical" vs. "life support critical" and some changes could be made at substations so that data-centers could participate in shedding. I think you are right that they will be thinking about this and adding to this probably filing preemptive lawsuits to protect their business. Such changes can violate SLA contracts businesses have with power companies and Texas is very pro-business so I can not compare it to California.
imglorp · 2h ago
Texas has shown no interest in life support critical. They prioritized operator profits over uptime. Hundreds died as a result.
>Texas has shown no interest in life support critical.
I'm not sure that this is correct. I was initially worried about how Mom would fare since she lives alone and is over 80. During the entire one week period of power problems in Feb. 2021 my Mom never lost power, not even a quick brown-out. Her home is within a half mile of a local hospital which also never lost power. The area around the hospital did not lose power so businesses and homes close by had no issues with heating, cooking, bathing, etc during the cold blast. That fact allowed me to stay here at my place a couple hours away and manage my own situation which was fairly easy compared to many others in the state.
Your other statements are quite true and to date no one who played a part in mismanagement of utility power in Texas has been held accountable nor will they ever be in a libertarian state where regulations exist only to guarantee a profitable situation for a commercial entity. In fact, most electricity customers in Texas ended up paying for the huge cost increases that occurred as those in charge tweaked the system in real time to maximize their own profits.
Texas needs regulations worse than most other states. Grifters, fraudsters, and thieves have filled too many critical positions for too long.
toast0 · 2h ago
> Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity.
If you know when the power will be cut, you can start the generators before the cut, and depending on your equipment, you may be able to synchronize the generator(s) with the grid and switch over without hitting the batteries. I assume big datacenters are on three phase, can you switch over each phase as it crosses zero, or do you need to do them all at once?
bluGill · 2h ago
Again, that doesn't matter because everyone knows the grid isn't 99% reliable. They just pull the big power switch to the whole building and watch all the backup systems work. If anything fails it was broke already and they fix/replace it. Because this happens often they have confidence that most systems will work - even where it doesn't computers fail unexpectedly anyway and so they have redundancy of computers anyway (and if it is really important redundancy of a data center in a different state/country)
Synchronizing generators is a thing, but it isn't useful for this situation since they need to be able to handle the sudden without warning power loss where generators cannot be synchronized anyway.
stevetron · 27m ago
How often does that inverter burn-out a transistor? Is there a backup inverter? Do you keep replacement transistors on-site?
Bender · 2h ago
At least in the data-centers I helped manage the inverters were in-line running 100% duty-cycle, meaning frequency sync is not required as there is no bypass. The servers never see the raw commercial power. Data-centers in the US are indeed 3-phase. FWIW the big Cats did have controllers that would maintain sync even when commercial power was gone but we did not need it. There wasn't even a way to physically merge commercial and generator power. ATS inputs and outputs were a binary choice.
I know what you mean though, the generators I worked with in the military had a manual frequency sync that required slowly turning a dial and watching light bulbs that got brighter with frequency offset. Very old equipment for Mystic Star, post-WWII era equipment. 50's to 90's
dylan604 · 1h ago
In the facilities I have been in (not managed), they were all in-line as you describe as well. Mains power is dirty. Having a data center without line condition on mains would be insane.
jimmygrapes · 2h ago
Manually syncing several of the MEP012 generators was always far more stressful to me than any physical dangers!
tw04 · 1h ago
If power is out everywhere for an extended period of time, they aren’t doing anything but life saving surgery. Pulling up an EMR will be near the bottom of the list of concerns.
nradov · 2h ago
All of the major cloud EHRs run in multiple availability zones.
mschuster91 · 1h ago
The data center might be... but are all fiber routers, amplifiers, PoPs etc. along the line from the datacenter to the hospitals and other healthcare providers backed up as well?
Particularly the "last mile" often enough is only equipped with batteries to bridge over small brownouts or outages, but not with full fledged diesel engines.
And while hospitals, at least those that deal with operating patients, are on battery banks and huge ass diesel engines... private small practices usually are not, if you're lucky the main server has a half broken UPS where no one ever looked after that "BATTERY FAULT" red light for a year. But the desktop computers, VPN nodes, card readers or medical equipment? If it's not something that a power outage could ruin (such as a MRT), it's probably not even battery backed.
There's a German saying "the emperor is naked, he has no clothes". When it comes to the resilience of our healthcare infrastructure, the emperor isn't just naked, the emperor's skin is rotting away.
nradov · 1h ago
Starlink
vel0city · 2h ago
If they cannot handle the grid power being pulled due to load shedding, they have no business handling critical applications.
bluGill · 2h ago
In particular there is no way you can predict when a backhoe will take our your power line. (Even if they call to get lines located - sometimes the mark is in the wrong spot, though most are they didn't call in the first place). There are lots of other ways the power can go down suddenly and nothing you can do about it except have some other redundancy.
chaz6 · 9m ago
Are they clean enough to stay within the limits of regulations around pollution when run for long periods of time?
abeppu · 43m ago
I wasn't involved in the specific details but I remember being told that during the power outage from hurricane Sandy, even datacenters that had sufficient generators had trouble getting the diesel to keep them running, because everyone wanted diesel at the same time and both the supply and distribution were bottlenecked.
How long can most DCs run with just the fuel onhand? Have standards around that changed over time?
jabart · 36m ago
At a call center, that had a whole datacenter in the basement. They had two weeks of fuel on hand at all times. Being on a border of a state, they also had a 2nd grid connection in case one failed.
The whole area lost power for weeks but gym was open 24/7 which became very busy during that time.
stevetron · 24m ago
It seems to me that Hurricane Sandy caused an issue with fueling backup generators in a New York City datacenter, where there was a bucket-brigade of people carrying fuel in pails up flights of steps several stories.
stogot · 39m ago
I recall ranges I’ve heard from operators as 24h to upwards of 72 (rare)
lokar · 2h ago
For any big facility there will be pretty strict EPA limits on how long you can run the generators each year.
dylan604 · 1h ago
The EPA? Are they still a thing? I doubt anyone is concerned about the EPA under current management.
bilbo0s · 1h ago
Which is great.
Until there's new management.
You can't run a business by seesaw.
Best to just count on that rule being enforced and place the necessary battery backups and wind or solar in place to backstop the diesel. Then make any users who need to use those data centers eat that extra cost. There's no problem with us-east costing less than us-west, and us-texas costing most of all. That's how markets work.
Bender · 1h ago
Indeed. I have faith that Texas will find a way around such rules especially if they are being regulated into running them. A Texas company I worked for was highly proficient in maximum shrugs.
more_corn · 1h ago
The EPA doesn’t really have the resources to enforce that. And certainly won’t have that capability under the trump administration.
lowwave · 2h ago
Nice to know that, but are the data-centers Carrington Event proof? Always wondered that, but never worked in a data-center.
w10-1 · 40m ago
Regulating high-demand customers does seem most reasonable. But when the Texas legislature gets involved, it's typically to build a political franchise by creating government power than can be used arbitrarily against a fixed asset. In this case, nothing prevents the grid authority from harassing the data center during the many overload periods when a power contract is coming up for renewal.
Not that businesses aren't abusing the process, either. The article mentions that 80-90% of planned data centers won't be built, due to duplicate applications. They duplicate both to secure a "phantom" slot for power, and to get localities to compete with incentives for business. It's hard to plan a grid when financial speculation and gamesmanship is driving the planning.
The worst aspect is that Texas corruption is explicitly being offered as a model for other states in the PJM interconnect.
I wonder how much of the high economic growth rate in the 1950's-1980's came from the lack of gamesmanship and political franchises. But it's probably unrealistic to think professionals could step back from the very maximalist positions that are selecting them as leaders.
dev_l1x_be · 4m ago
Only if we had a cheap, easy to control energy source that could power the AI revolution.
Emissions limits usually hard-cap the number of annual hours of backup generator runtime. Every minute the generators are used during load shedding is a minute less of backup runtime for other contingencies.
jqpabc123 · 2h ago
The main problem with the Texas grid is really very simple.
Their "free market" real time power auction makes no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market.
Some things are too important to leave to the "free market".
infecto · 2h ago
I hear opinions like yours often but I am not sure it’s simple or that the reasons you’re citing are grounded in reality.
What is the alternative, have a state imposed monopoly with a single power company like PGE who I would not see as an ideal operator either. Same can be true for a lot of other similar generators across the company.
You’ll probably bring up the winter storm outage which is inexcusable but their neighbor to the north SPP had very similar failings in being prepared and only faired better because they have interconnects.
Texas has had some of the fastest adoption for wind and solar. It is far from perfect but I also think there is benefit to having multiple generation companies supplying to the grid. You have companies with different expertise and perhaps innovation.
bityard · 26m ago
There is lots of middle ground. Here in Michigan, (most) electricity and gas providers are for-profit companies. But they are heavily regulated by the state, and must get approval from the state before they are allowed to change rates. Our rates are not dirt-cheap, but they are not Coastal either.
When I lived in the capital, we got our power from the Lansing Board of Water and Light, which is 100% publicly owned. Their rates are still some of the lowest in the Midwest. The main downside is that until recently their main energy source was coal. (We used to live downwind of the smokestacks. You couldn't smell it, but lung cancer was definitely in the air.)
MBCook · 1h ago
They could just connect to the rest of the national grid to get power from other states when running short. That would work fine.
Except then they’d be subject to regulations. And we can’t have that now can we?
infecto · 32m ago
Don’t confuse some of Texas’s shortcomings with faults in the free market system for energy. This is my only issue in these conversations. Not everything the Texas market does is bad and they were not the only grid impacted by the weather storm but certainly the most severe because of the lack of interconnects.
dogleash · 1h ago
>Except then they’d be subject to regulations. And we can’t have that now can we?
Why so salty? There are n>0 things the federal government does that I disagree with. On a subject I know nothing about why would I assume that Texas is wrong to avoid it's rules?
macintux · 1h ago
I’d be curious how many people in Texas have died from power loss compared with the rest of the country combined over the last few years.
apendleton · 1h ago
> What is the alternative
Other markets in the US are generally energy + capacity markets -- you get paid both for what you actually provide and for your ability to provide a certain level of power, whereas Texas is an energy-only market (EOM). It needn't be the case that that if you don't do an EOM, you have to have a monopoly.
infecto · 1h ago
Definitely you could operate on a capacity model instead of generation. There are a lot of levers. My issue is mostly around how much uninformed hate a “free market” energy system gets.
lokar · 1h ago
PG&E only runs the grid(s), not generation
The government can structure a public market in many different ways (they do this in many aspects of the economy). It’s not limited to real time auction vs single provider.
infecto · 1h ago
Incorrect. They do have generation but it’s not a majority producer. My point being that PGE still is helping set the rate via the CPUC. They are purchasing power through some of the spot markets. They are unable to even manage their own transmission lines effectively though.
bryanlarsen · 2h ago
This can be a "have your cake and eat it too" situation. California, most of Northern Europe and many other jurisdictions have a power market without this problem. They do this by also establishing a market for capacity and/or reliability.
zdragnar · 2h ago
> California
> without a problem
That doesn't really line up.
bryanlarsen · 1h ago
California's myriad problems are delivery side, not supply side.
floatrock · 1h ago
There's a lot of things to take a critical eye towards on the Texas power market, but
> ...no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market.
...is a bit of an exaggeration.
When Texas has those cold-snap/freeze days a few years ago, wholesale rates went up to $9,000 per megawatt-hour. So $9/kWh. Wholesale.
A large amount of energy suppliers went out of business because they didn't properly hedge for such an event.
You can bet those who are left have started to react to market price signals like that. Whether it's through financial engineering or boots-and-poles engineering is a fair discussion to have, but to say "no allowances for reliability signals" is a bit disingenuous.
When someone says "It's all really very simple...", it's almost certainly not.
mothballed · 2h ago
In a free market Texas could connect to the interstate grid which would average some of the localized reliability issues.
MBCook · 1h ago
It’s because they want their “free market” that they don’t connect to the national grid.
barbacoa · 27m ago
Texas is connected to the Eastern and Mexican grids and can share power through high voltage DC interconnect stations. They simply don't sync to the phase of other grids.
mothballed · 1h ago
Other states/feds were happy to take the "free market" of Texas during WWII when they direly needed it and other Gulf states begged Texas to connect.
It's a holier than thou thing for the other states. "Free market" buying from Texas when they need it, when they don't it's "fuck the free market" and cut texas off via legislation and refuse to provide the same assistance that Texans provided.
lacker · 1h ago
It sounds better than the northern California system where occasionally PG&E will cut off the power of random neighborhoods because the grid is overloaded.
MBCook · 1h ago
Good. If people want to build massive AI data centers to pollute and waste power for theoretical gain propped up by VC money and hype cycles then let them power themselves in an emergency too.
If they’re that important, shouldn’t be a problem right? AI makes so much money right?
People come first. Not virtual hype driven land grabs.
tzury · 1h ago
Hospitals, first responders, and such, are all cloud driven operations nowadays.
It is really strange to see them categorized under
Data centers and other large, non-critical power consumers
criddell · 1h ago
Whose fault is it if hospitals place critical data and services on systems for which the SLA cannot guarantee the service they require?
tzury · 1h ago
Google Cloud in Texas:
Region: us-south1 (Dallas)
Zones: us-south1-a, us-south1-b, us-south1-c
Availability: Dallas, Texas is also listed as a location for Vertex AI.
Azure in Texas:
Region: "South Central US" (paired with "North Central US")
Azure Government: Region in Austin, Texas
Availability: Azure has a physical presence in Texas.
Imagine a vendor picking multi zone, or even Multi cloud...
This sounds like someone was trying to mitigate the "AI is eating our small town electricity" and threw the baby along with the water.
thrance · 1h ago
Aren't there special clouds accredited for such use cases? Ones that probably won't be targeted by this sort of bills?
bee_rider · 2h ago
This seems like the sort of thing that ought to be negotiated into the data center’s contract. Why force the grid operator to make every data center subject to curtailment?
Of course, it is easy enough to deal with curtailment for many services. But, it should be on the table, either way.
lokar · 2h ago
Yeah, if they needed a law it was to say you must have X% of your load subject to a load shed agreement.
thayne · 1h ago
This seems like it will dicincentivize power plants from being built in Texas, which is probably a good thing in the long run.
sschueller · 21m ago
No, it just tells them to run on gas directly like Elon's xAI power plant. [1] No permits needed.
Nobody has mentioned Bitcoin mining, but this sounds like it was developed exactly for that use case. Mining actually helps balance the grid because they can immediately shut it down during high demand.
davidcbc · 56m ago
Yeah, I add a bunch of Sleep statements in my code so that whenever we need a performance boost I can just turn those off.
No comments yet
miltonlost · 42m ago
> Mining actually helps balance the grid because they can immediately shut it down during high demand.
Mining is what is also creating part of that demand, so they're "balancing" by not in the first place working to create their crypto nonsense. To help "balance" they would put more in than take out.
add-sub-mul-div · 46m ago
I partly agree, ideally the worthless crypto/AI data centers can be isolated and shut off separately from data centers hosting random small businesses and web sites etc.
But I'd rather have an unbalanced grid without Bitcoin waste to start with than a balanced grid that can briefly shut down Bitcoin if needed.
Don't grid operators already have this power? That's the sort of thing they do during "rolling blackouts", isn't it?
bloomingeek · 2h ago
This is a terrible idea. We live in the digital age, so almost everything depends on data. As in all mechanical devices, generators have a tendency to either break down(Let's not forget about associated switching gear.) or become subject to maintenance cut backs instituted my buffoons in management. We cannot depend on human errors, we can depend on the electric grid, if properly handled and maintained. Depending on generators just adds another link in the failure chain.
Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.
hermitdev · 2h ago
> This is a terrible idea.
No, it isn't. Any decent datacenter will have on-site generation in event of power grid failure, anyway. When I was an intern, the company I worked for would routinely go off grid during the summer at a call from the electric company. The electric company actually gave us significant incentives to do so, because us running on our own 12MW generator was effectively like the grid operator farming out a 12MW peaker unit.
bluGill · 2h ago
Not only will a data center of a generator, they will test it regularly and if it doesn't work get it fixed.
The power company has a long list of who has backup power. I know of one factory where the generator was installed in the 1920s on a boiler from the 1880's - it is horribly inefficient, but the power company still gives the owners incentive to keep it working because for 4x the normal cost of power and 12 hours notice that generator can run the entire town it is in, which they do every 5 years when things really go wrong with the grid.
dylan604 · 1h ago
> they will test it regularly and if it doesn't work get it fixed.
What is your definition of regularly, and what qualifies as getting it fixed? I know lots of places that had things scheduled, but on the day of, something "came up" that the test was pushed. I've seen others where they tested by only firing up the generator, but didn't actually use it to power the facility. I've also seen repair tags that sat "unlooked" at for years.
Not every facility is managed/financed the same for such a blanket statement as yours.
bluGill · 48m ago
Most places I know of where like that until 'something' happened and now they take a lot more care.
bloomingeek · 1h ago
Exactly! And one of points, based on real work experience, I was trying to make.
scotty79 · 1h ago
There was recent news that a datacenter is going to be built that will consume few times more power than all homes in the state. I don't think they are gonna have on-site backup power. Although they'll probably have on-site powerplant for normal operations.
_verandaguy · 2h ago
> We live in the digital age, so almost everything depends on data
Data that I can't consume if my house is browned out and my router doesn't work (on top of heating/cooling, lights, and other basic living-related services that are less essential than the almighty ONT).
> As in all mechanical devices, generators have a tendency to either break down(Let's not forget about associated switching gear.) or become subject to maintenance cut backs instituted my buffoons in management
Famously, power infrastructure relies on no moving parts whatsoever since the abolition of contactors, relays, rotors (but not stators), turbines (both water and wind), and control rod actuators, though even before abolition, none of these devices needed any maintenance.
> We cannot depend on human errors, we can depend on the electric grid, if properly handled and maintained
The electric grid, which famously has no human or mechanical errors like line sag or weirdly-designed interconnects or poorly-timed load shedding.
> Depending on generators just adds another link in the failure chain.
Weird way to frame a redundancy layer, but sure.
> Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.
Again, weird way to frame this. You're actually technically right about this, but the redundancy offered through a better-integrated interconnect goes both ways, rather than just externalizing weaknesses in TX's own interconnect design.
mothballed · 2h ago
IIRC the reason Texas cannot get 'assistance' from other states is that the feds made it illegal to connect to most interstate grids without following their regulatory regimes. I believe Texas does connect to Mexico and possible some other regional grids although I don't really understand the exemption for those.
In this case it's not really Texas 'not allowing' other states to help but the other states not allowing Texas. Conceivably federal law could be updated to remove those regulations and Texas would absolutely connect to the interstate grid at that point.
RHSeeger · 2h ago
> Conceivably federal law could be updated to remove those regulations and Texas would absolutely connect to the interstate grid at that point.
That, of course, ignores the fact that those regulations are in place for a reason. Texas refuses to play by the rules, and the impact of that is that they don't get help when it's important. It is unfortunate, but a direct consequence of the choices they made.
vel0city · 2h ago
The Texas Interconnection does tie in to surrounding grids through DC-ties. Those are limited in how much power can be sent through them and ultimately isolate the AC frequency.
more_corn · 1h ago
Rules like hardening your system to be resilient in high or low temperatures.
Super abusive. Let’s do away with safety systems that literally save human lives. Heck Texas doesn’t like them so let’s do away with them for the whole nation? How many people died during the last couple heat and cold induced grid outages in Texas? I lost count after a couple dozen. But those people were weak or poor anyway right? Texas strong!
joezydeco · 2h ago
I'm happier with Texas being independent. Why should my state brown out because a bunch of companies put data centers in the hottest part of the continent?
Workaccount2 · 2h ago
Don't put all your digital services in a texas datacenter with no fallback then...
darth_avocado · 2h ago
The whole point of using a cloud data center is to be able to handle grid outages. I’d be using the cabinet under my table for otherwise.
vel0city · 2h ago
> We live in the digital age, so almost everything depends on data
Agreed, the datacenters need to be extremely durable. What's more durable than proving you're able to withstand a power outage event? The grid does go down from time to time; they need to be ready to handle it. That's not a Texas-only kind of thing; power outages happen all over the US.
If the datacenter can't handle the outage that was announced as a probability ahead of time, they have no business running critical applications.
I could see operators of datacenters in Texas wondering about this. Also, it's underrated how much critical infrastructure is dependent on datacenters running. Like, are you going to pull someone's EHR system down that serves a local hospital, while keeping the local hospital on a critical circuit?
Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity.
Some data-centers are indeed on circuits deemed to be critical but I could see regulations changing this so that they are "business critical" vs. "life support critical" and some changes could be made at substations so that data-centers could participate in shedding. I think you are right that they will be thinking about this and adding to this probably filing preemptive lawsuits to protect their business. Such changes can violate SLA contracts businesses have with power companies and Texas is very pro-business so I can not compare it to California.
https://en.wikipedia.org/wiki/2021_Texas_power_crisis
I'm not sure that this is correct. I was initially worried about how Mom would fare since she lives alone and is over 80. During the entire one week period of power problems in Feb. 2021 my Mom never lost power, not even a quick brown-out. Her home is within a half mile of a local hospital which also never lost power. The area around the hospital did not lose power so businesses and homes close by had no issues with heating, cooking, bathing, etc during the cold blast. That fact allowed me to stay here at my place a couple hours away and manage my own situation which was fairly easy compared to many others in the state.
Your other statements are quite true and to date no one who played a part in mismanagement of utility power in Texas has been held accountable nor will they ever be in a libertarian state where regulations exist only to guarantee a profitable situation for a commercial entity. In fact, most electricity customers in Texas ended up paying for the huge cost increases that occurred as those in charge tweaked the system in real time to maximize their own profits.
Texas needs regulations worse than most other states. Grifters, fraudsters, and thieves have filled too many critical positions for too long.
If you know when the power will be cut, you can start the generators before the cut, and depending on your equipment, you may be able to synchronize the generator(s) with the grid and switch over without hitting the batteries. I assume big datacenters are on three phase, can you switch over each phase as it crosses zero, or do you need to do them all at once?
Synchronizing generators is a thing, but it isn't useful for this situation since they need to be able to handle the sudden without warning power loss where generators cannot be synchronized anyway.
I know what you mean though, the generators I worked with in the military had a manual frequency sync that required slowly turning a dial and watching light bulbs that got brighter with frequency offset. Very old equipment for Mystic Star, post-WWII era equipment. 50's to 90's
Particularly the "last mile" often enough is only equipped with batteries to bridge over small brownouts or outages, but not with full fledged diesel engines.
And while hospitals, at least those that deal with operating patients, are on battery banks and huge ass diesel engines... private small practices usually are not, if you're lucky the main server has a half broken UPS where no one ever looked after that "BATTERY FAULT" red light for a year. But the desktop computers, VPN nodes, card readers or medical equipment? If it's not something that a power outage could ruin (such as a MRT), it's probably not even battery backed.
There's a German saying "the emperor is naked, he has no clothes". When it comes to the resilience of our healthcare infrastructure, the emperor isn't just naked, the emperor's skin is rotting away.
How long can most DCs run with just the fuel onhand? Have standards around that changed over time?
The whole area lost power for weeks but gym was open 24/7 which became very busy during that time.
Until there's new management.
You can't run a business by seesaw.
Best to just count on that rule being enforced and place the necessary battery backups and wind or solar in place to backstop the diesel. Then make any users who need to use those data centers eat that extra cost. There's no problem with us-east costing less than us-west, and us-texas costing most of all. That's how markets work.
Not that businesses aren't abusing the process, either. The article mentions that 80-90% of planned data centers won't be built, due to duplicate applications. They duplicate both to secure a "phantom" slot for power, and to get localities to compete with incentives for business. It's hard to plan a grid when financial speculation and gamesmanship is driving the planning.
The worst aspect is that Texas corruption is explicitly being offered as a model for other states in the PJM interconnect.
I wonder how much of the high economic growth rate in the 1950's-1980's came from the lack of gamesmanship and political franchises. But it's probably unrealistic to think professionals could step back from the very maximalist positions that are selecting them as leaders.
https://energyeducation.ca/encyclopedia/Energy_density
https://www.linkedin.com/posts/bjornlomborg_there-are-no-low...
Their "free market" real time power auction makes no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market.
Some things are too important to leave to the "free market".
What is the alternative, have a state imposed monopoly with a single power company like PGE who I would not see as an ideal operator either. Same can be true for a lot of other similar generators across the company.
You’ll probably bring up the winter storm outage which is inexcusable but their neighbor to the north SPP had very similar failings in being prepared and only faired better because they have interconnects.
Texas has had some of the fastest adoption for wind and solar. It is far from perfect but I also think there is benefit to having multiple generation companies supplying to the grid. You have companies with different expertise and perhaps innovation.
When I lived in the capital, we got our power from the Lansing Board of Water and Light, which is 100% publicly owned. Their rates are still some of the lowest in the Midwest. The main downside is that until recently their main energy source was coal. (We used to live downwind of the smokestacks. You couldn't smell it, but lung cancer was definitely in the air.)
Except then they’d be subject to regulations. And we can’t have that now can we?
Why so salty? There are n>0 things the federal government does that I disagree with. On a subject I know nothing about why would I assume that Texas is wrong to avoid it's rules?
The government can structure a public market in many different ways (they do this in many aspects of the economy). It’s not limited to real time auction vs single provider.
> without a problem
That doesn't really line up.
> ...no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market.
...is a bit of an exaggeration.
When Texas has those cold-snap/freeze days a few years ago, wholesale rates went up to $9,000 per megawatt-hour. So $9/kWh. Wholesale.
A large amount of energy suppliers went out of business because they didn't properly hedge for such an event.
You can bet those who are left have started to react to market price signals like that. Whether it's through financial engineering or boots-and-poles engineering is a fair discussion to have, but to say "no allowances for reliability signals" is a bit disingenuous.
When someone says "It's all really very simple...", it's almost certainly not.
It's a holier than thou thing for the other states. "Free market" buying from Texas when they need it, when they don't it's "fuck the free market" and cut texas off via legislation and refuse to provide the same assistance that Texans provided.
If they’re that important, shouldn’t be a problem right? AI makes so much money right?
People come first. Not virtual hype driven land grabs.
It is really strange to see them categorized under
This sounds like someone was trying to mitigate the "AI is eating our small town electricity" and threw the baby along with the water.
Of course, it is easy enough to deal with curtailment for many services. But, it should be on the table, either way.
[1] https://www.climateandcapitalmedia.com/35-gas-turbines-no-pe...
No comments yet
Mining is what is also creating part of that demand, so they're "balancing" by not in the first place working to create their crypto nonsense. To help "balance" they would put more in than take out.
But I'd rather have an unbalanced grid without Bitcoin waste to start with than a balanced grid that can briefly shut down Bitcoin if needed.
And prioritizing humans (home heating, food freezing, etc ) over servers is a good thing.
https://www.spglobal.com/automotive-insights/en/blogs/2025/0...
Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.
No, it isn't. Any decent datacenter will have on-site generation in event of power grid failure, anyway. When I was an intern, the company I worked for would routinely go off grid during the summer at a call from the electric company. The electric company actually gave us significant incentives to do so, because us running on our own 12MW generator was effectively like the grid operator farming out a 12MW peaker unit.
The power company has a long list of who has backup power. I know of one factory where the generator was installed in the 1920s on a boiler from the 1880's - it is horribly inefficient, but the power company still gives the owners incentive to keep it working because for 4x the normal cost of power and 12 hours notice that generator can run the entire town it is in, which they do every 5 years when things really go wrong with the grid.
What is your definition of regularly, and what qualifies as getting it fixed? I know lots of places that had things scheduled, but on the day of, something "came up" that the test was pushed. I've seen others where they tested by only firing up the generator, but didn't actually use it to power the facility. I've also seen repair tags that sat "unlooked" at for years.
Not every facility is managed/financed the same for such a blanket statement as yours.
In this case it's not really Texas 'not allowing' other states to help but the other states not allowing Texas. Conceivably federal law could be updated to remove those regulations and Texas would absolutely connect to the interstate grid at that point.
That, of course, ignores the fact that those regulations are in place for a reason. Texas refuses to play by the rules, and the impact of that is that they don't get help when it's important. It is unfortunate, but a direct consequence of the choices they made.
Super abusive. Let’s do away with safety systems that literally save human lives. Heck Texas doesn’t like them so let’s do away with them for the whole nation? How many people died during the last couple heat and cold induced grid outages in Texas? I lost count after a couple dozen. But those people were weak or poor anyway right? Texas strong!
Agreed, the datacenters need to be extremely durable. What's more durable than proving you're able to withstand a power outage event? The grid does go down from time to time; they need to be ready to handle it. That's not a Texas-only kind of thing; power outages happen all over the US.
If the datacenter can't handle the outage that was announced as a probability ahead of time, they have no business running critical applications.