When this happens in Canada (usually a Rogers outage), the other networks seem to have a hard time dealing with the extra load.
wkat4242 · 19h ago
Only for roaming customers though. Here in Europe a customer in their home country can only use their home network unless they're calling emergency services. Only when roaming multiple options would be available.
So I wouldn't expect all that much extra load really.
wdb · 9h ago
Never have signal with EE at home barely 1 bar. It's ridiculous and I don't live in a flat or something.
drrob · 1d ago
I'm on Vodafone, I can confirm they're okay.
goodcanadian · 1d ago
I think O2 is OK. My phone company is not O2, but it uses their network.
tim333 · 1d ago
I had an annoying O2 fail on me incident at about 6pm.
alephnerd · 20h ago
> O2: Unknown
Isn't that normal for O2? /s
wkat4242 · 19h ago
So it's like three is in Ireland :)
alephnerd · 19h ago
At least you can drown your network sorrows in (relatively) affordable pints of Guinness and the healthy craft beer scene.
Separate note, but I am astonished by how expensive London is - I can pay engineers Bangalore level salaries but they have to deal with Chicago level CoL.
wkat4242 · 9h ago
Ireland's the same now :( Cost of living is insane, especially rent. It's still aftershocks from the financial crash in 2007, the whole economy was leaning on the housing development industry and that has never recovered. So there's a huge backlog in new developments.
yabones · 1d ago
I wonder if it will be the same cause as the big Rogers outage in '22, a good old fashioned BGP botch.
Diff with Rogers is that they took out their entire network: cellular, home/biz internet, home phone, corporate circuits (including MPLS links), most cable TV, a bunch of their broadcast radio (AM/FM) network just dead dead dead.
Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.
Apparently the workaround was to remove/disable your SIM and hope another network has a stronger signal.
Oh, and the CTO was on holiday and had no idea for a while because… their phone was on roaming with Rogers and therefore dead.
I wonder if Rogers still does planned-in-advance multi-stage potentially-enterprise-breaking updates on Fridays
ecshafer · 1d ago
In a financial company I worked at we would do some of the biggest, riskiest changes at 5pm on a Friday (or Saturday evening if we were worried about impacting international trades). The logic being that we would have the most time to fix things before markets open monday.
NikolaNovak · 1d ago
Our release window is Saturday morning. All the support people are on, most users are not, gives us 36hrs.
We absolutely do not release during week if we can help it. But we are traditional ERP application so pretty much everything we do is contrary to the HN/modern zeitgeist :-)
signal11 · 1d ago
Being able to release safely during the week is super important for eg financial services, for fairly obvious reasons.
In trading and market making contexts for instance, we release 100s of times a day — including Fridays. This includes bog-standard infra changes like roleswaps and server rebuilds. The releases that happen on weekends tend to be highly disruptive infra changes, eg unexpected changes to some kind of physical connectivity where we’re not comfortable with carrying weekday risk.
We didn’t explicitly set to to optimise QoL for engineers (the real driver for safe intraday change was being responsive as a business) but not usually being on call on weekends was a big plus.
Scoundreller · 22h ago
> All the support people are on
Either Rogers thinks it can’t make mistakes, or nobody was informed that a potential enterprise breaking change was taking place.
Took some hours to get any official indication from them (partly because of their employees dependency on the service they provide).
Scoundreller · 22h ago
Well this was like 6AM Friday Eastern time, soooo…
atemerev · 1d ago
Yes. It is either Friday evening or Sunday in finance.
dlenski · 1d ago
The '22 Rogers outage, hah. As I recall it didn't affect me at all since I was at home and work in Vancouver all day… but it was a great excuse for not responding to workplace on-call messages which I got in the evening
> Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.
Didn't know that part, amazing.
It sounds kind of like connecting to a WiFi access point which has a broken/non-working uplink to the Internet. Modern smartphones pretty much automatically detect and avoid such APs, and indeed the whole SSID if they need to, but it sounds like the stuck-in-1985 2G baseband layer has no equivalent connectivity check.
No comments yet
whycome · 1d ago
I know one outcome of it was to ensure that they were equipped with SIMs for
Competitor networks just in case
dlenski · 1d ago
You're saying that Rogers personnel now have non-Rogers SIM cards?
Scoundreller · 22h ago
I’m surprised the CTO on holiday didn’t have a sat phone.
eqvinox · 15h ago
I'm in the industry, though not a CTO. Nobody [that I know, on the tech side] has a sat phone.
xp84 · 1d ago
I mean, incredibly critical personnel probably should be! There may only be a few dozen such people, but I wouldn't want the added chaos caused in the event of a Rogers outage if I couldn't get in touch with the key decision makers and most critical operations engineers because of the very outage they're meant to fix. And in the e-sim era that is hopefully very cheap and without any real downsides.
addandsubtract · 1d ago
> their phone was on roaming with Rogers and therefore dead
I thought your phone uses all available networks (ie the strongest one) while roaming. Is that not the case?
g_p · 1d ago
When roaming, your home network is needed for routing incoming calls to you, and handling authenticating your device to the visited network.
Reminds me of recent outages in Russia due to buggy rollouts of Great Russian Firewall aka Sovereign Internet. Were there any state-level infrastructure updates planned recently?
jibbit · 1d ago
age verification starts tomorrow
codedokode · 1d ago
We also have mobile internet disabled/throttled sometimes when there are drone attacks or large international forums. Weak-minded people with Internet dependency like to complain about this online as if their online game is more important than an international forum.
beagle3 · 1d ago
Serious question: who gets to decide that some international forum is more important than residents’ use? - be it games, video calls, or whatever else.
codedokode · 1d ago
The government has an authority to decide, according to the laws? By the way they also often block roads for security of important foreign guests and cause lot of traffic jams.
beagle3 · 23h ago
Well, yes. It’s the same kind of issue. AFAIK, these things are decided opaquely and do not follow any formal rules - which would make them ripe for abuse.
The GGP seemed to be ridiculing “those gamers who think their ping is more inportant”, but there is no way for a person to tell if the network limits, or road blocks for that matter, are reasonable or an instance of corruption.
abdulhaq · 12h ago
they also restrict major roads like the M4 for the benefit of the Olympic committee
situationista · 1d ago
Related to current Starlink outage? probably not, but interesting coincidence
webprofusion · 15h ago
Thought that as well.
rwmj · 1d ago
My BT landline doesn't even have a dial tone at the moment, which is a new one. Internet via Openreach (as the fibre provider) is OK.
18172828286177 · 1d ago
It’s weird to me that the providers aren’t communicating to customers about this. What if you were waiting for a call from a doctor, or similar?
369548684892826 · 1d ago
They've probably let everyone know by SMS, we'll get the message when everything starts working again
Marsymars · 22h ago
Seems to be the norm, unfortunately. I have a day’s worth of emails that were never delivered a few weeks ago due to an issue with Apple’s Hide My Email service, and AFAIK there hasn’t been any statement from Apple on the matter.
No comments yet
a2128 · 1d ago
The on-call person couldn't get the call due to an outage :(
AndrewThrowaway · 8h ago
But the network is not working!
gandalfian · 1d ago
Data is working on Vodafone mvno. Can't call Out or text, Can't make calls On o2 or EE either. Edit EE working. Edit all mobiles now seem to be working OK.
MattPalmer1086 · 1d ago
I haven't noticed anything today and heard no news about it. Must only affect some parts of the network.
Posting this from a phone on the Three network.
jonathantf2 · 1d ago
It's just inbound calls to EE numbers (so if you've ported in you're not affected)
g_p · 1d ago
Even with a ported number, inbound call routing still heavily relies on the "number range" owner to direct the incoming call to the correct network.
If the original number range owner has their subscriber database go down, they can't do the lookup for the network to direct the incoming call towards, so it can cause disruption. The same is true if the incoming signalling endpoints are unavailable, as the incoming call requests won't be responded to.
jonathantf2 · 1d ago
Tis what I meant - I have an EE MVNO SIM but originally an O2 number and I can recieve calls just fine.
mjpa86 · 1d ago
I'm on EE with a ported in number and didn't get a call I should have earlier...
chmod775 · 1d ago
> A map showing reports of EE outage reports made to DownDetector suggests that those in London, Birmingham, Manchester and Glasgow are the worst affected.
No. Those are the most densely populated areas of the UK - obviously they appear as bright red spots on the map.
Sitting at a table in a restaurant in London with some family and O2, ER and Three are fine.
fecal_henge · 1d ago
Stop looking at your phone.
crinkly · 1d ago
Everyone went for a piss.
aftbit · 1d ago
Now we're all just taking the piss.
geocar · 1d ago
At the same time?
sitkack · 1d ago
They are talking about the OP while doing coke.
spauldo · 1d ago
Your family doesn't? That's weird.
wut42 · 1d ago
Another Down Detector bullshit article.... it's getting incredibly tiring. Every time a provider (Phone, Internet or even cloud services) suffer issues ALL of them are reported as down.
raverbashing · 1d ago
Who downs detector the down detector? Or even better who is the redundancy for dd if the site is actually up?
racedude · 1d ago
Uh oh big broken now
andy_ppp · 1d ago
Honestly I know this sound snarky but it’s 100% true - Three has become so unbelievably bad in East London lately I’d struggle to know if it was affected by this outage or if it was business as usual for them. I could go on about how broken their billing and app and site were but meh… need to change provider.
You reminded me that a couple of jobs ago (I tend to stay with jobs for a long time) we discovered that a local paging tower had failed because one of our techs could not receive pages at home. Our assumption was that we were basically the last people using their service in the area.
heraldgeezer · 1d ago
Is this voice only, data only, or voice and data?
2G,3G,4G,5G?
For voice, is CS down, VoLTE down or both?
Article is not clear on this, but I mainly see voice call complaints?
kerv · 1d ago
are they powered by starlink?
te_chris · 1d ago
EE ok in Ldn
tahoeskibum · 1d ago
And now Starlink is down as well. I wonder if they are related.
jon-wood · 1d ago
I see we’re still using Down Detector as a source for stories, which may as well be called People Are Talking on Twitter Detector. It doesn’t do anything smart, it’s just looking for keywords on social media, sometimes by coincidence this indicates an actual outage.
Or are there at all. AWS status page likes to be fully green, even during issues...
nailer · 1d ago
Was about to say the same. So many of my colleagues looki] for officially confirmed statuses if a cloud provider, AI, blockchain or github starts acting funny and troubleshooting problems thinking it's their own systems. My response is: did you check for people talking on X? About half an hour later there's an official incident.
dave78 · 1d ago
Yeah, it's far from ideal, but in my experience its accuracy is better than most anything else readily available, including the official status pages maintained by most tech companies.
jon-wood · 13h ago
It's fine in simpler cases but as we're seeing here, is absolutely useless at providing information on failures in complex systems. It appears that it was one network suffering issues but this was then reported as "EVERY MOBILE NETWORK IN THE UK IS DOWN" because people just go "I tried to make a call and it didn't connect". That could be anything from a single cell tower being hit by a truck to a nationwide power outage.
user_7832 · 1d ago
Yeah, and not only do you get to see if it's down or not (reddit infamously always says it's up even when there are issues), but you also get to see the raw data of reports. Ofttimes I've seen the trend go up and realized it's a very recent issue - even before downdetector itself recognizes it as such.
Human reading > DD reading >> "All our services are operational" when they're absolutely f--ing not.
avalys · 1d ago
Actually it’s quite smart! Probably the smartest possible solution to the problem they tried to solve.
“Smart” doesn’t have to mean complex and technically sophisticated.
NitpickLawyer · 1d ago
Google can detect flu break-outs much faster than the CDC for example, because people tend to search for symptoms before they let anyone know officially, visit a doctor, etc.
teeray · 1d ago
Down Detector will be useful as long as marketing departments control official status pages.
ElijahLynn · 1d ago
People submit reports through down detector.
whalesalad · 1d ago
Do you have a better suggestion? This is the best we can get, without having a public protocol for asking any service, "are you alive?"
mmsc · 1d ago
China watching with open eyes.
This will happen the day that they try to take Taiwan, worldwide, in my opinion.
aftbit · 1d ago
I do think there's a better than average chance that WW3 starts not with an open nuclear exchange as our parents imagined, but instead with a substantial cyberattack which shuts down power, water treatment, communications, hospitals, public transportation, etc. This might even be deniable / grey zone for a few hours or days while the belligerent parties use the chaos to accomplish some Blitzkrieg style attacks.
hollerith · 1d ago
If that were possible, then why didn't Russia, which might have the most experienced pool of cyber-attack skills in the world, do it to Ukraine?
aftbit · 8h ago
They did, to some extent, though not to the degree I described.
As an American, I would be more worried about China than Russia though. They makes a lot of our hardware and firmware, giving them plenty of chances to embed killswitches and zero-days. They have possibly the most successful industrial espionage program in the world, giving them the opportunity to find vulns in other systems and embed agents inside critical platforms. They have deeply internalized the concept of fighting where their enemy is weakest not where they are strongest, so they have likely invested in attacking the American military at home rather than on the field.
ifwinterco · 1d ago
They did try, but ukraine were suspicious something was going to happen (massive army suddenly forming near their border) and they had spent time securing stuff
cess11 · 1d ago
They'd also done it on and off for like a decade already so everyone was used to it and had mitigations in place.
sim7c00 · 1d ago
it would get them nothing.
they did attack satcom systems to the point of bricking them.
what do you think would happen if you turn off critical infra for a country?
mass civilian death/suffering. military likely hardly affected but extremely motivated...
its counter productive.
aftbit · 8h ago
That doesn't seem to be stopping Russia from engaging in the energy war, where they're "wasting" incredibly valuable and rare precision missiles attacking substations and power generation.
So far, this has been only moderately successful in impacting the military, because most of Ukraine's military production is not actually located in Ukraine, and because they've gotten quite good at repairing their electrical grid.
This was generally true of allied strategic bombing campaigns in WWII as well. Simple adaptations like building walls between sections of the plant to require more bombs to attack any given target, hardened shelters for skilled employees, and staging parts outside of the plant enabled some targets to maintain better than 50% uptime during aggressive bombing campaigns. Look into the Oil campaign[1] for more details.
It is interesting to see how precision missiles and cheap drones may change this in the future.
That's stage 2. Stage 1 is election fixing, subversion and capture of foreign governments through bribery and kompromat.
messe · 1d ago
Yeah that's why we're expecting stage 2 next.
pmontra · 1d ago
Only if they absolutely need it. Nobody would spoil an asset like that. Maybe they would turn off mobile in Taiwan if they control their network. I didn't check which technology provider they use.
nailer · 1d ago
I don't think it would permanently spoil anything. China would just stop the phone network from being used to coordinate a response.
pmontra · 1d ago
No mobile network can be an inconvenience but who has to respond will have the means to communicate no matter what. Furthermore every single common person will feel a personal level of danger and they won't simply shrug about the destiny of a remote island somewhere on the map.
nailer · 9h ago
The people we are talking about live in Taiwan, so Taiwan is not a remote island for them.
rwmj · 1d ago
Why? They'll take down BT landlines in case I need to call my auntie and tell her about the news?
exe34 · 1d ago
tbh breaking public internet/phones will probably be done by a local government trying to do something nasty. Stop people coordinating and turning up places they might get in the way.
I think my tin foil hat was askew. There. All better.
https://www.bbc.co.uk/news/articles/cnvmvqrnq7go
> A spokesperson from BT, which owns EE, apologised and said the firm was "currently addressing an issue impacting our services".
> Vodafone and Three have confirmed to the BBC they do not have network issues.
So I wouldn't expect all that much extra load really.
Isn't that normal for O2? /s
Separate note, but I am astonished by how expensive London is - I can pay engineers Bangalore level salaries but they have to deal with Chicago level CoL.
https://www.reuters.com/business/media-telecom/rogers-commun...
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.
Apparently the workaround was to remove/disable your SIM and hope another network has a stronger signal.
Oh, and the CTO was on holiday and had no idea for a while because… their phone was on roaming with Rogers and therefore dead.
I wonder if Rogers still does planned-in-advance multi-stage potentially-enterprise-breaking updates on Fridays
In trading and market making contexts for instance, we release 100s of times a day — including Fridays. This includes bog-standard infra changes like roleswaps and server rebuilds. The releases that happen on weekends tend to be highly disruptive infra changes, eg unexpected changes to some kind of physical connectivity where we’re not comfortable with carrying weekday risk.
We didn’t explicitly set to to optimise QoL for engineers (the real driver for safe intraday change was being responsive as a business) but not usually being on call on weekends was a big plus.
Either Rogers thinks it can’t make mistakes, or nobody was informed that a potential enterprise breaking change was taking place.
Took some hours to get any official indication from them (partly because of their employees dependency on the service they provide).
> Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.
Didn't know that part, amazing.
It sounds kind of like connecting to a WiFi access point which has a broken/non-working uplink to the Internet. Modern smartphones pretty much automatically detect and avoid such APs, and indeed the whole SSID if they need to, but it sounds like the stuck-in-1985 2G baseband layer has no equivalent connectivity check.
No comments yet
I thought your phone uses all available networks (ie the strongest one) while roaming. Is that not the case?
The GGP seemed to be ridiculing “those gamers who think their ping is more inportant”, but there is no way for a person to tell if the network limits, or road blocks for that matter, are reasonable or an instance of corruption.
No comments yet
Posting this from a phone on the Three network.
If the original number range owner has their subscriber database go down, they can't do the lookup for the network to direct the incoming call towards, so it can cause disruption. The same is true if the incoming signalling endpoints are unavailable, as the incoming call requests won't be responded to.
No. Those are the most densely populated areas of the UK - obviously they appear as bright red spots on the map.
What you have is essentially a population map: https://xkcd.com/1138/
2G,3G,4G,5G?
For voice, is CS down, VoLTE down or both?
Article is not clear on this, but I mainly see voice call complaints?
Human reading > DD reading >> "All our services are operational" when they're absolutely f--ing not.
“Smart” doesn’t have to mean complex and technically sophisticated.
This will happen the day that they try to take Taiwan, worldwide, in my opinion.
As an American, I would be more worried about China than Russia though. They makes a lot of our hardware and firmware, giving them plenty of chances to embed killswitches and zero-days. They have possibly the most successful industrial espionage program in the world, giving them the opportunity to find vulns in other systems and embed agents inside critical platforms. They have deeply internalized the concept of fighting where their enemy is weakest not where they are strongest, so they have likely invested in attacking the American military at home rather than on the field.
they did attack satcom systems to the point of bricking them.
what do you think would happen if you turn off critical infra for a country?
mass civilian death/suffering. military likely hardly affected but extremely motivated...
its counter productive.
So far, this has been only moderately successful in impacting the military, because most of Ukraine's military production is not actually located in Ukraine, and because they've gotten quite good at repairing their electrical grid.
This was generally true of allied strategic bombing campaigns in WWII as well. Simple adaptations like building walls between sections of the plant to require more bombs to attack any given target, hardened shelters for skilled employees, and staging parts outside of the plant enabled some targets to maintain better than 50% uptime during aggressive bombing campaigns. Look into the Oil campaign[1] for more details.
It is interesting to see how precision missiles and cheap drones may change this in the future.
1: https://en.wikipedia.org/wiki/Oil_campaign_of_World_War_II
I think my tin foil hat was askew. There. All better.