> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
So they depend on GCP for (some of) their services
its-kostya · 22h ago
If that is true, and there is no other BGP shenanigans, then I suspect this dependency will not be around for long
yencabulator · 18h ago
From the article:
> Workers KV is in the process of being transitioned to significantly more resilient infrastructure for its central store: regrettably, we had a gap in coverage which was exposed during this incident.
beastman82 · 22h ago
My WAG is it comprises 95% of the company infrastructure
IX-103 · 21h ago
I heard that it was a "mandatory dependency" to mitigate "insider risk" or something. There's no way it's going anywhere. Odds are they'll just enforce even slower rollouts "to catch things early".
SahAssar · 7h ago
WAG = wild-assed guess?
tempaccount420 · 5h ago
Maybe beastman82 is wagging his tail?
pizzafeelsright · 21h ago
ceo just said not for long
asteroidburger · 22h ago
Sub-processor pages are an easy way to verify that sort of thing.
It's likely their auth infra based on what the Google outage is
artursapek · 21h ago
Their KV store was definitely down.
devmor · 1d ago
What do you mean by this? The Google outage is a widespread outage of most GCP services.
pageandrew · 1d ago
Google is claiming the root cause is with some of their central IAM services, which would have a cascading effect to the rest of their services.
devmor · 1d ago
Where did you see this information? Was it on a social media channel? I do see the IAM services in the list of affected services in the incident report.
> Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue
ikiris · 1d ago
Scroll up. Its literally in this HN comment section highly upvoted.
devmor · 13h ago
It was not when I posted the reply that you are replying to 2 hours after I posted it.
ikiris · 1d ago
The comment was self explanatory, and no, it wasn't a widespread GCP outage. Most everything was up except for GCS and firebase, and later on identity stuff started causing cascading issues but not when this was posted.
zerd · 23h ago
> it wasn't a widespread GCP outage.
If this wasn't widespread, what is?
Incident affecting API Gateway, Agent Assist, AlloyDB for PostgreSQL, Apigee, Apigee Edge Private Cloud, Apigee Edge Public Cloud, Apigee Hybrid, Cloud Data Fusion, Cloud Firestore, Cloud Logging, Cloud Memorystore, Cloud Monitoring, Cloud Run, Cloud Security Command Center, Cloud Shell, Cloud Spanner, Cloud Workstations, Contact Center AI Platform, Contact Center Insights, Data Catalog, Database Migration Service, Dataform, Dataplex, Dataproc Metastore, Datastream, Dialogflow CX, Dialogflow ES, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Composer, Google Cloud Console, Google Cloud DNS, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud Pub/Sub, Google Cloud SQL, Google Cloud Storage, Google Compute Engine, Identity Platform, Identity and Access Management, Looker Studio, Managed Service for Apache Kafka, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Persistent Disk, Personalized Service Health, Pub/Sub Lite, Speech-to-Text, Text-to-Speech, Vertex AI Search
ikiris · 20h ago
Our entire infra in GCP stayed up just fine, we just couldn't manage anything. IDK what to tell you. Many of the things you list here were not down at all.
mirashii · 18h ago
That it wasn’t down for you does not mean it wasn’t down for others or even almost everyone. Certainly, Google wouldn’t have listed the services as having an outage if nobody was impacted. You can’t extrapolate from “works for me” to “it must have been working for everyone”.
ikiris · 16h ago
Dude, I literally was an SRE there. I'm well aware of how this stuff works.
If some of those things listed had actual widespread outages, it would have been muchmuch worse.
solardev · 11h ago
I don't understand your argument? Wasn't GCP's own status page calling them outages? Some of our upstream providers (who use GCP) were definitely affected and down.
As a former SRE there, is "widespread outage" a specific, special kind of classification that's not obvious to the public just by looking at the status page...? Or what do you mean?
iimblack · 18h ago
So weird to argue when google themselves listed these as having an outage.
neo_doom · 1d ago
Yeah this is going to be a problem. I haven't seen an issue this widespread across so many services in a while.
tete · 1d ago
Seems to be semi regular now that everyone puts all their eggs in only a few baskets.
solardev · 10h ago
I gotta say, it's kinda nice when that happens... work just kinda pauses for everyone, from providers to customers. It kinda feels like a national holiday, and everyone downstream from the affected cloud can just kinda sit back and relax cuz there's nothing they can do anyway except wait.
When it's your own outage, it's all-hands-on-deck panic mode. When it's half the internet down, it's no longer your problem, lol
prauscher · 6h ago
I guess it depends on what your company's acceptable level of downtime is. If you're like Cloudflare (who handled this well), you take this as a sign to build fault tolerance around your 3rd party providers.
If your application is mission-critical, downtime is anything but a holiday.
tete · 1d ago
Big blog post about how they saved the internet upcoming. ;)
Let me guess, someone pushed out a bad BGP config?
CSMastermind · 1d ago
For an outage this large and widespread that would have to be the main culprit.
No comments yet
aranchelk · 1d ago
Seems to be affecting functionality of their "Verify you are human" dialogs as well as Workers.
clairegraham · 1d ago
Yep, KV is broken too. Any worker that depends on KV is throwing exceptions. I was able to get into the dash, but it's very slow. Error rates started to go up significantly around 18:00 UTC.
After many tries I also got into the dashboard, but it's not that usable, constant error pop-ups.
bgwalter · 1d ago
It does. Another question is why do we get these dialogues always from Cloudflare and never from Akamai in the first place?
bgwalter · 1d ago
Downvoting this comment and flagging the submission does not address the serious issue. These verification dialogues make the Internet unusable.
perching_aix · 1d ago
Nor does venting about it in unrelated threads, or asserting your opinion as fact.
scubbo · 1d ago
It's not much of a reach to go from "discussion about impact on human-verification dialogs" to 'discussion about human-verification dialog policy". This isn't an incident-management channel, it's a discussion forum - tangents are fine!
bgwalter · 1d ago
I complained in the apnews.com thread, because the apnews.com verification, which is annoying by itself, did not work at all this time. That is hardly unrelated.
pier25 · 1d ago
They've changed the title to "Broad Cloudflare service outages"
sidcool · 21h ago
Cloudflare's lava lamps are dimming.
PeterStuer · 13h ago
So both Cloudflare authentication as well as Google's identity systems suffered major dowtime yesterday. Are there technical dependecies between these?
tom1337 · 11h ago
Cloudflare doesn't say this directly but in their blog they've written
> The cause of this outage was due to a failure in the underlying storage infrastructure used by our Workers KV service, which is a critical dependency for many Cloudflare products and relied upon for configuration, authentication and asset delivery across the affected services. Part of this infrastructure is backed by a third-party cloud provider, which experienced an outage today and directly impacted availability of our KV service.
ourmandave · 1d ago
Is it coincidence that there's a Scheduled Maintenance in Tokyo for 18:00 UTC in progress, and the problems started at 18:19 UTC?
alexcroox · 1d ago
Unrelated, they have a few services that rely on GCP which is down. Still, I imagine the people working on the maintenance for Tokyo turned white during that job worried it was caused by them...
perching_aix · 1d ago
Guess we'll find out from the postmortem. Always the silver lining with these, get to learn from and enjoy a good writeup.
I really do appreciate the transparency and ownership that comes with these. We all fuck up, but a lot of companies would rather hide their mistakes than own up to them. Cloudflare's approach makes me trust them more.
perching_aix · 1d ago
> Do these get posted publicly?
Yes.
jonfw · 1d ago
There is always scheduled maintenance on that page, so that's not much of a signal in my experience
bhaney · 1d ago
Probably
poorman · 23h ago
Can’t wait to read this post-mortem. Seems odd that a Google Cloud outage would bring down Cloudflare services.
pier25 · 1d ago
Our Workers apps are up again
edit:
It works in the US but EU customers are still reporting our services as down.
edit:
EU customers are reporting ok
iimblack · 1d ago
They updated the incident noting that it's not just authentication affected.
b0a04gl · 1d ago
distributed systems break, that’s the whole point
what actually matters is how fast they localize damage and how invisible that feels to the end user
if kv failing takes down auth, ui, and workers, then failure isolation’s missing
recovery is fine, but if your fix needs global coordination to unbreak local flows, that’s a design smell
not saying perfect uptime, but the post-outage ux should feel smoother, not shakier
right now it feels like the system survived but the interface didn’t
pier25 · 1d ago
Workers KV has been down for like +30mins. This is impacting us seriously.
Their API is down too.
Amazing that something can impact their whole infrastructure like this given how much redundance they have.
> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
I bet that 3rd party service is GCP.
I would be pretty pissed if I were a CF customer that used Workers KV for redundancy because it was heavily marketed as running on CF data centers.
nijave · 1d ago
>can impact their whole infrastructure
CDN and WAF seem to be working fine. I think CF rushed a lot of newer services out without the reliability some of their older/core services enjoy
stri8ted · 1d ago
The same is true for Google.
vimwizard · 1d ago
proxy seems available in general, must just be local to workers because only one of my sites going thru ZT tunnel with identity access rules is affected
ineedaj0b · 1d ago
solar flare?
CoopaTroopa · 1d ago
No, Cloudflare.
joduplessis · 1d ago
Hopefully they also publish the prompt that did this.
daxfohl · 1d ago
They should make the AI lead the postmortem.
tough · 1d ago
i was thinking about this too
vsgherzi · 1d ago
They're just moving fast and breaking things 100x faster. Who cares what code does just vibe it all away /s
So they depend on GCP for (some of) their services
> Workers KV is in the process of being transitioned to significantly more resilient infrastructure for its central store: regrettably, we had a gap in coverage which was exposed during this incident.
https://www.cloudflare.com/gdpr/subprocessors/cloudflare-ser...
Google denies they had any outages.
https://x.com/Google/status/1933246051512644069
https://nitter.net/Google/status/1933246051512644069
Word on the street is that there are large BGP routing issues behind all of this.
At the same time I have not noticed anything being down firsthand. I am in Europe.
[1] https://bishopfox.com/blog/bgp-hijacking-technical-post-mort...
[1] https://www.ihr.live/
> Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue
If this wasn't widespread, what is?
Incident affecting API Gateway, Agent Assist, AlloyDB for PostgreSQL, Apigee, Apigee Edge Private Cloud, Apigee Edge Public Cloud, Apigee Hybrid, Cloud Data Fusion, Cloud Firestore, Cloud Logging, Cloud Memorystore, Cloud Monitoring, Cloud Run, Cloud Security Command Center, Cloud Shell, Cloud Spanner, Cloud Workstations, Contact Center AI Platform, Contact Center Insights, Data Catalog, Database Migration Service, Dataform, Dataplex, Dataproc Metastore, Datastream, Dialogflow CX, Dialogflow ES, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Composer, Google Cloud Console, Google Cloud DNS, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud Pub/Sub, Google Cloud SQL, Google Cloud Storage, Google Compute Engine, Identity Platform, Identity and Access Management, Looker Studio, Managed Service for Apache Kafka, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Persistent Disk, Personalized Service Health, Pub/Sub Lite, Speech-to-Text, Text-to-Speech, Vertex AI Search
If some of those things listed had actual widespread outages, it would have been much much worse.
As a former SRE there, is "widespread outage" a specific, special kind of classification that's not obvious to the public just by looking at the status page...? Or what do you mean?
When it's your own outage, it's all-hands-on-deck panic mode. When it's half the internet down, it's no longer your problem, lol
If your application is mission-critical, downtime is anything but a holiday.
Currently down, but reference: https://blog.cloudflare.com/the-ddos-that-almost-broke-the-i...
No comments yet
Edit: The CF status page has acknowledged it's a broad outage across many services: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
> The cause of this outage was due to a failure in the underlying storage infrastructure used by our Workers KV service, which is a critical dependency for many Cloudflare products and relied upon for configuration, authentication and asset delivery across the affected services. Part of this infrastructure is backed by a third-party cloud provider, which experienced an outage today and directly impacted availability of our KV service.
I really do appreciate the transparency and ownership that comes with these. We all fuck up, but a lot of companies would rather hide their mistakes than own up to them. Cloudflare's approach makes me trust them more.
Yes.
edit:
It works in the US but EU customers are still reporting our services as down.
edit:
EU customers are reporting ok
Their API is down too.
Amazing that something can impact their whole infrastructure like this given how much redundance they have.
> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
I bet that 3rd party service is GCP.
I would be pretty pissed if I were a CF customer that used Workers KV for redundancy because it was heavily marketed as running on CF data centers.
CDN and WAF seem to be working fine. I think CF rushed a lot of newer services out without the reliability some of their older/core services enjoy