Cloudflare 1.1.1.1 Incident on July 14, 2025

If you were using systemd-resolved however, it retries all servers in the order they were specified, so it's important to interleave upstreams.

Using the servers in the above example, and assuming IPv4 + IPv6:

    1.1.1.1
    2001:4860:4860::8888
    9.9.9.9
    2606:4700:4700::1111
    8.8.8.8
    2620:fe::fe
    1.0.0.1
    2001:4860:4860::8844
    149.112.112.112
    2606:4700:4700::1001
    8.8.4.4
    2620:fe::9

Also note that Quad9 is default filtering on this IP while the other two or not, so you could get intermittent differences in resolution behavior. If this is a problem, don't mix filtered and unfiltered resolvers. You definitely shouldn't mix DNSSEC validatng and not DNSSEC validating resolvers if you care about that (all of the above are DNSSEC validating).

dnsmasq with a list of smaller trusted DNS providers sounds perfect, as long as it is not considered bad etiquette to spam multiple DNS providers for every resolution?

But where to find a trusted list of privacy focused DNS resolvers. The couple I tried from random internet advice seemed unstable.

I need to do a write-up one day

Unfortunately, the configuration mistake that caused this outage disabled Cloudflare's BGP advertisements of both 1.1.1.0/24 and 1.0.0.0/24 prefixes to its peers.

Btw, I really don't understand why it does not accept an IP (1.1.1.1), so you have to give an address (one.one.one.one). It would be more sensible to configure a DNS server from an IP rather than from an address to be resolved by a DNS server :/

If your device doesn't support proper failover use a local DNS forwarder on your router or an external one.

In Switzerland I would use Init7 (isp that doesn't filter) -> quad9 (unfiltered Version) -> eu dns0 (unfiltered Version)

European users might prefer one of the alternatives listed at https://european-alternatives.eu/category/public-dns over US corporations subject to the CLOUD act.

From the longer term graphs it looks like volume returned to normal https://imgur.com/a/8a1H8eL

Cloudflare has a reasonable culture around incident response, but it doesn't incentivize proactive prevention.

Traditional monitoring systems like Nagios and Icinga have settings where they only open events/alerts if a check failed three times in a row, because spurious failed checks are quite common.

If you spam your operators with lots of alerts for monitoring checks that fix themselves, you stress the unnecessarily and create alert blindness, because the first reaction will be "let's wait if it fixes itself".

I've never operated a service with as much exposure as CF's DNS service, but I'm not really surprised that it took 8 minutes to get a reliable detection.

Let's say you've got a metric aggregation service, and that service crashes.

What does that result in? Metrics get delayed until your orchestration system redeploys that service elsewhere, which looks like a 100% drop in metrics.

Most orchestration take a sec to redeploy in this case, assuming that it could be a temporary outage of the node (like a network blip of some sort).

Sooo, if you alert after just a minute, you end up with people getting woken up at 2am for nothing.

What happens if you keep waking up people at 2am for something that auto-resolves in 5 minutes? People quit, or eventually adjust the alert to 5 minutes.

I know you often can differentiate no data and real drops, but the overall point, of "if you page people constantly, people will quit" I think is the important one. If people keep getting paged for too tight alarms, the alarms can and should be loosened... and that's one way you end up at 5 minutes.

It’s corporate newspeak. “legacy” isn’t a clear term, it’s used to abstract and obfuscate.

> Legacy components do not leverage a gradual, staged deployment methodology. Cloudflare will deprecate these systems which enables modern progressive and health mediated deployment processes to provide earlier indication in a staged manner and rollback accordingly.

I know what this means, but there’s absolutely no reason for it to be written in this inscrutable corporatese.

But, yes, a second mitigation here would be defense in depth - in an ideal world, all your systems use the same ops/deploy/etc stack, in this one, you probably want an extra couple steps in the way of potentially taking a large public service offline.

Anecdotally, I figured out their DNS was broken before it hit their status page and switched my upstream DNS over to Google. Haven't gotten around to switching back yet.

Clients cache DNS resolutions to avoid having to do that request each time they send a request. It's plausible that some clients held on to their cache for a significant period.

That said, it's a good idea to specifically pick multiple resolvers in different regions, on different backbones, using different providers, and not use an Anycast address, because Anycast can get a little weird. However, this can lead to hard-to-troubleshoot issues, because DNS doesn't always behave the way you expect.

My Pi-holes both use OpenDNS, Quad9, and CloudFlare for upstream.

Most of my devices use both of my Pi-holes.

[0] https://github.com/folbricht/routedns

Last 30 days, 8.8.8.8 has 99.99% uptime vs 1.1.1.1 has 99.09%

For me cloudflare 1.1.1.1 and 1.0.0.1 have a mean response time of 15.5ms over the last 3 months, 8.8.8.8 and 8.8.4.4 are 15.0ms, and 9.9.9.9 is 13.8ms.

All of those servers return over 3-nines of uptime when quantised in the "worst result in a given 1 minute bucket" from my monitoring points, which seem fine to have in your mix of upstream providers. Personally I'd never rely on a single provider. Google gets 4 nines, but that's only over 90 days so I wouldn't draw any long term conclusions.

The theory is CF had the capacity to soak up the junk traffic without negatively impacting their network.

The Cache API is a standard taken from browsers. In the browser, cache.delete obviously only deletes that browser's cache, not all other browsers in the world. You could certainly argue that a global purge would be more useful in Workers, but it would be inconsistent with the standard API behavior, and also would be extraordinarily expensive. Code designed to use the standard cache API would end up being much more expensive than expected.

With all that said, we (Workers team) do generally feel in retrospect that the Cache API was not a good fit for our platform. We really wanted to follow standards, but this standard in this case is too specific to browsers and as a result does not work well for typical use cases in Cloudflare Workers. We'd like to replace it with something better.

That said, I don't use workers and don't plan to. I personally try to stay away from non cross-platform stuff because I've been burned too heavily with vendor/platform lock-in in the past.

Like mentioned by other comments, do it on your own if you are not happy with the stability. Or just pay someone to provide it - like your ISP..

And TBH I trust my local ISP more than Google or CF. Not in availability, but it's covered by my local legislature. That's a huge difference - in a positive way.

If you have a more advanced local resolver of some sort (systemd for example) you can configure whatever behaviour you want.

Yes.

> Shouldn’t the fix be to ensure that these are served out of completely independent silos [...]?

Yes.

> If so, why were they on the same infrastructure?

Apparently, they weren’t independent enough: something in CF has announced both addresses and that got out.

The solution for the end user is, of course, to use 1.1.1.1 and 8.8.8.8 (or any other combination of two different resolvers).