Activeloop (YC S18) is hiring a VP of Engineering in Mountain View (on-site) (careers.activeloop.ai)

We've gotten rate-limited out of the blue on clustered development servers in the past 3 years now, this last one was on servers we setup 830 days ago, before we knew that getting rate-limited/banned on DNS servers where even possible. The worst thing about the last incident was that we entered a death spiral, DNS resolution failing started a logging job, that failed (due to DNS resolution failing to call log server) that then started a job about the failing DNS resolution.. You get the gist..

Of course, this is an issue of engineering and code, not only a rate-limiting issue.

However, many developers rely and depend on upstream DNS resolution to "Just Work" when you add it to a server, which has been the case with Googles DNS servers for the past 15+ years that I've been a sysop. I'm just hoping that this time, this will get SOME attention, because either you want dev-ops to use Cloudflare DNS on servers or you don't - and if you don't - there should be an official warning that this WILL happen, you WILL get rate-limited eventually.

Comments (7)

phillipseamore · 9h ago

What kind of volume was this? I have a server that does some rather specific DNS monitoring resulting in millions of unique lookups with 1.1.1.1 a day.

tmikaeld · 8h ago

That's the frustrating part of this and the inconsistency, we're doing benchmarks one day, making thousands of lookups, adding/removing domains, then during normal day operations we're getting blocked.

phillipseamore · 8h ago

Is this only DNS or have issues with accessing CF networks? Do you own the subnet the server is on is it shared with others? Wondering if this is because of other traffic from the subnet and also affects you.

tmikaeld · 7h ago

These are on spread out external IPs (VPSs) so not within CF networks or specific IP subnets. The common denominator is that at certain bursts of traffic, we get blocked.

If this had some kind of pattern we could avoid or improve, I wouldn't even bring it up.

gertop · 10h ago

> However, many developers rely and depend on root DNS resolution to "Just Work" when you add it to a server

As a sysops you're probably aware that neither Google nor CloudFlare are DNS root servers.

Using actual root servers through your own resolver would have avoided this issue. Bind doesn't even need any config for that use case.

Bender · 1h ago

Adding to this data-center servers should at very least use a proper set of caching DNS servers at the edge of their network and those should talk to the root servers as to not add to the global abuse of the anycast clusters. I've seen some companies go as far as to run Unbound on each and every server to improve the retry and caching mechanisms to great success. Unbound can also raise the min TTL as some applications get quite abusive if they use really low TTL's and the applications are making requests for every action. i.e. raising every TTL up to at least 30 seconds if they were lower. The excessive retries can compound really fast especially when applications and systems are not properly configured which is often the case. If people are not sure what I mean, run a UDP capture at the edge of your network and one may find that for each request an application makes there could be as many as 12 DNS requests. It adds up very fast.

    App1 Unbound -> Data-center edge Unbound instances [1-4] -> Root DNS  Anycast clusters

Unbound can be configured to either pick the fastest resolvers or to cycle through all of them and retry in the background when one fails to re-select it when it starts resolving again. This avoids a lot of outages otherwise known as "It's always DNS".

tmikaeld · 9h ago

Of course, it depends on the use-case, what I meant was "upstream DNS". I've edited.

Artifact (YC W25) Is Hiring (ycombinator.com)

Thunder Compute (YC S24) Is Hiring a C++ Low-Latency Systems Developer (ycombinator.com)

GovEagle (YC W23) Is Hiring (ycombinator.com)

Motion (YC W20) Is Hiring a Senior Engineers (jobs.ashbyhq.com)

Tabular (YC S24) Is Hiring (ycombinator.com)

Continue (YC S23) is hiring software engineers in San Francisco (ycombinator.com)

Instant (YC S22) Is Hiring a Founding TypeScript Engineer (instantdb.com)

Jiga (YC W21) Is Hiring Engineers (workatastartup.com)

KaiPod Learning (YC S21) Is Hiring VP of Engineering (ycombinator.com)

Hightouch (YC S19) Is Hiring (ycombinator.com)

Helpcare AI (YC F24) Is Hiring (docs.google.com)

Stellar Sleep (YC S23) is hiring a product engineer in SF (ycombinator.com)

OneText (YC W23) Is Hiring a DevOps/DBA Lead Engineer

Toma (YC W24) Is Hiring Engs #3-4 (AI for Automotive) (ycombinator.com)

Waypoint Transit (YC W25) is hiring a software engineer (workatastartup.com)

GroMo (YC W21) Is Hiring (ycombinator.com)

Archil (YC F24) Is Hiring a Distributed Systems Engineer (In-Person, SF)

Modern Realty (YC S24) Is Hiring (workatastartup.com)

Hestus, Inc. (YC S24) Is Hiring an ML Engineer to Revolutionize CAD (ycombinator.com)

Activeloop (YC S18) is hiring a VP of Engineering in Mountain View (on-site) (careers.activeloop.ai)

Optery (YC W22) – Engineering Team Lead and Engineers with Node.js (U.S., Latam) (jobs.ashbyhq.com)

Extend (YC W23) is hiring engineers to build LLM document processing (jobs.ashbyhq.com)

Parity (YC S24) is hiring founding engineers to build an AI SRE (in-person, SF) (ycombinator.com)

Freshpaint (YC S19) is hiring back end and front end engineers (Remote, US only)

MobileBoost (YC S21) Is Hiring a Founding Back End/Platform Engineer (Remote) (ycombinator.com)

Gym Class (YC W22) Is Hiring Character Animation Engineering Lead (ycombinator.com)

Foundry (YC F24) is hiring – Come build a world model for the web

Bild AI (YC W25) is hiring a founding engineer in SF (ycombinator.com)

Tenjin (YC S14) Is Hiring a Senior Ad Attribution Engineer (Ruby, Go) (ycombinator.com)

Onyx (YC W24) Is Hiring for ML Engineer (ycombinator.com)

Recover (YC W21) Is Hiring (ycombinator.com)

Don't use Cloudflares 1.1.1.1 on servers

Comments (7)