Interesting site.
It would be good to differentiate between Denial of Service (DoS) and Denial of Wallet (DoW).
The contributors are running into one of the many common tradeoffs of Serverless, an inability to set spending caps.
I wouldn’t go along with the last line conclusion of the original post - “done with cloud”.
It might be old fashioned, but I really like Infrastructure as a Service (IaaS) rather than Platform as a Service (PaaS). Spin up cloud compute hosts as required, and avoid anything platform specific as much as possible. It’s easy to end up married to AWS, GCP or Azure because the cost of a migration project is always a bit too high to pay off.
My use cases might be smaller scale, but I currently like using Terraform and Ansible with EC2. My Terraform setup is quite specific to the AWS environment (EC2 hosts, VPC, subnets, etc) and would require modification if I jumped ship, but much of my detail is in the Ansible playbooks, which simply expect Debian Stable host(s) to configure with my chosen stack. I currently front all that with Cloudflare, and use the cache and built-in bot protections along with rate limits and WAF rules to try to mitigate against abuse that could run up a bill (such as excessive egress).
sshine · 30m ago
I'm in this boat also.
Being able to migrate at low cost (and in little time) should be a high priority for any software infrastructure with availability expectations.
Especially with providers with very reasonable bandwidth costs (~€1.19/TB at Hetzner) beyond the first 20TB.
I also use Terraform, and it does lock you to one provider. Switching provider means switching the complete set of resources. They probably have the fundamental concepts in place (a DNS record, a VPS), but things diverge somewhat quickly, e.g. when it comes to virtual networking, SSH key management, external storage.
Keeping a redeployment script to a secondary cloud in your back hand means you don't have to sit and translate Terraform once the shit hits the fan.
I'm currently migrating my Terraform to Terranix, which just replaces the HCL with Nix expressions; since they both compile to JSON, using Nix lets you generalise across providers at the cost of a more complex language. But you also get things like arbitrary conditionals, e.g. "if X then provision Y else provision Z".
Ansible sounds neat for bootstrapping the servers, but it runs a little short on the ongoing maintenance because it isn't declarative, so you get runaway state divergences.
Also, sorry for hijacking this thread to talk about our lord and savior. :-D
huksley · 3h ago
Would be quite difficult to migrate to self-host, there is no open source version of Firebase APIs, and while there is alternatives like Supabase, it is a significant rewrite of both frontend and backend.
steveharman · 5h ago
It does seem crazy that the default state for Firebase, CloudFlare et.al is not to alert the account owner when out-of-the-ordinady usage charges hit, say $5k above "the norm". We're not talking a service interruption, just a simple heads-up email to confirm that all is well.
WalterGR · 6h ago
How does the average person recover from this? One screw up and it’s bankruptcy?
poly2it · 1h ago
Not sure what their finances are looking like, but they seem to have had a premium service with paying users.
The service author said this in regard to their bill:
> GCP seems to finally be budging with regard to the bill. They acknowledged the DDoS and are running it through the bureaucracy. I do have some confidence that they'll make this right, but I took destructive actions to stop the charges (deleting buckets). I did have a mostly complete backup of customer data on another cloud, but this has destroyed small business side hustle, where I built a community of over 100,000 users over seven years.
The contributors are running into one of the many common tradeoffs of Serverless, an inability to set spending caps.
I wouldn’t go along with the last line conclusion of the original post - “done with cloud”.
It might be old fashioned, but I really like Infrastructure as a Service (IaaS) rather than Platform as a Service (PaaS). Spin up cloud compute hosts as required, and avoid anything platform specific as much as possible. It’s easy to end up married to AWS, GCP or Azure because the cost of a migration project is always a bit too high to pay off.
My use cases might be smaller scale, but I currently like using Terraform and Ansible with EC2. My Terraform setup is quite specific to the AWS environment (EC2 hosts, VPC, subnets, etc) and would require modification if I jumped ship, but much of my detail is in the Ansible playbooks, which simply expect Debian Stable host(s) to configure with my chosen stack. I currently front all that with Cloudflare, and use the cache and built-in bot protections along with rate limits and WAF rules to try to mitigate against abuse that could run up a bill (such as excessive egress).
Being able to migrate at low cost (and in little time) should be a high priority for any software infrastructure with availability expectations.
Especially with providers with very reasonable bandwidth costs (~€1.19/TB at Hetzner) beyond the first 20TB.
I also use Terraform, and it does lock you to one provider. Switching provider means switching the complete set of resources. They probably have the fundamental concepts in place (a DNS record, a VPS), but things diverge somewhat quickly, e.g. when it comes to virtual networking, SSH key management, external storage.
Keeping a redeployment script to a secondary cloud in your back hand means you don't have to sit and translate Terraform once the shit hits the fan.
I'm currently migrating my Terraform to Terranix, which just replaces the HCL with Nix expressions; since they both compile to JSON, using Nix lets you generalise across providers at the cost of a more complex language. But you also get things like arbitrary conditionals, e.g. "if X then provision Y else provision Z".
Ansible sounds neat for bootstrapping the servers, but it runs a little short on the ongoing maintenance because it isn't declarative, so you get runaway state divergences.
Also, sorry for hijacking this thread to talk about our lord and savior. :-D
The service author said this in regard to their bill:
> GCP seems to finally be budging with regard to the bill. They acknowledged the DDoS and are running it through the bureaucracy. I do have some confidence that they'll make this right, but I took destructive actions to stop the charges (deleting buckets). I did have a mostly complete backup of customer data on another cloud, but this has destroyed small business side hustle, where I built a community of over 100,000 users over seven years.
https://www.reddit.com/r/googlecloud/comments/1jzoi8v/ddos_a...
They left a message on their site: http://simmer.io/.
And just yesterday, they launched their new side hustle: https://stopuncappedbilling.com/.