When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony).
Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.
VectorLock · 22m ago
Amazon refunded you and you hate them for it?
I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
Vvector · 11m ago
If it is a "free tier", Amazon should halt the application when it exceeds quota. Moving the account to a paid tier and charging $100k is not the right thing to do.
cjbgkagh · 8m ago
Amazon is currently permissive which splits opposition, this won’t always be the case, they will tighten the screws eventually as they have done in the past in other areas. Amazon because it’s so broadly used undermines the utility of chargebacks, you can do it but it’ll be a real hassle to not be able to use Amazon for shopping. A lot of people will just eat the costs, is Amazon knows this they will force the situation more often because it’ll make them more money.
JJMcJ · 1h ago
This is an example of why cloud hosting is so scary.
Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money.
But I don't want to be forced into bankruptcy because my five visitor a week demo project suddenly becomes the target of a DDOS for no reason at all and the hosting company decides this isn't a "usually" so please send the wire transfer.
Bluecobra · 33m ago
When I am playing around in the cloud I am super paranoid about charges, so I end up locking the ACLs to only permit traffic to my home IP. It’s too bad that they don’t have a better built in way of making sandbox labs. When I was doing cloud training with A Cloud Guru, it would generate a whole global AWS instance that would only last for 30 minutes.
These aren’t limits though, they are just budget notifications.
What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500.
I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 90%. It’s great, but it’s not a limit. I can still get screwed if somehow, my projects become targets, and I don’t see the emails immediately or aren’t able to act on them immediately.
It blows my mind there’s no way I can just say, “there’s no conceivable outcome where I would want to spend more than $10 or more than $100 or whatever so please just cut me off as soon as I get anywhere close to that.”
The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects, and thus exposing yourself to ruinous liability.
sbarre · 51m ago
I feel that the likely answer here is that instrumenting real-time spending limit monitoring and cut-off at GCP/AWS scale is Complicated/Expensive to do, so they choose to not do it.
I suppose you could bake the limits into each service at deploy time, but that's still a lot of code to write to provide a good experience to a customer who is trying to not pay you money.
Not saying this is a good thing, but this feels about right to me.
Xelbair · 43m ago
I don't care if it is expensive for them.
I'm not running their business, I'm their customer - it is inconvenient for me.
And frankly any pay-as-you-go scheme should be regulated to have maximum spending limit setting.
Not only in IT.
VectorLock · 19m ago
Its not expensive for them, its expensive for their customers. If you went over your spending limit and they deleted all your shit, people would be absolutely apoplectic. Instead they make you file a relatively painless ticket and explain why you accidentally went over what you wanted to spend. This is an engineering trade-off they made to make things less painful for their customers.
appreciatorBus · 48m ago
I agree that that’s the likely explanation. It just feels infuriating that the services are sold as easy to get started and risk free with generous free tiers, inviting people and companies to try out small projects, yet each small experiment contains an element of unlimited risk with no mitigation tools.
bee_rider · 43m ago
I’ve accidentally hit myself with a bigger than expected AWS bill (just $500 but as a student I didn’t really want to spend that much). So I get being annoyed with the pricing model.
But, I don’t think the idea of just stopping charging works. For example, I had some of their machine image thingies (AMI) on my account. They charged me less than a dollar a month, totally reasonable. The only reasonable interpretation of “emergency stop on all charges completely” would be to delete those images (as well as shutting down my $500 nodes). This would have been really annoying, I mean putting the images together took a couple hours.
And that’s just for me. With accounts that have multiple users—do you really delete all the disk images on a business’s account, because one of their employees used compute to hit their spend limit? No, I think cloud billing is just inherently complicated.
falcor84 · 29m ago
> The only reasonable interpretation of “emergency stop on all charges completely” would be to delete those images
I disagree; a reasonable but customer-friendly interpretation would be to move these into a read-only "recycle bin" storage for e.g. a month, and only afterwards delete them if you don't provide additional budget.
appreciatorBus · 26m ago
Yeah I get it. It just irks that it's something I'd like to spend more time with and learn, but at every corner I feel like I'm exposing myself. For what I have done w/AWS & GCP so far with personal accounts, complete deletion of all resources & images would be annoying to be sure, but still preferable to unlimited liability. Ofc most companies using it won't be in that boat so IDK.
hamandcheese · 16m ago
> The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects
Yeah, I'm sure this is it. There is no way that feature is worth the investment when it only helps them sell to... broke individuals? (no offense. Most individuals are broke compared to AWS's target customer).
yowlingcat · 7m ago
[delayed]
agwa · 1h ago
Those are not in fact limits:
> There can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge. This is due to a delay between when an AWS resource is used and when that resource usage is billed. You might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you, and your actual costs or usage may continue to increase or decrease after you receive the notification.
ldoughty · 10m ago
I work in this space regularly. There can be a delay of 2-3 days from the event to charge. Seems some services report faster than others. But this means by the time you get a billing alert it has been ongoing for hours if not days.
JackSlateur · 25m ago
To all of those who say "this is not limit, only notifications": yes, notifications that can trigger whatever you want, including a shutdown of whatever you have
Is this a perfect solution: no
Is this still a solution: yes
master_crab · 55m ago
As others have said these are not limits, just notifications. You can’t actually create a limit unless you self create one using another AWS service (surprise) like lambda to read in the reports and shut things down.
And as others have also mentioned, the reports have a delay. In many cases it’s several hours. But worst case, your CURs (Cost usage reports) don’t really reflect reality for up to 24 hours after the fact.
Foobar8568 · 51m ago
As far as I know, neither Google, Amazon or Azure have a budget limit, only alerts.
This is a reason why I am not only clueless of anything related to cloud infrastructure unless it's stuff I am doing on the job, nor I am willing to build anything on these stacks.
And while I guess I have less than 10 products build with these techs, I am appeal by the overall reliability of the services.
Oh lastly, for Azure, in different European regions you can't instance resources, you need to go through your account representative who asks authorization from the US. So much for now having to deal with infrastructure pain.
It's just a joke.
Loudergood · 1h ago
"Limits" like this are how I woke up one Sunday morning in my college dorm with a $7k bill from dreamhost.
meepmorp · 53m ago
To paraphrase Rainer Wolfcastle - the budgets do nothing!
You get a warning. There's no service cutoffs or hard limits on spending.
ChrisMarshallNY · 42m ago
These "refund after overcharge" things are not without benefit to the corporations.
They get a nice tax write-off.
It's couch-cushion change for them, but it adds up. They have whole armies of beancounters, dreaming this stuff up.
It's also the game behind those "coupons" you get, for outrageously-priced meds that aren't covered by insurance.
If they charge $1,000 for the medication, but give you a "special discount" for $90%, they get to write off $900.
alberth · 29m ago
I’m fairly certain that’s incorrect.
Businesses are only taxed on actual revenue earned.
What you decide to charge—whether $100, $50, or even giving it away for free—is purely a business decision, not a tax one.
—
This is different from a nonprofit donation scenario though. For example, if your service normally costs $X but you choose to provide it for free (or at a discount) as a donation to a non-profit, you can typically write off the difference.
ChrisMarshallNY · 24m ago
You may be right (this is not my forte), but the invoice is real. So is the forgiveness. I don’t see how the IRS can legitimately deny a write-off.
I’ve heard stories like this, many times, from businesses people.
They certainly believe in the pattern.
franktankbank · 40m ago
Smells like fraud.
__loam · 16m ago
Yeah man the whole industry is like that. OpenAI gets to say they raised X billion dollars and update their valuation but they don't mention that it's all cloud compute credits from a gigantic Corp that owns a huge amount of the business. They claim to be a non-profit to do the research then when they've looted the commons, they switch to for profit to pay out the investors. There's shit like this throughout the industry.
jcims · 46m ago
I've got a $25k bill right now because I had enabled data-plane audit logging on an sqs queue that about a year ago I had wired to receive a real-time feed of audit events. So for every net-new audit event there would be an infinite loop of write events to follow. My average daily bill is about $2 on that account and has been for nearly ten years. It suddenly ballooned to $3k/day and zero warning or intervention from AWS.
cjbgkagh · 1h ago
I use AWS out of expedience but I hate the no-hard-cap experience and this is my primary reason for shifting (WIP) to self hosting. Plus self hosting is cheaper for me anyway. In general I would like a legally forced liability limit on unbounded subscription services, perhaps a list maintained at the credit card level. If the supplier doesn’t like the limit they can stop supplying. The surprise $100K liabilities are pure insanity.
hk1337 · 18m ago
It’s interesting because on the posted site there’s only 2 AWS posts on the main page and they’re rather mild compared to the other posts using google, vercel, cloudformation, etc.
wiether · 56m ago
> When I was learning to program through a bootcamp I spun up an elastic beanstalk instance
Didn't the bootcamp told you to, at least, setup a budget alert?
I'm not trying to reduce AWS' responsibility here, but if a teaching program tells you to use AWS but doesn't teach you how to use it correctly, you should question both AWS and the program's methods.
mikeocool · 42m ago
> I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion
They refunded you $100k with few questions asked, and you hate them for it?
I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them.
I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”
0cf8612b2e1e · 16m ago
Given how complicated configuring AWS is, surely there could be some middle ground between stop all running services and delete every byte of data. The former is surely what the typical low spend account would desire.
franktankbank · 41m ago
In what world is that not the preferable solution? Want to know if your shit is actually robust just set your cap and ddos yourself as the first test of you architecture.
VectorLock · 18m ago
Yes, a sign of resilient architecture is to shut down when it encounters some stress.
bratwurst3000 · 9m ago
allways set cost alarm and max spending. AWS has great tools to controll costs.
You could have blocked this with good config but I understand its confusing and not super apparent. IMHO there should be a pop up or sth asking " you want to stop the instance the moment it costs anything?"
its so easy to get billed a ridicules amount if money
motorest · 1h ago
> When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity.
Is it just me or is this just a cheap excuse to grab a payment method from unsuspecting free-tier users?
zdragnar · 57m ago
AWS services aren't designed for people just learning to program. Beanstalk and other services have billing limits you can set, but those aren't hard limits because they are measured async to keep performance up.
With that said, AWS is notoriously opaque in terms of "how much will I pay for this service" because they bill so many variable facets of things, and I've never really trusted the free tier myself unless I made sure it wasn't serving the public.
croes · 2h ago
That’s why I prefer prepaid cards or those I can easily freeze to prevent any booking.
kleinsch · 1h ago
Freezing a card doesn’t mean the debt is erased. They can still take you to collections.
1oooqooq · 1h ago
"your honor, they provided the credit card to prove their identity for the free plan and now we want to collect 100k"
motorest · 1h ago
And then they pull out the invoice where they prove without any doubt that you actually used pay-per-use services and ran up a 100k bill because you failed to do any sort of configuration.
bingabingabinga · 40m ago
I didn't use them, some bots did. Sort it out with them.
motorest · 3m ago
> I didn't use them, some bots did. Sort it out with them.
For you to put together this sort of argument with a straight face, you need to have little to no understanding of how the internet in general, services and web apps work. Good luck arguing your case with a judge.
cjbgkagh · 1h ago
I’ve not read the fine print but I’d be worried that there would be something in there that allows this.
Nextgrid · 1h ago
There are light-years between what a company thinks their ToS “allow” and what a court would actually determine is allowed. Plenty of ToS clauses in major contracts are unenforceable by law.
In this situation if it were to actually go to court I’d expect the actual bill to get significantly reduced at the very least. But because it’s a completely bullshit charge based on bandwidth usage (which costs them nothing) it won’t go anywhere near that and will just get written off anyway.
jsheard · 1h ago
If your card is declined and they don't feel like forgiving the bill, won't they just send debt collectors after you instead?
jauntywundrkind · 54m ago
> cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard just to confuse the customer into losing money.
I feel like this brand of sentiment is everywhere. Folks want things simple. We often figure out what we need to do to get by.
Over time we learn the reason for a handful of the options we initially defaulted through, find cause to use the options. Some intrepid explorers have enough broader context and interest to figure much more out but mostly we just set and forget, remembering only the sting of facing our own ignorance & begrudging the options.
This is why k8s and systemd are have such a loud anti-following.
xhkkffbf · 1h ago
Something similar happened to me, but not at the outrageous scale. I wanted to try some AI example on Bedrock. So the tutorial said I needed to set up some OpenSearch option. Voila. A few days later I had a bill for $120. The scale is not as horrible, but the principle is the same.
DanielHB · 1h ago
You are attributing to greed what can easily be explained by just not giving an f. They don't care that much about small customers.
hvb2 · 1h ago
> The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
By that logic, any technology that you can get certified in is too complicated?
Most systems are now distributed and presenting a holistic view of how it was designed to work can be useful to prevent simple mistakes.
Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
wongarsu · 1h ago
> By that logic, any technology that you can get certified in is too complicated?
That is a common view in UX, yes. It's a bit of an extreme view, but it's a useful gut reaction
> Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
In the US roads are designed so that you need as close to no knowledge as possible. You need to know some basic rules like the side of the road you drive on or that red means stop, but there is literal text on common road signs so people don't have to learn road signs. And the driving license is a bit of a joke, especially compared to other Western countries
There is something to be said about interfaces that are more useful for power users and achieve that by being less intuitive for the uninitiated. But especially in enterprise software the more prevalent effect is that spending less time and money on UX directly translates into generating more revenue from training, courses, paid support and certification programs
awestroke · 1h ago
> By that logic, any technology that you can get certified in is too complicated?
In IT, I am inclined to agree with that. In real engineering, it's sometimes necessary, especially dangerous technology and technology that people trust with their life
cuu508 · 37m ago
> dangerous technology and technology that people trust with their life
Software runs on so many things we depend on IMO it also in many cases falls in the "dangerous technology" category.
Non-hobby OSes, non-hobby web browsers, device drivers, software that runs critical infrastructure, software that runs on network equipment, software that handles personal data, --IMHO it would not be unreasonable to require formal qualifications for developers working on any of those.
cyanydeez · 1h ago
The history of making things complicated often involves "unintended" use by malicious actors.
But infact, it is intended side effects. Things like Jaywalking or "no spitting" laws let police officers harass more people _At their whim_. And they're fullying designed that way but left as "unintended" for the broader public scrutiny.
So, just like, learn that "logic" is not some magic thing you can sprinkle on everything and find some super moral or ethic reality. You have to actually integrate the impact through multiple levels of interaction to see the real problem with "it's just logic bro" response you got here.
anal_reactor · 52m ago
The problem with the AWS certificate is that the entity issuing the certificate and the entity honoring the certificate have opposing priorities. When a company wants to use AWS, preferably they'd want to avoid needlessly expensive solutions and vendor lock-in, while Amazon wants to teach people how to choose needlessly expensive solutions with vendor lock-in.
It is a fake degree.
tim333 · 1h ago
Not really. I think he's saying complicated for a cloud server. I don't think you can get degrees in digitalocean set up.
ctvo · 1h ago
I took a workshop class and was told to setup a track saw. The course didn't bother explaining how to utilize it properly or protect yourself. I ended up losing a finger. I truly hate Stanley Tools with a passion and if I ever need to use another track saw, I'll use someone else.
hamdingers · 1h ago
This analogy would make sense if the saw lacked a basic and obvious safety feature (billing limits) because Stanley profited immensely from cutting your finger off.
jsheard · 1h ago
Protect yourself how? Most cloud providers don't support any way to immediately abort spending if things get out of hand, and when running a public-facing service there are always variables you can't control.
Even if you rig up your own spending watchdog which polls the clouds billing APIs, you're still at the mercy of however long it takes for the cloud to reconcile your spending, which often takes hours or even days.
Galanwe · 1h ago
> I ended up losing a finger
You forgot to mention Stanley Tools paid for the hospital bill.
joshstrange · 3h ago
I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one:
I raised that exact same issue to AWS in ~2015 and even though we had an Enterprise support plan, AWS response was basically: well, you problem.
We then ended up deleting the S3 bucket entirely, as that appeared to be the only way to get rid of the charges, only for AWS to come back to use a few weeks later telling us there are charges for an S3 bucket we previously owned. After explaining to them (again) that this way our only option to get rid of the charges, we never heard back.
jsheard · 1h ago
You have to wonder how many people quietly got burned by that in the 18 years between S3 launching and that viral post finally prompting a response.
franktankbank · 2h ago
Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
pooper · 2h ago
> Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior".
> AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.
The stench of this bovine excrement is so strong that it transcends space time somehow.
franktankbank · 2h ago
Even pooper is upset about the stench. Tech is fuckin dumb in the corps, the only logical explanation to me is kickbacks to the CTO or similar.
pooper · 1h ago
Pooping at the job is one thing but pooping at the job and trying to sell it as a favor to the customer is a whole different game.
cyanydeez · 1h ago
you don't need kickbacks at this level. They're all judged by their 6-month outlook on revenue and their market shares.
This is just obfuscating grift justified by the "well, you own the severless functions!"
thousand_bats · 3h ago
> I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs.
Agreed about that. I was hired onto a team that inherited a large AWS Lambda backend and the opacity of the underlying platform (which is the value proposition of serverless!) has made it very painful when the going gets tough and you find bugs in your system down close to that layer (in our case, intermittent socket hangups trying to connect to the secrets extension). And since your local testing rig looks almost nothing like the deployed environment...
I have some toy stuff at home running on Google Cloud Functions and it works fine (and scale-to-zero is pretty handy for hiding in the free tier). But I struggle to imagine a scenario in a professional setting where I wouldn't prefer to just put an HTTP server/queue consumer in a container on ECS.
fishmicrowaver · 2h ago
I've had similar experiences with Azures services. Black boxes impossible to troubleshoot. Very unexpected behavior people aren't necessarily aware of when they initially spin these things up. For anything important I just accept the pain of deploying to kubernetes. Developers actually wind up preferring it in most cases with flux and devsoace.
Ekaros · 1h ago
I recently had customer who had smart idea to protect Container Registry with firewall... Breaking pretty much everything in process. Now it kinda works after days of punching enough holes in... But I still have no idea where does something like Container registry pull stuff from, or App Service...
And does some of their suggested solutions actually work or not...
trillic · 1h ago
Convince them to add IPv6 and you’ll be set for life
mikepurvis · 2h ago
Is that what people do is test/develop primarily with local mocks of the services? I assumed it was more like you deploy mini copies of the app to individual instances namespaced to developer or feature branch, so everyone is working on something that actually fairly closely approximates prod just without the loading characteristics and btw you have to be online so no working on an airplane.
icedchai · 2h ago
There are many paths. Worst case, I've witnessed developers editing Lambda code in the AWS console because they had no way to recreate the environment locally.
If you can't run locally, productivity drops like a rock. Each "cloud deploy" wastes tons of time.
tonkinai · 2h ago
Mocks usually don’t line up with how things run in prod. Most teams just make small branch or dev environments, or test in staging. Once you hit odd bugs, serverless stops feeling simple and just turns into a headache.
siva7 · 3h ago
How to destroy your competition. Love it. Also why i dislike AWS. Zero interest to protect their SMB customers from surprise bills. Azure isn't much better but at least they got a few more protections in place.
icedchai · 2h ago
Same, I was hoping for tales of woe and cloud lock-in, of being forced to use Lambda and Dynamo for something that could easily run on a $20/month VPS with sqlite.
kijin · 2h ago
The webflow one at the top has an interesting detail about them not allowing you to offload images to a cheaper service. Which you can probably work around by using a different domain.
ksynwa · 4m ago
I have only seen this services from afar but it is unbelievable to me that first-class cost overrun protection is not a given. I thought customers would not be using these services without such a safeguard. I have a pathological predisposition to use prepaid services exactly for avoiding such surprises.
dakiol · 3h ago
> I had cloudflare in front of my stuff. Hacker found an uncached object and hit it 100M+ times. I stopped that and then they found my origin bucket and hit that directly.
Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?
solatic · 3h ago
No. Your buckets should be private, with a security rule that they can only be accessed by your CDN provider, precisely to force the CDN to be used.
rwmj · 2h ago
Why isn't that the default?
I'm glad I use a Hetzner VPS. I pay about EUR 5 monthly, and never have to worry about unexpected bills.
kdps · 2h ago
Don't they charge for every TB exceeding the included limit? (website says "For each additional TB, we charge € 1.19 in the EU and US, and € 8.81 in Singapore.")
jsheard · 1h ago
They do, but the risk of having to pay $1.44/TB after the first 20TB is easier to swallow than say, CloudFront's ~$100/TB after 1TB.
Cpoll · 32m ago
> CloudFront's ~$100/TB after 1TB.
I had to double-check because that sounds hilariously wrong. I can't find it anywhere in the pricing. It's at most 0.08/TB.
Am I missing something?
jsheard · 29m ago
You're missing the unit, it's $0.085 per GB, not TB, and that's only for NA/EU traffic. I rounded up a bit because other regions cost more, plus you get billed per request as well.
They do offer progressively cheaper rates as you use more bandwidth each month, but that doesn't have much impact until you're spending comical amounts already.
wiether · 41m ago
Buckets are private by default.
And it's getting harder and harder to make them public because of people misconfiguring them and then going public against AWS when they discover the bill.
hvb2 · 1h ago
Because just using a cdn without proper caching headers is just another service you're paying for without any savings.
The real question is if they considered caching and thus configured it appropriately. If you don't, you're telling everyone you want every request to go to origin
graemep · 2h ago
Because not all uses for buckets fit that.
Buckets are used for backups, user uploads, and lots of things other than distributing files publicly.
graemep · 1h ago
I would say its probably not a good idea to make a bucket directly publicly accessible, but people do not do that.
A lot of the point of serverless is convenience and less admin and things like adding a layer in front of the bucket that could authenticate, rate limit etc. is not convenient and requires more admin.
gdbsjjdn · 3h ago
This story is giving "I leave OWASP top 10 vulns in my code because hacker mindset".
It's not that hard to configure access controls, they're probably cutting corners on other areas as well. I wouldn't trust anything this person is responsible for.
charcircuit · 2h ago
It's about rate limiting, not access controls. Without implementing limits your spend can go above what your budget is. Without cloud you hit natural rate limits of the hardware you are using to host.
philwelch · 1h ago
That might be the more general solution but in this context it is absolutely also an access control issue.
mschuster91 · 2h ago
with "classic" hosting, your server goes down from being overloaded to the hoster shutting it off.
with AWS, you wake up to a 6 figures bill.
gonzo41 · 3h ago
No, s3 objects should always be private and then have a cloudfront proxy in front of them at the least. You should always have people hitting a cache for things like images.
caboteria · 1h ago
The real serverless horror isn't the occasional mistake that leads to a single huge bill, it's the monthly creep. It's so easy to spin up a resource and leave it running. It's just a few bucks, right?
I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.
hvb2 · 1h ago
You don't think this happens on prem? Servers running an application that is no longer used?
Sure they're probably VMs but their cost isn't 0 either
sgarland · 46m ago
With that model, your cost doesn't change, though. When/if you find you need more resources, you can (if you haven't been doing so) audit existing applications to clear out cruft before you purchase more hardware.
the__alchemist · 3h ago
"Serverless" is a an Orwellian name for a server-based system!
Biganon · 2h ago
"There's no cloud; it's just someone else's computer"
Spivak · 28m ago
But your so called "no-code" system runs on code. Checkmate atheists.
There becomes a point where being mad that the specific flavor of PaaS termed serverless achtually has severs is just finding a thing to be mad at.
I used 1TB of traffic on a micro instance and it cost me $150 (iirc). Doesn't have to be this way.
Havoc · 2h ago
Putting any sort of pay per use product onto the open internet has always struck me as insane. Especially with scaling enabled.
At least stick a rate limited product in front of it to control the bleed. (And check whether the rate limit product is in itself pay per use...GCP looking at you)
omnicognate · 2h ago
It would help to round to the cent. With 3 digits to the right of the dot it's ambiguous whether it's a decimal point or a thousands separator, and the font and underline makes the comma vs dot distinction a bit unclear.
thedanbob · 1h ago
A number of the titles appear to have 69 or 420 cents added to the amount that appears in the story.
trcf22 · 3h ago
After a quick check on Vercel stories, it seems all payments were discarded or mistakes in the first place.
Does it really happen to really have to pay such a bill? Do you need to tweet about it to be reimbursed?
interloxia · 1h ago
Someone at a community group I'm in messed up playing with Azure through their free for non-profits offering^. We were out about 1.2k€. Not huge but huge for us.
Encouraged by comments on HN over the years I had them ask support to kindly to wave it. After repeating the request a few times they eventually reduced their bill to <100€ but refused to wave it entirely.
So even without shaming on social media, But it probably depends. It's worth at least asking.
^The deal changed about six months ago.
cuu508 · 2m ago
It's waive, not wave
Alifatisk · 3h ago
> Do you need to tweet about it to be reimbursed?
This is what scares me, is social media the only way to get things sorted out nowadays? What if I don't have a large following nor an account in the first place, do I have to stomach the bill?
pelagicAustral · 2h ago
This is exactly what happened to me during Covid... I had a flight that got cancelled at the beginning of the pandemic since the country closed the orders (essentially). A year after, still on lock downs and et al, I wanted to enquire about a refund, for months I got not answer, until I caught wind that people using Twitter were actually getting results. Now, I don’t use social media at all, so I had to create a Twitter account, twit about my case et voila! 30 mins after I got a response and they send me a PM with a case number... Not even going to mention the airline, but it is infuriating...
wg002 · 3h ago
I can't imagine them sending it to collections. What kind of recourse would a company like Vercel have if you don't pay it?
pjmlp · 3h ago
No, at least in enterprise consulting for these kind of hosting, usually there is a contact person on the support team that one can reach directly.
However these projects are measured in ways that make Oracle licenses rounding errors.
Which naturally creates market segmentation on who gets tier 1 treatment and everyone else.
viraptor · 3h ago
Once you're in a contract + TAM territory, pricing works very differently. Also, temporary experiments and usage overruns become an interesting experience where the company may just forget to bill you a few thousands $ just because nobody looked at the setup recently. Very different situation to a retail user getting unexpected extra usage.
Havoc · 3h ago
Relying on the mercy of a support agent that may be having a bad day is a poor strategy
tonyhart7 · 2h ago
I mean if developer got charged with 100k, more often than not the bank would decline that first maybe if you didn't have that high credit limit
but what happen if this happen to corporate account and somewhere resource get leaked???
multi billions dollar company probably just shrug it off as opex and call it a day
prisenco · 1h ago
This is why when I contract for an early stage startup, I pose the question:
"What if your app went viral and you woke to a $20k cloud bill? $50k? $80k?"
If the answer is anything less than "Hell yeah, we'll throw it on a credit card and hit up investors with a growth chart" then I suggest a basic vps setup with a fixed cost that simply stops responding instead.
There is such a thing as getting killed by success and while it's possible to negotiate with AWS or Google to reduce a surprise bill, there's no guarantee and it's a lot to throw on a startup's already overwhelming plate.
The cloud made scaling easier in ways, but a simple vps is so wildly overpowered compared to 15 years ago, a lot of startups can go far with a handful of digitalocean droplets.
delduca · 1h ago
Couple years ago I was charged in USD 4K on Google Cloud after trying recursive cloud functions.
I told them that was a mistake and they forgot the debit, they just asked to no do again.
shayway · 48m ago
I guess I'm missing something, why is this 'serverless' horrors? If anything it seems to specifically be serverful horrors.
jamil7 · 2h ago
Don’t most of these services have config options to protect against doing this? I haven’t used most of these services but it running up a bill during traffic spikes but not going down seems like it’s working as intended?
swiftcoder · 2h ago
Nope, basically none of these services have a way to set a hard budget. They let you configure budget warnings, but it’s generally up to you to login and actually shut down everything to prevent from being billed for overages (or you have to build your own automation - but the billing alerts may not be reliable)
paseante · 2h ago
Yeah I also left my website hosted on Google Cloud because costs popped from everywhere, and there is basically no built-in functionality to limit them. So I didn't really slept relaxed (I actually slept great, but I hope you get the point) knowing that a bug could cost me... who knows how much.
Actually, as the website of OP says, for spending control you have budget notifications and with that you can disable the billing for all the project altogether through some API call or something, I don't remember exactly, that is all there is. But still it looks like this functionality is just not there.
mortsnort · 36m ago
You can write Google cloud functions to disable your credit card when certain thresholds are met pretty easily, but it's unethical that this isn't just a toggle somewhere in settings.
Cpoll · 19m ago
Does that actually stop the spend immediately? If not, you're still on the hook for the bill. I suppose you can walk away and let them try to come after you, but that wouldn't work for a company.
johnebgd · 1h ago
We are building bare metal for our workloads… I don’t care if cloud is supposed to be cheaper because it never is. You can get a decent small business firewall to handle 10gbit fiber for $600 from unifi these days. Just another reason I’m glad I moved out of the Bay Area and nyc to a midwestern town for my company. I have a basement and can do rad things in my house to grow my business.
nchmy · 52m ago
bUt wuT aBowT deV OpS?!
stressback · 45m ago
I read a lot of the posts at the little blog here and, uh, every single one sounds like a complete amateur making a cloud configuration mistake. I haven't found one that is the provider's fault or the fault of "serverless"
I would be embarrassed to put my name on these posts admitting I can't handle my configs while blaming everyone but myself.
Serverless isn't a horror, serverlesshorrors poster. You are the horror. You suck at architecting efficient & secure systems using this technology, you suck at handling cloud spend, and you suck at taking responsibility when your "bug" causes a 10,000x discrepancy between your expected cost and your actual bill.
Just because you don't understand it doesn't mean it sucks
Bluecobra · 1m ago
That being said, the cloud providers could do a better job explaining to new/naive users that great power comes with great responsibility and there is no hand holding. Someone might be more hesitant to willy nilly spin up something if a wizard estimates that the maximum cost could be $X per month.
Cpoll · 27m ago
You're not wrong about cloud configuration mistakes, but a tool that lets you increase costs 10000x (without even letting you set a safety) is a hell of a chainsaw.
I'm more worried about the overconfident SRE that doesn't stay up at night worrying about these.
ChrisMarshallNY · 2h ago
At one time, I considered using Firebase as a backend, but then, I kept reading stories like these, and decided to roll my own. I'm fortunate to be able to do that.
It's kind of amazing, though. I keep getting pressure from the non-techs in my organization to "Migrate to the Cloud." When I ask "Why?" -crickets.
Industry jargon has a lot of power. Seems to suck the juice right out of people's brains (and the money right out of their wallets).
zkmon · 3h ago
Maintaining your own containers or VMs is hard considering how much risk appetite you have for the issues at infra level. So, yeah, when you complain about the costs of serverless, you are just paying for your low risk appetite low cost of your IT management.
wg002 · 3h ago
This site is a bit dated. I remember in response to this Vercel added a way to pause your projects when hitting a spend limit. I enabled it for my account.
Still, it made me question why I'm not using a VPS.
aurareturn · 2h ago
Vercel used to be called Zeit. They had a server product called Now that gave you 10 1CPU/1GPU instances for $10/month (or $20 I forgot). It was the best deal.
When Vercel switched everything to serverless, it all became pretty terrible. You need 3rd party services for simple things like DB connection pooling, websockets, cron jobs, simple queue, etc because those things aren’t compatible with serverless. Not to mention cold starts. Just a few weeks ago, I tried to build an API on Next.js+Vercel and get random timeouts due to cold start issues.
Vercel made it easier to build and deploy static websites. But really, why are you using Next.js for static websites? Wordpress works fine. Anything works fine. Serverless makes it drastically harder to build a full app with a back end.
pjmlp · 2h ago
Serverless is the most common deployment on MACH projects.
Because when everything is a bunch of SaaS Lego bricks, serverless is all one needs for integration logic, and some backend like logic.
Add to it that many SaaS vendors in CMS and ecommerce space have special partner deals with Vercel and Nelify.
there should be some kind of insurance for bugs that introduce unusually expensive usage
mdaniel · 3m ago
I believe any such policy would need its premiums based on the services used (and likely the qualifications of the staff) since, unlike rebuilding a house, the financial risk is almost unlimited with out of control cloud spend
It reminds me of the Citi(?) employee who typed the wrong decimal place in a trade: computers make everything so easy!
1oooqooq · 1h ago
last employer asked for an estimate to migrate to cloud.
it would be 2x more expensive and halve developer speed. also we would lose some internal metric systems honed over 20yr.
ceo told to go ahead anyway (turn out company was being sold to Apollo)
first thing we did was a way to bootstrap accounts into aws so we could have spend limits from day one.
can't imagine how companies miss that step.
EGreg · 2h ago
Are there any protections these days at the cloud provider level?
Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?
JJMcJ · 1h ago
That's why I like VPS setups. You hit the monthly maximum, and it just stops working.
I host demos, not running a business, so it's less of an issue to get interrupted. Better an interruption than a $50,000 bill for forgetting to shut off a test database from last Wednesday.
prisenco · 1h ago
Unless a startup has five+ nines service contracts with their customers already, a little bit of downtime once in a while is not the end of the world the cloud services want us to believe.
qcnguy · 1h ago
That's not comparable. With a VPS there is no monthly maximum, just a max load on a second by second basis. You can be hit with traffic of which 90% bounces because your server is down, get nowhere near your intended monthly maximum, and then the rest of the month is quiet.
bc569a80a344f9c · 1h ago
Not _really_. AWS has a budget tool, but it doesn’t natively support shutting down services. Of course, you can ingest the alerts it sends any way you want, including feeding them into pipelines that disable services. There’s plenty of blueprints you can copy for this. More seriously - and this is a legitimate technical limitation - of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes. That means there’s some lag, and you are responsible for any costs incurred that go over budget between such reporting runs.
franktankbank · 1h ago
Just set alerts that are not really timely and homeroll your own kill scripts its easy. It doesn't really work but its not really any harder than just fucking self hosting.
fnord77 · 1h ago
looking forward to the "LLM token horrors" version
nurettin · 1h ago
I've had this twice. Once with oracle, once with azure. They both charged me $2000-$5000 for simply opening and closing a database instance (used only for a single day to test a friend's open source project)
To be fair, support was excellent both times and they waived the bills after I explained the situation.
qcnguy · 1h ago
How did you run up a $5000 bill for just testing a project? What kind of project was it that could put so much load on the DB?
api · 1h ago
There should also be a general category for "cloud horrors" for things that cost $50k/month to host that would be $1500/month on a bare metal provider like Datapacket or Hetzner.
I'm old enough to remember when cloud was pitched as a big cost saving move. I knew it was bullshit then. Told you so.
nchmy · 49m ago
even $1500/mo on hetzner is a seriously large app. You could get 300 cpus and 1.5TB of RAM for that price.
game_the0ry · 1h ago
I have a feeling I will be downvoted for this, but...
Have the people posting these horror stories never heard of billing alerts?
skippyboxedhero · 1h ago
Many of the stories on the site are from people who have billing alerts.
If you have bot spam, how do you actually think their billing alerts work? The alert is updated every 100ms and shuts off your server immediately? That isn't how billing alerts can or should work.
Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.
I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money.
But I don't want to be forced into bankruptcy because my five visitor a week demo project suddenly becomes the target of a DDOS for no reason at all and the hosting company decides this isn't a "usually" so please send the wire transfer.
https://docs.aws.amazon.com/cost-management/latest/userguide...
What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500.
I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 90%. It’s great, but it’s not a limit. I can still get screwed if somehow, my projects become targets, and I don’t see the emails immediately or aren’t able to act on them immediately.
It blows my mind there’s no way I can just say, “there’s no conceivable outcome where I would want to spend more than $10 or more than $100 or whatever so please just cut me off as soon as I get anywhere close to that.”
The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects, and thus exposing yourself to ruinous liability.
I suppose you could bake the limits into each service at deploy time, but that's still a lot of code to write to provide a good experience to a customer who is trying to not pay you money.
Not saying this is a good thing, but this feels about right to me.
And frankly any pay-as-you-go scheme should be regulated to have maximum spending limit setting. Not only in IT.
But, I don’t think the idea of just stopping charging works. For example, I had some of their machine image thingies (AMI) on my account. They charged me less than a dollar a month, totally reasonable. The only reasonable interpretation of “emergency stop on all charges completely” would be to delete those images (as well as shutting down my $500 nodes). This would have been really annoying, I mean putting the images together took a couple hours.
And that’s just for me. With accounts that have multiple users—do you really delete all the disk images on a business’s account, because one of their employees used compute to hit their spend limit? No, I think cloud billing is just inherently complicated.
I disagree; a reasonable but customer-friendly interpretation would be to move these into a read-only "recycle bin" storage for e.g. a month, and only afterwards delete them if you don't provide additional budget.
Yeah, I'm sure this is it. There is no way that feature is worth the investment when it only helps them sell to... broke individuals? (no offense. Most individuals are broke compared to AWS's target customer).
> There can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge. This is due to a delay between when an AWS resource is used and when that resource usage is billed. You might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you, and your actual costs or usage may continue to increase or decrease after you receive the notification.
Is this a perfect solution: no Is this still a solution: yes
And as others have also mentioned, the reports have a delay. In many cases it’s several hours. But worst case, your CURs (Cost usage reports) don’t really reflect reality for up to 24 hours after the fact.
This is a reason why I am not only clueless of anything related to cloud infrastructure unless it's stuff I am doing on the job, nor I am willing to build anything on these stacks.
And while I guess I have less than 10 products build with these techs, I am appeal by the overall reliability of the services.
Oh lastly, for Azure, in different European regions you can't instance resources, you need to go through your account representative who asks authorization from the US. So much for now having to deal with infrastructure pain. It's just a joke.
You get a warning. There's no service cutoffs or hard limits on spending.
They get a nice tax write-off.
It's couch-cushion change for them, but it adds up. They have whole armies of beancounters, dreaming this stuff up.
It's also the game behind those "coupons" you get, for outrageously-priced meds that aren't covered by insurance.
If they charge $1,000 for the medication, but give you a "special discount" for $90%, they get to write off $900.
Businesses are only taxed on actual revenue earned.
What you decide to charge—whether $100, $50, or even giving it away for free—is purely a business decision, not a tax one.
—
This is different from a nonprofit donation scenario though. For example, if your service normally costs $X but you choose to provide it for free (or at a discount) as a donation to a non-profit, you can typically write off the difference.
I’ve heard stories like this, many times, from businesses people.
They certainly believe in the pattern.
Didn't the bootcamp told you to, at least, setup a budget alert?
I'm not trying to reduce AWS' responsibility here, but if a teaching program tells you to use AWS but doesn't teach you how to use it correctly, you should question both AWS and the program's methods.
They refunded you $100k with few questions asked, and you hate them for it?
I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them.
I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”
its so easy to get billed a ridicules amount if money
Is it just me or is this just a cheap excuse to grab a payment method from unsuspecting free-tier users?
With that said, AWS is notoriously opaque in terms of "how much will I pay for this service" because they bill so many variable facets of things, and I've never really trusted the free tier myself unless I made sure it wasn't serving the public.
For you to put together this sort of argument with a straight face, you need to have little to no understanding of how the internet in general, services and web apps work. Good luck arguing your case with a judge.
In this situation if it were to actually go to court I’d expect the actual bill to get significantly reduced at the very least. But because it’s a completely bullshit charge based on bandwidth usage (which costs them nothing) it won’t go anywhere near that and will just get written off anyway.
I feel like this brand of sentiment is everywhere. Folks want things simple. We often figure out what we need to do to get by.
Over time we learn the reason for a handful of the options we initially defaulted through, find cause to use the options. Some intrepid explorers have enough broader context and interest to figure much more out but mostly we just set and forget, remembering only the sting of facing our own ignorance & begrudging the options.
This is why k8s and systemd are have such a loud anti-following.
By that logic, any technology that you can get certified in is too complicated?
Most systems are now distributed and presenting a holistic view of how it was designed to work can be useful to prevent simple mistakes.
Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
That is a common view in UX, yes. It's a bit of an extreme view, but it's a useful gut reaction
> Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
In the US roads are designed so that you need as close to no knowledge as possible. You need to know some basic rules like the side of the road you drive on or that red means stop, but there is literal text on common road signs so people don't have to learn road signs. And the driving license is a bit of a joke, especially compared to other Western countries
There is something to be said about interfaces that are more useful for power users and achieve that by being less intuitive for the uninitiated. But especially in enterprise software the more prevalent effect is that spending less time and money on UX directly translates into generating more revenue from training, courses, paid support and certification programs
In IT, I am inclined to agree with that. In real engineering, it's sometimes necessary, especially dangerous technology and technology that people trust with their life
Software runs on so many things we depend on IMO it also in many cases falls in the "dangerous technology" category.
Non-hobby OSes, non-hobby web browsers, device drivers, software that runs critical infrastructure, software that runs on network equipment, software that handles personal data, --IMHO it would not be unreasonable to require formal qualifications for developers working on any of those.
But infact, it is intended side effects. Things like Jaywalking or "no spitting" laws let police officers harass more people _At their whim_. And they're fullying designed that way but left as "unintended" for the broader public scrutiny.
So, just like, learn that "logic" is not some magic thing you can sprinkle on everything and find some super moral or ethic reality. You have to actually integrate the impact through multiple levels of interaction to see the real problem with "it's just logic bro" response you got here.
It is a fake degree.
Even if you rig up your own spending watchdog which polls the clouds billing APIs, you're still at the mercy of however long it takes for the cloud to reconcile your spending, which often takes hours or even days.
You forgot to mention Stanley Tools paid for the hospital bill.
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...
About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.
We then ended up deleting the S3 bucket entirely, as that appeared to be the only way to get rid of the charges, only for AWS to come back to use a few weeks later telling us there are charges for an S3 bucket we previously owned. After explaining to them (again) that this way our only option to get rid of the charges, we never heard back.
It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior".
> AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.
The stench of this bovine excrement is so strong that it transcends space time somehow.
This is just obfuscating grift justified by the "well, you own the severless functions!"
Agreed about that. I was hired onto a team that inherited a large AWS Lambda backend and the opacity of the underlying platform (which is the value proposition of serverless!) has made it very painful when the going gets tough and you find bugs in your system down close to that layer (in our case, intermittent socket hangups trying to connect to the secrets extension). And since your local testing rig looks almost nothing like the deployed environment...
I have some toy stuff at home running on Google Cloud Functions and it works fine (and scale-to-zero is pretty handy for hiding in the free tier). But I struggle to imagine a scenario in a professional setting where I wouldn't prefer to just put an HTTP server/queue consumer in a container on ECS.
And does some of their suggested solutions actually work or not...
If you can't run locally, productivity drops like a rock. Each "cloud deploy" wastes tons of time.
Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?
I'm glad I use a Hetzner VPS. I pay about EUR 5 monthly, and never have to worry about unexpected bills.
I had to double-check because that sounds hilariously wrong. I can't find it anywhere in the pricing. It's at most 0.08/TB.
Am I missing something?
They do offer progressively cheaper rates as you use more bandwidth each month, but that doesn't have much impact until you're spending comical amounts already.
And it's getting harder and harder to make them public because of people misconfiguring them and then going public against AWS when they discover the bill.
The real question is if they considered caching and thus configured it appropriately. If you don't, you're telling everyone you want every request to go to origin
Buckets are used for backups, user uploads, and lots of things other than distributing files publicly.
A lot of the point of serverless is convenience and less admin and things like adding a layer in front of the bucket that could authenticate, rate limit etc. is not convenient and requires more admin.
It's not that hard to configure access controls, they're probably cutting corners on other areas as well. I wouldn't trust anything this person is responsible for.
with AWS, you wake up to a 6 figures bill.
I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.
Sure they're probably VMs but their cost isn't 0 either
There becomes a point where being mad that the specific flavor of PaaS termed serverless achtually has severs is just finding a thing to be mad at.
I used 1TB of traffic on a micro instance and it cost me $150 (iirc). Doesn't have to be this way.
At least stick a rate limited product in front of it to control the bleed. (And check whether the rate limit product is in itself pay per use...GCP looking at you)
Does it really happen to really have to pay such a bill? Do you need to tweet about it to be reimbursed?
Encouraged by comments on HN over the years I had them ask support to kindly to wave it. After repeating the request a few times they eventually reduced their bill to <100€ but refused to wave it entirely.
So even without shaming on social media, But it probably depends. It's worth at least asking.
^The deal changed about six months ago.
This is what scares me, is social media the only way to get things sorted out nowadays? What if I don't have a large following nor an account in the first place, do I have to stomach the bill?
However these projects are measured in ways that make Oracle licenses rounding errors.
Which naturally creates market segmentation on who gets tier 1 treatment and everyone else.
but what happen if this happen to corporate account and somewhere resource get leaked???
multi billions dollar company probably just shrug it off as opex and call it a day
"What if your app went viral and you woke to a $20k cloud bill? $50k? $80k?"
If the answer is anything less than "Hell yeah, we'll throw it on a credit card and hit up investors with a growth chart" then I suggest a basic vps setup with a fixed cost that simply stops responding instead.
There is such a thing as getting killed by success and while it's possible to negotiate with AWS or Google to reduce a surprise bill, there's no guarantee and it's a lot to throw on a startup's already overwhelming plate.
The cloud made scaling easier in ways, but a simple vps is so wildly overpowered compared to 15 years ago, a lot of startups can go far with a handful of digitalocean droplets.
I told them that was a mistake and they forgot the debit, they just asked to no do again.
I would be embarrassed to put my name on these posts admitting I can't handle my configs while blaming everyone but myself.
Serverless isn't a horror, serverlesshorrors poster. You are the horror. You suck at architecting efficient & secure systems using this technology, you suck at handling cloud spend, and you suck at taking responsibility when your "bug" causes a 10,000x discrepancy between your expected cost and your actual bill.
Just because you don't understand it doesn't mean it sucks
I'm more worried about the overconfident SRE that doesn't stay up at night worrying about these.
It's kind of amazing, though. I keep getting pressure from the non-techs in my organization to "Migrate to the Cloud." When I ask "Why?" -crickets.
Industry jargon has a lot of power. Seems to suck the juice right out of people's brains (and the money right out of their wallets).
Still, it made me question why I'm not using a VPS.
When Vercel switched everything to serverless, it all became pretty terrible. You need 3rd party services for simple things like DB connection pooling, websockets, cron jobs, simple queue, etc because those things aren’t compatible with serverless. Not to mention cold starts. Just a few weeks ago, I tried to build an API on Next.js+Vercel and get random timeouts due to cold start issues.
Vercel made it easier to build and deploy static websites. But really, why are you using Next.js for static websites? Wordpress works fine. Anything works fine. Serverless makes it drastically harder to build a full app with a back end.
Because when everything is a bunch of SaaS Lego bricks, serverless is all one needs for integration logic, and some backend like logic.
Add to it that many SaaS vendors in CMS and ecommerce space have special partner deals with Vercel and Nelify.
https://macharchitecture.com/
It reminds me of the Citi(?) employee who typed the wrong decimal place in a trade: computers make everything so easy!
it would be 2x more expensive and halve developer speed. also we would lose some internal metric systems honed over 20yr.
ceo told to go ahead anyway (turn out company was being sold to Apollo)
first thing we did was a way to bootstrap accounts into aws so we could have spend limits from day one.
can't imagine how companies miss that step.
Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?
I host demos, not running a business, so it's less of an issue to get interrupted. Better an interruption than a $50,000 bill for forgetting to shut off a test database from last Wednesday.
To be fair, support was excellent both times and they waived the bills after I explained the situation.
I'm old enough to remember when cloud was pitched as a big cost saving move. I knew it was bullshit then. Told you so.
Have the people posting these horror stories never heard of billing alerts?
If you have bot spam, how do you actually think their billing alerts work? The alert is updated every 100ms and shuts off your server immediately? That isn't how billing alerts can or should work.