Serverless Horrors

You can ring up tens of thousands+ overnight with AWS. The scale of potential damages is nowhere even close.

For the cloud, I have the good will of the cloud provider and appealing to social media. Not the same thing.

The broken water pipe should be covered by buildings insurance, but I can imagine it not being covered by some policies. Luckily a broken water pipe is likely not as expensive as not having e.g. third party liability protection if part of your roof falls off and hits someone.

What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500.

I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 90%. It’s great, but it’s not a limit. I can still get screwed if somehow, my projects become targets, and I don’t see the emails immediately or aren’t able to act on them immediately.

It blows my mind there’s no way I can just say, “there’s no conceivable outcome where I would want to spend more than $10 or more than $100 or whatever so please just cut me off as soon as I get anywhere close to that.”

The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects, and thus exposing yourself to ruinous liability.

> There can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge. This is due to a delay between when an AWS resource is used and when that resource usage is billed. You might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you, and your actual costs or usage may continue to increase or decrease after you receive the notification.

This is a reason why I am not only clueless of anything related to cloud infrastructure unless it's stuff I am doing on the job, nor I am willing to build anything on these stacks.

And while I guess I have less than 10 products build with these techs, I am appeal by the overall reliability of the services.

Oh lastly, for Azure, in different European regions you can't instance resources, you need to go through your account representative who asks authorization from the US. So much for now having to deal with infrastructure pain. It's just a joke.

And as others have also mentioned, the reports have a delay. In many cases it’s several hours. But worst case, your CURs (Cost usage reports) don’t really reflect reality for up to 24 hours after the fact.

Is this a perfect solution: no Is this still a solution: yes

You get a warning. There's no service cutoffs or hard limits on spending.

That’s an insane amount of both money and stress. You’re at Amazon’s mercy if they will or will not refund it. And while this is in process you’re wondering if your entire financial future is ruined.

For a postpaid service with usage-based billing, there are no separate "free" and "paid" plans (= what you're clearly thinking of when you're saying "tiers" here.)

The "free tier" of these services, is a set of per-usage-SKU monthly usage credit bonuses, that are set up in such a way that if you are using reasonable "just testing" amounts of resources, your bill for the month will be credited down to $0.

And yes, this does mean that even when you're paying for some AWS services, you're still benefitting from the "free tier" for any service whose usage isn't exceeding those free-tier limits. That's why it's a [per-SKU usage] tier, rather than a "plan."

If you're familiar with electricity providers telling you that you're about to hit a "step-up rate" for your electricity usage for the month — that's exactly the same type of usage tier system. Except theirs goes [cheap usage] -> [expensive usage], whereas IaaS providers' tiers go [free usage] -> [costed usage].

> Amazon should halt the application when it exceeds quota.

There is no easy way to do this in a distributed system (which is why IaaS services don't even try; and why their billing dashboards are always these weird detached things that surface billing only in monthly statements and coarse-grained charts, with no visibility into the raw usage numbers.)

There's a lot of inherent complexity of converting "usage" into "billable usage." It involves not just muxing usage credit-spend together, but also classifying spend from each system into a SKU [where the appropriate bucket for the same usage can change over time]; and then a lot of lookups into various control-plane systems to figure out whether any bounded or continuous discounts and credits should be applied to each SKU.

And that means that this conversion process can't happen in the services themselves. It needs to be a separate process pushed out to some specific billing system.

Usually, this means that the services that generate billable usage are just asynchronously pushing out "usage-credit spend events" into something like a log or message queue; and then a billing system is, asynchronously, sucking these up and crunching through them to emit/checkpoint "SKU billing events" against an invoice object tied to a billing account.

Due to all of the extra steps involved in this pipeline, the cumulative usage that an IaaS knows about for a given billing account (i.e. can fire a webhook when one of those billing events hits an MQ topic) might be something like 5 minutes out-of-date of the actual incoming usage-credit-spend.

Which means that, by the time any "trigger" to shut down your application because it exceeded a "quota" went through, your application would have already spent 5 minutes more of credits.

And again, for a large, heavily-loaded application — the kind these services are designed around — that extra five minutes of usage could correspond to millions of dollars of extra spend.

Which is, obviously, unacceptable from a customer perspective. No customer would accept a "quota system" that says you're in a free plan, yet charges you, because you accrued an extra 5 minutes of usage beyond the free plan's limits before the quota could "kick in."

But nor would the IaaS itself just be willing to eat that bill for the actual underlying costs of serving that extra 5 minutes of traffic, because that traffic could very well have an underlying cost of "millions of dollars."

So instead they just say "no, we won't implement a data-plane billable-usage-quota feature; if you want it, you can either implement it yourself [since your L7 app can observe its usage 'live' much better than our infra can] or, more idiomatically to our infra, you can ensure that any development project is configured with appropriate sandboxing + other protections to never get into a situation where any resource could exceed its the free-tier-credited usage in the first place."

As I asked before, if what is causing overages is not web requests but storage should they just delete everything?

Software runs on so many things we depend on IMO it also in many cases falls in the "dangerous technology" category.

Non-hobby OSes, non-hobby web browsers, device drivers, software that runs critical infrastructure, software that runs on network equipment, software that handles personal data, --IMHO it would not be unreasonable to require formal qualifications for developers working on any of those.

I’ve heard stories like this, many times, from businesses people.

They certainly believe in the pattern.

I don't want to go too far down the rabbit hole of hn speculation, but if another entity owes you 100k, and they go bankrupt, there absolutely are tax implications.

It’s basic stuff.

One of those things is more important to different types of business. In some situations, any downtime at all is worth thousands per hour. In others, the service staying online is only worth hundreds of dollars a week.

So yes, the solution is as simple as giving the user hard spend caps that they can configure. I'd also set the default limits low for new accounts with a giant, obnoxious, flashing red popover that you cannot dismiss until you configure your limits.

However, this would generate less profit for Amazon et al. They have certainly run this calculation and decided they'd earn more money from careless businesses than they'd gain in goodwill. And we all know that goodwill has zero value to companies at FAANG scale. There's absolutely no chance that they haven't considered this. It's partially implemented and an incredibly obvious solution that everyone has been begging for since cloud computing became a thing. The only reason they haven't implemented this is purely greed and malice.

Maybe rather than completely stopping the service, it'd be better to rate limit the service when approaching/reaching the cap.

If your business suddenly starts generating Tbs of traffic (that is not a ddos), you'd be thrilled to pay overage fees because your business just took off.

You don't usually get $10k bandwidth fees because your misconfigured service consumes too much CPU.

And besides that, for most of these cases, a small business can host on-prem with zero bandwidth fees of any type, ever. If you can get by with a gigabit uplink, you have nothing to worry about. And if you're at the scale where AWS overages are a real problem, you almost certainly don't need more than you can get with a surplus server and a regular business grade fiber link.

This is very much not an all-or-nothing situation. There is a vast segment of industry that absolutely does not need anything more than a server in a closet wired to the internet connection your office already has. My last job paid $100/mo for an AWS instance to host a GitLab server for a team of 20. We could have gotten by with a junk laptop shoved in a corner and got the exact same performance and experience. It once borked itself after an update and railed the CPU for a week, which cost us a bunch of money. Would never have been an issue on-prem. Even if we got DDoSed or somehow stuck saturating the uplink, our added cost would be zero. Hell, the building was even solar powered, so we wouldn't have even paid for the extra 40W of power or the air conditioning.

It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior".

> AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.

The stench of this bovine excrement is so strong that it transcends space time somehow.

And does some of their suggested solutions actually work or not...

If you can't run locally, productivity drops like a rock. Each "cloud deploy" wastes tons of time.

And in a lot of cases it's hard to find out if a production application can be switched off. Since the cost is typically small for an unused application, I don't know if there are many people willing to risk being wrong

Buckets are used for backups, user uploads, and lots of things other than distributing files publicly.

The real question is if they considered caching and thus configured it appropriately. If you don't, you're telling everyone you want every request to go to origin

And it's getting harder and harder to make them public because of people misconfiguring them and then going public against AWS when they discover the bill.

You just shouldn't be using S3 to serve files directly. You can run most public and many private uses through CloudFront. Which gives you additional protections and reduces things like per object fetch costs.

> you hit natural rate limits

Seen by your customers or the public as a "denial of service." Which may actually be fine for the people who truly do want to limit their spending to less than $100/month.

The serverless function has higher-order features included as part of the package: you get an automatic runtime (just as with PHP but in this case it can be golang or dotnet), the function gets a unique endpoint URL, it can be triggered by events in other cloud services, you get execution logging (and basic alerting), multiple functions can be chained together (either with events or as a state machine), the function's compute can be automatically scaled up depending on the traffic, etc.

Think of it as: What do I have to do, in order to scale up the conpute of this URL? For hardware it's a call to DELL to order parts, for VMs or containers it's a matter of scaling up that runtime, or adding more instances - neither of those processes are simple to automate. One key characteristic of the function is that it will scale horizontally basically however much you want (not fully true, aws has a limit of 1500 instances/second iirc, but that's pretty massive), and it will do it automatically and without the request sources ever noticing.

Functions are also dirt cheap for low/burst traffic, and deployment is almost as easy as in the PHP FTP example. Personally I also think they are easier to test than traditional apps, due to their stateless nature and limited logical size (one endpoint). The main downsides are cost for sustained load, and latency for cold starts.

With that said, they are not "endgame". Just a tool - a great one for the right job.

I think figuring out how to do this faster is less trivial than it might sound. I agree that synchronous checks aren’t reasonable. But let’s take Lambdas. They can run for 15 minutes, and if you consolidate within five minutes after a resource has been billed, that gives you a twenty minute lag.

I’m not trying to make apologies for Amazon, mind you. Just saying that this isn’t exactly easy at scale, either. Sure, they bill by invocation, but that’s far from synchronous, too. In fact, getting alerts might very well be happening at the frequency of billing reconciliation, which might be an entirely reasonable thing to do. You could then argue that that process should happen more frequently, at Amazon’s cost.

The majority of these massive bills are due to traffic, there is pretty much no way that AWS could stop your server in time...if they had the choice, which they don't.

I think my original point was unclear: I am pointing out that if you just think about how this stuff can possibly work, billing alerts can not work in the way you expect. The alert is updated async, the horse has bolted and you are trying to shut the gate.

I don't use AWS for personal stuff because I know their billing alerts won't stop me spending a lot. Don't use them if that is a concern.

I do use AWS at work, we are a relatively big customer and it is still very expensive for what it is. The actual hardware is wildly overpriced, their services aren't particularly scalable (for us), and you are basically paying all that overage for network...which isn't completely faultless either. Imo, using them in a personal capacity is a poor idea.