Frustrated at Young Programmers
A younger coworker told me that aws lambdas are cheap and we should use sns to spin up an entire aws lambda to have the server wait on an aws lambda while the lambda waits for a database. Are you kidding?
My suggestion was then compared to using assembly language to program the entire process as if my suggestion was completely absurd. Was I being trolled or are young programmers just drunk with over architecting everything?
How common is it for there to be architecture at companies where servers are triggering lambdas only to have the lambda wait on a database call?
Not to mention they wanted this to be an entirely new repo that held async tasks with a new deployment infra. "Microservices and proper architecture" they called it. Seriously? So now I have to have two PRs and two deploys to get a feature in place when it could have been one? Also everything is slower and more complicated? Do people not get it?
I'd try to show how much a single process could scale by writing a simple benchmark, and comparing the performance, cost and latency of the alternative.
The results should speak for themselves at that point.
(But also, spinning up a lambda to wait for a db request? Wat? Must be missing some more context/details)
I think there's a more pervasive problem of all engineers, especially younger ones not realizing that spinning up new serverless infra is harder then firing a coroutine.
>(But also, spinning up a lambda to wait for a db request? Wat? Must be missing some more context/details)
there's this crud server that serves as an admin view for a bunch of IOT devices. Sometimes we want to run a task that doesn't block this server. So they use lambdas for those external task runners.
Except those tasks are mostly database operations. It is a crud app after all. But I'm trying to tell them that you don't need to spin up a new CPU when the database is doing all the computation. Have the server wait for the database. Don't have the server wait for a lambda that waits for the Database. If you do that you're just spinning up a CPU and telling it to wait. It's more complicated.
At my company they think doing this is normal. It's pervasive how normal people feel this type of thing is. I'm just baffled. I'm wondering maybe it's just pervasive across the board. I don't think they realized that query calls on lambdas just make the lambdas wait on the database. The abstraction is so heavy they likely aren't even thinking about how they are provisioning an entirely new CPU to wait for a database.
Only a troll would defend that two CPUs are cheaper than one. Especially if you already have this one.
I'm a web developer, not even specialized in backend and I could follow and agree with what you said.
Hope you get better people to interact with next time :P
Why wait for an aws lambda to wait for a database when you can just wait for the database yourself?
You spin up an entire CPU just to ask it to wait for something when you can just fire off a coroutine. Spinning up CPUs are for spinning up new computational resources. It's pointless to ask it to wait for shit. It's like asking someone to wait in line for you while you stand next to him waiting for him.
It's a little bit like the difference between writing a monolithic piece of software and writing a bunch of command-line utilities and scripting them together using the shell.
Why would you autoscale more computation to wait for the database. Let’s be clear here. The database is the thing doing MORE computations. It makes more sense to autoscale the thing carrying out the task.
It makes zero sense to autoscale the thing simply waiting for the task. The only thing where I see this making sense is if your database already has massive compute resources allocated to it and your server has so little resources it can’t wait on the database. In that case autoscaling cpus on the requester side makes sense but this is rare. It’s so rare it’s minuscule that you’ll have more than 10k concurrent connections.
>But you also create clearer language/runtime independent code encapsulation.
I don’t understand why a folder isn’t clear? Why must it be an entirely new repo and a new deploy process and an entirely new allocation of cpus to wait for the database? Guys. You want to organize your code? Consider folders. Don’t use new repos. Don’t use new deploy processes. Infrastructure exists to scale not to organize code. Adding new infrastructure makes code complexity, organization and accessibility worse while increasing scalability. Don’t use infra to organize code. Use infrastructure to scale performance. That’s a rule of thumb. There are exceptions but those exceptions are rare af.
You want other services to touch this function? You can then make it an http route as well: So the server can trigger a coroutine in the handler OR the handler itself (which is also a coroutine) is the database call that reacts to other services calling the http route. Same effect as a lambda.
>It's a little bit like the difference between writing a monolithic piece of software and writing a bunch of command-line utilities and scripting them together using the shell.
Highly disagree. Imagine one of those command line utilities is “ls”. This overreliance on lambdas is akin to adding new Computers to the cluster in order to call ls.
You might be billed for a new CPU for each request, but in practice this may not be how it's implemented by your provider. It may not be any more computationally expensive than Async callbacks you would implement yourself anyway.
> Highly disagree. Imagine one of those command line utilities is “ls”. This overreliance on lambdas is akin to adding new Computers to the cluster in order to call ls.
If your application needs to walk the files in a directory by name, you don't need to use 'ls' either. You can use standard library routines instead. However, by shipping 'ls' as a separate binary it is now significantly easier to reuse.
There isn't a universal best solution. But serverless does have technical merits.
Independent of costs, it's far easier to commit an async callback to your existing code then spin up new infra structure. This isn't just a cost issue. It's 100x simpler to use your existing code to do this.
>If your application needs to walk the files in a directory by name, you don't need to use 'ls' either. You can use standard library routines instead. However, by shipping 'ls' as a separate binary it is now significantly easier to reuse.
This is just an example of the ludicrousness of provisioning an external CPU to do it. What you are suggesting with a lambda is equal in stupidity. But for your example here, I'm saying we agree that it's stupid to provision an entire CPU to run ls. Where we don't agree is that you think it's fine to provision an entire CPU to wait for another database CPU.
>There isn't a universal best solution. But serverless does have technical merits.
First off, there is. Don't spin up a CPU and all the related infra to wait for a database when your existing server CAN already do this. There is no trade off here. When a server waits for a lambda that waits for a database it is definitively slower then the server just waiting for the database. To top it off it is also more expensive, and more complicated in infrastructure. Literally. No. Tradeoff. How can you be talking apples vs. organges when one solution is obviously superior?
Second I never said serverless doesn't have technical merits. I said using lambdas as task runners along side an existing server to do Database calls has no merit.
It's totally fine to use lambdas to do CPU intensive operations. That would be the point of a lambda right? That's why lambdas exist? If you don't have a server and you want to use "serverless architecture" this makes sense. Use the CPU to do computations.... don't use the CPU to wait on another CPU (the database) to do computations. That's kind of wtf??
I'll grant you that for "serverless architecture" at the very least you need to provision a CPU create the query in the first place. This makes sense. But if you already have some fargate server or ec2 instance.... why??
I can only assume that you are responsible for some major architectural decision where a server spins up a lambda to wait for the lambda, while the lambda waits for the database. Basically by being responsible for something that uses lambdas entirely incorrectly you become too biased to admit that something is obviously wrong. Literally you can't argue with the absurdity of spending more effort to provision and pay for additional CPUs to wait for the database when your existing server already can do the waiting. Spinning up more computer power to only pass off the actual computing action to a database? How many times have I repeated that this is absurd?
It sounds like they do not respect your take on it, but you aren't respecting theirs, either.
If your server is your typical crud server that hits a database, your database will be overwhelmed before your server gets hit to the point where it can't even wait.
It's not about respect, I don't mind the disrespect. It's more about how they're wrong and they don't realize it and this over engineering is pervasive in the industry.
I don't think you understanding how cheap it is on the CPU for a server to wait for an external IO call.
But do they mind? Because so far you are saying anyone who doesn't agree with you doesn't understand. Sounds like while you might not mind being disrespected, you fail to realize that other people do mind.
You also clearly have an over-sized server if you think 10K concurrent connections can be covered on it. If that is true, you could save a heap of money by downgrading that server and going to lambdas. The more you defend your position, the less tenable it becomes.
Is that disrespect or is that just sensitivity? From my POV this is fact. Like 1 + 1 = 2. I'm stating that. Or should I be inclusive and say that 1 + 1 maybe equals 2 but I respect your alternative opinion?
If I believe in something as a fact. I should be able to talk about it as a fact and talk about it as if the other person is wrong. You have that right as well. Same with my young coworker.
They probably do mind. But I feel that's just life. He called my way of doing it using "assembly language" even, and I let it go. If someone is wrong, we shouldn't be afraid to slightly offend them and tell them they are wrong. That goes both ways. I don't want to live in society where we have to dance around everyone's feelings even when they're utterly wrong and borderline delusional.
>You also clearly have an over-sized server if you think 10K concurrent connections can be covered on it. If that is true, you could save a heap of money by downgrading that server and going to lambdas. The more you defend your position, the less tenable it becomes.
This is not true. Your laptop on a 10 year old intel chip can handle 10k concurrent connections. Some guy pushed it up to a million here: https://unetworkingab.medium.com/millions-of-active-websocke...
The lambda architecture doesn't work for our case because our server maintains long running websocket connections. We just need async non blocking tasks. So I'm saying, launch a concurrent coroutine (aka go routine) and call it a day.
I think you're not aware of how much concurrency modern servers can handle. Back in the days of LAMP this would be a problem. But not anymore.
In short: The server is the origin of the task. By definition the task doesn't exist without the origin/trigger.
Why spend extra on a lambda? I think you're lost. Especially if you built architecture like this where servers wait for lambdas that just wait for database calls. (FYI waiting time is slightly longer this way, cost is slightly more expensive, and architecture becomes unnecessarily complex)
Literally first reply is someone who thinks a lambda is the right tool. I wasn't sure how bad it was until this young man replied.