Even with things like Python, CGI is pretty fast these days. If your CGI script takes a generous 400 milliseconds of CPU to start up and your server has 64 cores, you can serve 160 requests per second, which is 14 million hits per day per server. That's a high-traffic site.
That is, if your web service struggles to handle single-digit millions of requests per day, not counting static "assets", CGI process startup is not the bottleneck.
A few years ago I would have said, "and of course it's boring technology that's been supported in the Python standard library forever," but apparently the remaining Python maintainers are the ones who think that code stability and backwards compatibility with boring technology are actively harmful things, so they've been removing modules from the standard library if they are too boring and stable. I swear I am not making this up. The cgi module is removed in 3.13.
I'm still in the habit of using Python for prototyping, since I've been using it daily for most of the past 25 years, but now I regret that. I'm kind of torn between JS and Lua.
mjw1007 · 1h ago
The Python maintainers are removing the module _named_ cgi, but they're not removing the support for implementing CGI scripts, which is CGIHTTPRequestHandler in the http.server module.
All that was in the cgi module was a few functions for parsing HTML form data.
kragen · 57m ago
It would be very difficult indeed to make it impossible to implement CGI scripts in Python; you'd have to remove its ability to either read environment variables or perform stdio, crippling it for many other purposes, so I didn't think they had done that. Even if they removed the whole http package, you could just copy its contents into your codebase. It's not about making Python less powerful.
As a side note, though, CGIHTTPRequestHandler is for launching CGI programs (perhaps written in Rust) from a Python web server, not for writing CGI programs in Python, which is what the cgi module is for. And CGIHTTPRequestHandler is slated for removal in Python 3.15.
The problem is gratuitous changes that break existing code, so you have to debug your code base and fix the new problems introduced by each new Python release. It's usually fairly straightforward and quick, but it means you can't ship the code to someone who has Python installed but doesn't know it (they're dependent on you for continued fixes), and you can't count on being able to run code you wrote yourself on an earlier Python version without a half-hour interruption to fix it. Which may break it on the older Python version.
Amusingly that links to https://peps.python.org/pep-0206/ from 14th July 2000 (25 years ago!) which, even back then, described the cgi package as "designed poorly and are now near-impossible to fix".
That fails pretty hard at providing a rationale. Basically it says that CGI is an inefficient interface because it involves creating a new process! Even if that were true, "You shouldn't want to do such an inefficient thing" is very, very rarely a reasonable answer to a technical question like "How do I write a CGI script in Python?" or "How do I parse a CSV file in Python?"
There are certainly some suboptimal design choices in the cgi module's calling interface, things you did a much better job of in Django, but what made them "near-impossible to fix" was that at the time everyone reading and writing PEPs considered backwards compatibility to be not a bad thing, or even a mildly good thing, but an essential thing that was worth putting up with pain for. Fixing a badly designed interface is easy if you know what it should look like and aren't constrained by backwards compatibility.
pjmlp · 1h ago
Not to mention that if efficiency is a goal, probably Python isn't the language as well, so it is a very strange argument from Python developers.
kragen · 54m ago
It would have been a less strange argument 25 years ago, before the manycore era, when using Python involved less of a performance sacrifice. And there are still cases where Python is acceptably performant. However, the argument is from only 6 years ago, which makes it ridiculous.
riedel · 2h ago
Moving stuff out of the standard library seems like a reason. However, I think this all is a weird mix of arguments. IMHO new process spawning is a feature and not a bug in the use cases where CGI is used. Most of the stuff is low traffic config interfaces or remote invocable scripts. There was this trend to move stuff to fcgi. We had tons of cases of memory leaks in long running but really seldomly used stuff like mailing list servers. To me cgi is the poor man's alternative to serverless. However, I also do not really completely understand why a standard library has to support it. I have bash scripts running using the Apache CGI mod.
kragen · 2h ago
I would have phrased it, "serverless is a marketing term for CGI scripts."
I have bash CGI scripts too, though Shellshock and bash's general bug-proneness make me doubt that this was wise.
There are some advantages of having the CGI protocol implemented in a library. There are common input-handling bugs the library can avoid, it means that simple CGI programs can be really simple, and it lets you switch away from CGI when desired.
That said, XSS was a huge problem with really simple CGI programs, and an HTML output library that avoids that by default is more important than the input parsing—another thing absent from Python's standard library but done right by Django.
Right. Isn't that insane? If I hadn't found it by means of having modules removed that my code depended on, I would have thought it was satire.
That policy, and the heinous character assassination the PSF carried out against Tim Peters, mean I can no longer recommend in good conscience that anyone adopt Python.
vodou · 2h ago
I really understand your frustration. Everyone developing in Python for a long time has felt it a bit too often when breaking changes (even between minor version updates) once again ruins the day.
But I also understand that the world is not perfect. We all need to prioritize all the time. As they write in the rationale: "The team has limited resources, reduced maintenance cost frees development time for other improvements". And the cgi module is apparently even unmaintained.
I guess a "batteries included" philosophy sooner or later is caught up by reality.
What do you mean by "character assassination" carried out against Tim Peters? Not anything in the linked article I presume?
Kwpolska · 2m ago
He was banned for 3 months for opposing a change to the PSF bylaws that would allow the board to remove members with a simple majority vote.
No, it was an unrelated scandal. I don't have my bookmarks handy at the moment, so hopefully you can find a link.
As for prioritizing, I think the right choice is to deprioritize Python.
hot_gril · 2h ago
Idk about the internal affairs, I just really don't like Python for web backend kind of things. It's taking them way too long to sort out parallelism and packaging, while NodeJS got both right from the start and gracefully upgraded (no 2->3 mess).
Also I used Python way before JS, and I still like JS's syntax better. Especially not using whitespace for scope, which makes even less sense in a scripting language since it's hard to type that into a REPL.
eurleif · 13m ago
Node.js actually had no parallelism at the start, other than the ability to manually spawn new processes. Worker threads were only added in 2018 with v10.5.0, and only stabilized in 2019 with v12.
What Node.js had from the start was concurrency via asynchronous IO. And before Node.js was around to be JavaScript's async IO framework, there was a robust async IO framework for Python called Twisted. Node.js was influenced by Twisted[0], and this is particularly evident in the design of its Promise abstraction (which you had to use directly when it was first released, because JavaScript didn't add the async/await keywords until years later).
Yeah, generally I feel like the indentation sensitivity was the right idea (the alternative evidently being worse compiler error messages, bugs like the `goto fail` vulnerability, and greater verbosity) but it causes real difficulties with the REPL, as well as with shell one-liners.
Jupyter fixes the REPL problem, and it's a major advance in REPLs in a number of other ways, but it has real problems of its own.
jiggawatts · 2h ago
I don't even use Python, and even I've read Tim Peters' works and think highly of him! To have him so unceremoniously booted for upsetting a committee is absolutely insane.
This is a bit like Apple firing Steve Jobs for wearing sneakers to work because it violates some dress code.
pjmlp · 1h ago
I rather stick with PHP or JS, due to having a JIT in the box for such cases.
Since I learnt Python starting in version 1.6, it has mostly been for OS scripting stuff.
Too many hard learnt lessons with using Tcl in Apache and IIS modules, continuously rewriting modules in C, back in 1999 - 2003.
int0x29 · 16m ago
I don't think the JIT will help that much as each request will need to be JITed again. Unless Node and PHP are caching JIT output
WD-42 · 3h ago
I don't get it. Having a complaint about Python removing CGI from the stdlib is well and fine. But then you say you'd rather consider JS, which doesn't even have a std lib? Lua doesn't have a CGI module in stdlib either.
rollcat · 38m ago
> Lua doesn't have a CGI module in stdlib either.
Lua barely has any stdlib to speak of, most notably in terms of OS interfaces. I'm not even talking about chmod or sockets; there's no setenv or readdir.
You have to install C modules for any of that, which kinda kills it for having a simple language for CGI or scripting.
Don't get me wrong, I love Lua, but you won't get far without scaffolding.
kragen · 27m ago
Right, you need something more specific than Lua to actually write most complete programs in. The LuaJIT REPL does provide a C FFI by default, for example, so you don't need to install C modules.
But my concern is mostly not about needing to bring my own batteries; it's about instability of interfaces resulting from evaporating batteries.
kragen · 3h ago
I think it's fine to not have functionality in the standard library if it can be implemented by including some code in my project. It's not fine to have a standard library that stuff disappears from over time.
getdoneist · 1h ago
Ruby has been removing stuff from stdlib for some time now. But "moving" is the correct word, because it is simply moved to a stand-alone gem, and with packaging situation in Ruby being so good, it feels completely seamless.
spockz · 2h ago
Whenever code is removed from the Java standard library it is announced ages ahead of time and then typically it becomes available in a separate artefact so you can still use it if you depended on it.
kragen · 1h ago
I wrote an online securities trading system in Java, with a little Jython. Java is reasonably good at stability, but I find it unappealing for other reasons, especially for prototyping. Kotlin might be okay.
Jython no longer works with basically any current Python libraries because it never made the leap to Python 3, and the Python community stigmatizes maintaining Python 2 compatibility in your libraries. This basically killed Jython, and from my point of view, Jython was one of the best things about Java.
bravesoul2 · 3h ago
Node.js provides the defacto standard lib for JS backend and its got a good feature set.
That said these days I'd rather use Go.
kragen · 3h ago
Golang seems pretty comfortable from the stuff I've done in it, but it's not as oriented toward prototyping. It's more oriented toward writing code that's long-term maintainable even if that makes it more verbose, which is bad for throwaway code. And it's not clear how you'd use Golang to do the kind of open-ended exploration you can do in a Jupyter notebook, for example. How would you load new code into a Golang program that's already running?
Admittedly Python is not great at this either (reload has interacted buggily with isinstance since the beginning), but it does attempt it.
bravesoul2 · 2h ago
There are live reload utilities for Go but yeah it comes down to recompiling and restarting - which is quick but any state is gone.
I agree its not a rapid prototyping kind of language. AI assistance can help though.
kqr · 3h ago
Consider Perl. It's not quite as batteries-included as Python, but it is preinstalled almost everywhere and certainly more stable than JS and Lua. (And Python.)
rollcat · 22m ago
What's the landscape like, for when you need to scale your project up? As in: your project needs more structure, third-party integrations, atomic deployments, etc - not necessarily more performance.
Python has Werkzeug, Flask, or at the heavier end Django. With Werkzeug, you can translate your CGI business logic one small step at a time - it's pretty close to speaking raw HTTP, but has optional components like a router or debugger.
At the time Perl was the thing I used in the way I use Python now. I spent a couple of years after that working on a mod_perl codebase using an in-house ORM. I still occasionally reach for Perl for shell one-liners. So, it's not that I haven't considered it.
Lua is in a sense absolutely stable unless your C compiler changes under it, because projects just bundle whatever version of Lua they use. That's because new versions of Lua don't attempt backwards compatibility at all. But there isn't the kind of public shaming problem that the Python community has where people criticize you for using an old version.
JS is mostly very good at backwards compatibility, retaining compatibility with even very bad ideas like dynamically-typed `with` statements. I don't know if that will continue; browser vendors also seem to think that backwards compatibility with boring technology like FTP is harmful.
kqr · 3h ago
Ha, fun bit of history! Many of the listed problems with Perl can be configured away these days. I don't have time for a full list, but as two early examples:
- `perl -de 0` provides a REPL. With a readline wrapper, it gives you history and command editing. (I use comint-mode forn this, but there are other alternatives.)
- syscalls can automatically raise exceptions if you `use autodie`.
Why is this not the default? Because Perl maintainers value backward compatible. Improvements will always sit behind a line of config, preventing your scripts from breaking if you accidentally rely on functionality that later turns out to be a mistake.
That's a great read, thanks! And I didn't know about autodie, though I do use perl -de1 from time to time.
Perl feels clumsy and bug-prone to me these days. I do miss things like autovivification from time to time, but it's definitely bug-prone, and there are a lot of DWIM features in Perl that usually do the wrong thing, and then I waste time debugging a bug that would have been automatically detected in Python. If the default Python traceback doesn't make the problem obvious, I use cgitb.enable(format='text') to get a verbose stack dump, which does. cgitb is being removed from the Python standard library, though, because the maintainers don't know it can do that.
Three years ago, a friend told me that a Perl CGI script I wrote last millennium was broken: http://canonical.org/~kragen/sw/rfc-index.cgi. I hadn't looked at the code in, I think, 20 years. I forget what the problem was, but in half an hour I fixed it and updated its parser to be able to use the updated format IETF uses for its source file. I was surprised that it was so easy, because I was worse at writing maintainable code then.
Maybe we could do a better job of designing a prototyping language today than Larry did in 01994, though? We have an additional 31 years of experience with Perl, Python, JS, Lua, Java, C#, R, Excel, Haskell, OCaml, TensorFlow, Tcl, Groovy, and HTML to draw lessons from.
kqr · 1h ago
We can definitly do better than Perl. The easy proof is that modern Perl projects are supposed to start with a bunch of config to make Perl more sane, and many of them also include the same third-party libraries that e.g. improve exception handling and tweak the datetime functionality in the standard library.
One benefit Perl had that I think not many of the other languages do was being designed by a linguist. That makes it different -- hard to understand at first glance -- but also unusually suitable for prototyping.
kragen · 26m ago
What do you think a better design would look like?
bravesoul2 · 3h ago
Yeah high performance web used to be an art. Now it's find what you are doing that's stupidly wasteful that you did to ship fast, and stop doing that thing.
Your app could add almost no latency beyond storage if you try.
tonyhart7 · 2h ago
Yeah but 400ms is unacceptable this days
cenamus · 2h ago
If the whole site takes 5 seconds to fully hydrate and load its 20megs of JS I'll gladly take a server side rendered page that has finished loading in a second.
kragen · 2h ago
That's intended as an unreasonably high upper bound. On my cellphone, in Termux, python3 -m cgi takes 430–480ms. On my laptop it takes 90–150ms. On your server it probably takes less.
I agree that tens of milliseconds of latency is significant to the user experience, but it's not always the single most important consideration. My ping time to news.ycombinator.com is 162–164ms because I'm in Argentina, and I do unfortunately regularly have the experience of web page loads taking 10 seconds or more because of client-side JS.
nurettin · 1h ago
<PHP> I would like to have a word.
Tractor8626 · 3h ago
2400 rps on this hardware on hello world application - isn't it kinda bad?
And we trading performance for what exactly? Code certainly didn't become any simpler.
kqr · 3h ago
It's not great, but it is enough for many use cases. Should even handle a HN hug of death.
Tractor8626 · 2h ago
But why? What advantages we getting?
kqr · 1h ago
Hypothetically, strong modularisation, ease of deployment and maintenance, testability, compatibility with virtually any programming language.
In practise I'm not convinced -- but I would love to be. Reverse proxying a library-specific server or fiddling with FastCGI and alternatives always feels unnecessarily difficult to me.
slyall · 3h ago
It's only bad if you need to get more than 2000 rps
Which is only a small proportion of sites out there.
Tractor8626 · 2h ago
If there is no some other advantages - it is just bad.
masklinn · 1h ago
> It's only bad if you need to get more than 2000 rps
Or if you don't want to pay for an 8/16 for the sort of throughput you can get on a VPS with half a core.
gred · 1h ago
I'd rather not pay for 8 cores / 16 threads, though...
withinboredom · 50m ago
Depends on where you are shopping. I pay €211 every month for 96 threads and 384 gb of ram (clustered) -- disks are small (around 1tb each), but I'm still nowhere near 50% utilization there.
gred · 43m ago
Yeah, I pay $400/month to not be bothered with any installation or upgrade drama, ever. The problem is that I only get 16 threads.
johnisgood · 5m ago
I installed many Arch Linux for servers, they have been online for decades. There is no fuss. The only downtime was when I issued a reboot, but it was back in 5 seconds if not less.
So you really do not have to be bothered by installation or anything of these lines. You install once and you are fine. You should check out the Wiki pages of Arch Linux, for example. It is pretty straightforward. As for upgrades, Arch Linux NEVER broke. Not on my servers, and not on my desktop.
That said, to each their own.
kqr · 3h ago
I'd argue it's bad even if you get more than 1000 Bq of requests. You never want to approach 100 % utilisation, and I'd aim to stay clear of 50 %.
One big reason to avoid them was performance; it required extra disk access on every request and it was always better to put the configuration in the main config file if possible.
But now? When most servers have an SSD and probably spare RAM that Linux will use to cache the file system?
Ok, performance is still slightly worse as Apache has to parse the config on every request as opposed to once, but again, now that most servers have more powerfull CPU's? In many use cases you can live with that.
> I'm not a real programmer. I throw together things until it works then I move on. The real programmers will say "Yeah it works but you're leaking memory everywhere. Perhaps we should fix that." I’ll just restart Apache every 10 requests.
PHP got a very long way since then, but a huge part of that was correcting the early mistakes.
> PHP 8 is significantly better because it contains a lot less of my code.
faizshah · 7h ago
I’ve also thought about this moreso as part of a workflow for quickly prototyping stuff. At least for a lot of the modern JIT languages I believe their startup times will be dominated by your imports unless you go with a fastcgi model. This came up as I started adopting h2o web server for local scripts since it has clean and quick to write config files with mruby and fast-cgi handlers and is also crazy fast: https://h2o.examp1e.net/configure/fastcgi_directives.html
Another place this can be useful is for allowing customers to extend a local software with their own custom code. So instead of having to use say MCP to extend your AI tool they can just implement a certain request structure via CGI.
dolmen · 4h ago
An MCP frontend to CGI programs would not be a bad idea for a end user environment.
This makes me wonder if an MCP service couldn't be also implemented as CGI: an MCP framework might expose its feature as a program that supports both execution modes. I have to dig into the specs.
petesergeant · 7h ago
> The nascent web community quickly learned that this was a bad idea, and invented technologies like PHP
Well ackshually ... the technology here that was important was mod_php; PHP itself was no different to Perl in how it was run, but the design choice of mod_php as compared to mod_perl was why PHP scripts could just be dumped on the server and run fast, where you needed a small amount of thinking and magic to mod_perl working.
fcatalan · 3h ago
At that time I was developing with a friend what later was called a Learning Management System: It had content management, assignment uploads, event calendar, grade management, real time chat, forums... It was all plain C via CGI and it was hell to work with.
What almost brought us to tears the day we learned about PHP was how everything we had been painstakingly programming ourselves from scratch reading RFCs or reverse engineering HTTP was just a simple function call in PHP. No more debugging our scuffed urlencode implementation or losing a day to a stray carriage return in an HTTP header...
simonw · 7h ago
Right, but mod_php was an early addition to the PHP ecosystem and quickly became the default way of deploying it - I believe the first version of the Apache module was for PHP/FI Version 2.0 in 1996: https://www.php.net/manual/phpfi2.php#module
petesergeant · 7h ago
It was indeed, and I spent much time wailing and gnashing my teeth as a Perl programmer that nothing similar existed in Perl.
AdieuToLogic · 5h ago
> It was indeed, and I spent much time wailing and gnashing my teeth as a Perl programmer that nothing similar existed in Perl.
mod_perl2[0] provides the ability to incorporate Perl logic within Apache httpd, if not other web servers. I believe this is functionally equivalent to the cited PHP Apache module documentation:
Running PHP/FI as an Apache module is the most efficient
way of using the package. Running it as a module means that
the PHP/FI functionality is combined with the Apache
server's functionality in a single program.
The key difference was that you had to adapt your Perl to work with mod_perl, where mod_php "just worked" in the same way CGIs did -- you could throw your .php scripts up over FTP and they'd benefit from mod_php being installed. This was a massive difference in practice.
EDIT: I have managed to dig out slides from a talk I gave about this a million years ago with a good section that walks through history of how all this worked, CGIs, mod_perl, PSGI etc, for anyone who wants a brief history lesson: https://www.slideshare.net/slideshow/psgi-and-plack-from-fir...
pests · 4h ago
Just a story of my past.
I got into web dev in the tail end of perl and cgi-bin. I remember my first couple scripts which were just copy/paste from tutorials and what not, everyone knows how it goes. It was very magical to me how this "cgi-bin" worked. There was a "script kiddy hacking tool" I think named subseven (or similar) written partially in perl that you would trick your friends into running or you'd upload on filesharing. The perl part gave you your web based C&C to mess with people or open chats or whatever. I really got into programming trying to figure out how this all worked. I soon switched over to PHP and in my inexperience never realized the deployment model was so similar.
I do think this model of running the script once per request and then exiting really messed with my internal mental model of how programs and scripts worked. Once I was exposed to long running programs that could maintain state, their own internal data structures, handling individual requests in a loop, etc, was a real shock and took me awhile to conceptualize.
simonw · 4h ago
I had the same experience! I started out with Perl and CGI, then moved to PHP. Switching to a world where the web application kept running across multiple requests took me quite a bit of effort to get used to.
pests · 1h ago
It is so odd how request-centric my worldview was back then. I literally couldn’t fathom how “old legacy crusty languages” like c/c++ could possibly be behind a website. Learning google was built this way blew my mind.
It’s strange thinking back to the days where persisting information as simple as a view counter required persisting data to a flatfile* or something involving a database.
These days with node and our modern languages like go and rust it’s immediately obvious how it’s done.
I think it’s both a mix of me learning and growing and the industry evolving and growing, which I think all of us experience over time.
* for years using flat files was viewed as bad practice or amateurish. fun to learn years later that is how many databases work.
xnx · 6h ago
Indeed. Perl was better in many ways, but not in the one that mattered to its continued viability.
slashdave · 4h ago
Maybe my memory is bad, but I don't remember people jumping ship from CGI-bin because of performance. I do remember a lot of security problems.
kragen · 3h ago
I remember performance being the main reason people jumped ship from CGI in the period 01995–02002. The switch didn't solve security problems itself (except Shellshock, if you wrote CGI scripts in bash, but Shellshock wasn't publicly known until much later) but it sometimes came with a less slapdash approach to building web services which did solve security problems. On the other hand, it often instead came with a move to PHP, which had just unbelievable levels of security problems.
It's possible that your experience with people switching was later, when performance was no longer such a pressing concern.
sitharus · 3h ago
The main security issue I recall from CGI was caused by the web server having to execute the binary. This meant either executing as www-data, running the web server as root so it can call setuid, or using setuid binaries which have their own issues.
These were real issues on multi-user hosts, but as most of the time we don’t use shared hosting like that anymore it’s not an issue.
There were also some problems with libraries parsing the environment variables with the request data wrong, but that’s no different from a badly implemented http stack these days. I vaguely recall some issues with excessively log requests overflowing environment variables, but I can’t remember if that was a security problem or DoS.
aaronblohowiak · 7h ago
Ehhhhhhh. I believe but cannot cite that fork() got a lot cheaper over the last 30 years as well (independent of machine stats, I believe the Linux impl is inherently cheaper now but I can’t remember the details.) cgi bin works really well if you don’t have to pay for ssl or tcp connections to databases or other services, but you can maybe run something like istio if you need that. I have long thought that (fast)cgi is better model than proprietary “lambda” /“faas”. But the languages de jure and vendor lock in didn’t favor a standards based approach here.
AdieuToLogic · 5h ago
> I believe but cannot cite that fork() got a lot cheaper over the last 30 years as well ...
The fork[0] system call has been a relatively quick operation for the entirety of its existence. Where latency is introduced is in the canonical use of the execve[1] equivalent in newly created child process.
> ... cgi bin works really well if you don’t have to pay for ssl or tcp connections to databases or other services, but you can maybe run something like istio if you need that.
Istio[2] is specific to Kubernetes and thus unrelated to CGI.
I would gladly spend a little more on hardware if it makes the code much simpler and easier to debug.
eyberg · 8h ago
Outside of nostalgia there's no engineering reason to do this - definitely not for performance.
That same go program can easily go over 10k reqs/sec without having to spawn a process for each incoming request.
CGI is insanely slow and insanely insecure.
simonw · 8h ago
What makes it insecure? It's a pretty simple protocol - anything in there that makes is insecure beyond naive mistakes that could be avoided with a well designed library?
I agree that there's probably not much of an argument to switch to it from the well established alternative mechanisms we are using already.
The one thing in its favor is that it makes it easier to have a polyglot web app, with different languages used for different paths. You can get the same thing using a proxy server though.
eyberg · 7h ago
CGI has a very long history of security issues stemming primarily from input validation or the lack thereof.
AdieuToLogic · 5h ago
> CGI has a very long history of security issues stemming primarily from input validation or the lack thereof.
And a Go program reading from a network connection is immune from the same concerns how?
simonw · 7h ago
Right, but anything relating to input validation can be avoided by using a well designed library rather than implementing the protocol directly.
BirAdam · 7h ago
The language in use often has input validation libraries. The failure of the programmer to use them is not the fault of CGI. Further, proper administration of the machine can mitigate file injection, database injection, etc. Again, that people fail to do this isn’t the fault of CGI.
cranberryturkey · 7h ago
That's like saying forks and knives are vulnerable becuase you could stab someone with them.
tonyedgecombe · 4h ago
>EDIT: Looks like the way CGI works made it vulnerable to Shellshock in 2014:
From your linked article: If the handler is a Bash script, or if it executes Bash...
But we are talking about Python not Bash.
kragen · 3h ago
Yes, Shellshock is kind of a marginal case, but it probably does qualify as a security hole due in part to CGI itself, even though it doesn't affect Python programs (unless they spawn a shell). I don't know of any other examples of security problems caused by CGI, even partly. It's a very thin layer over HTTP.
slashdave · 3h ago
Because... it can execute an arbitrary executable? In the old days, it also ran as root.
kragen · 3h ago
No, on all the servers I have any experience with, it can only execute executables the server administrator configures as CGI programs, not executables supplied by an attacker, and they never ran as root. Apache in particular is universally run as a non-root user, since the very first release, and its suEXEC mechanism (used for running CGI programs as their owners for shared web hosting services) refuses to run any CGI program as root. I've never seen a web server on a Unix system running as root: not CERN httpd, not NCSA httpd, not Apache, not nginx, not python -m http.server, not any of the various web servers I've written myself.
I hesitate to suggest that you might be misremembering things that happened 30 years ago, but possibly you were using a very nonstandard setup?
sitharus · 3h ago
It definitely can’t. Either you had to put your script in cgi-bin, use an extension like .cgi in a directory with that feature explicitly enabled, or a magic sticky bit on the file if that was enabled.
You could configure the server to be insecure by, eg, allowing cgi execution from a directory where uploaded files are stored.
cperciva · 7h ago
Hey, it could be worse. Some people launch entire VMs to service individual requests.
klysm · 7h ago
That’s cloud-native!
giantrobot · 5h ago
You mean "Web Scale".
__float · 7h ago
Not everything needs 10k RPS, and in some sense there are benefits to a new process – how many security incidents have been caused by accidental cross-request state sharing?
And in a similar vein, Postgres (which is generally well liked!) uses a new backend process per connection. (Of course this has limitations, and sometimes necessitates pgbouncer, but not always.)
reddec · 4h ago
Few years ago I felt the same and created trusted-cgi.
However, through the years I learned:
- yes, forks and in general processes are fast
- yes, it saves memory and CPU on low load sites
- yes, it’s simple protocol and can be used even in shell
However,
- splitting functions (mimic serverless) as different binaries/scripts creates mess of cross scripts communication
- deployment is not that simple
- security wise, you need to run manager as root and use unique users for each script or use cgroups (or at least chroot). At that moment the main question is why not use containers asis
Also, compute wise, even huge Go app with hundreds endpoints can fit just few megabytes of RAM - there is no much sense to save so few memory.
At worst - just create single binary and run on demand for different endpoints
Tractor8626 · 4h ago
Even without pgbouncer postgres uses long lived connections (and long lived processes) . So bad example.
Uber famously switched from pg to mysql because their SWEs couldn't properly manage connections
anonzzzies · 3h ago
Was that the only reason? In our last testing (2021), on the same hardware and for our specific case (a billions of records database with many tables and specific workfloads), mysql consistently left postgres in the dust performance wise. Internal and external devs pointed out that probably postgres (or rather, our table structures & indexes) could be tweaked with quite a lot of work to be faster, but mysql performed well (for our purpose) even with some very naive options. I guess it depends on the case, but I cannot justify spending 1 cent (let alone far more as we have 100k+ tables) on something while something else is fast enough (by quite a margin) to begin with...
ivovk · 45m ago
It stops working when you need to connect to any external resource. Database, http clients etc maintain connection pools to skip initial connection phase, which can be costly. That’s why you usually need a running web application process
ben0x539 · 32m ago
At my last job, a lot of our web services also benefited immensely from in-process caches and batching (to be fair, some of them were the cache for downstream services), and their scaling requirements pretty much dominated our budget.
I can totally see how the cgi-bin process-per-request model is viable in a lot of places, but when it isn't, the difference can be vast. I don't think we'd have benefited from the easier concurrency either, but that's probably just because it was all golang to begin with.
8organicbits · 30m ago
You can solve that with a sidecar, a dedicated process (or container) that pools connections for you. Pgbouncer as one example.
That is, if your web service struggles to handle single-digit millions of requests per day, not counting static "assets", CGI process startup is not the bottleneck.
A few years ago I would have said, "and of course it's boring technology that's been supported in the Python standard library forever," but apparently the remaining Python maintainers are the ones who think that code stability and backwards compatibility with boring technology are actively harmful things, so they've been removing modules from the standard library if they are too boring and stable. I swear I am not making this up. The cgi module is removed in 3.13.
I'm still in the habit of using Python for prototyping, since I've been using it daily for most of the past 25 years, but now I regret that. I'm kind of torn between JS and Lua.
All that was in the cgi module was a few functions for parsing HTML form data.
As a side note, though, CGIHTTPRequestHandler is for launching CGI programs (perhaps written in Rust) from a Python web server, not for writing CGI programs in Python, which is what the cgi module is for. And CGIHTTPRequestHandler is slated for removal in Python 3.15.
The problem is gratuitous changes that break existing code, so you have to debug your code base and fix the new problems introduced by each new Python release. It's usually fairly straightforward and quick, but it means you can't ship the code to someone who has Python installed but doesn't know it (they're dependent on you for continued fixes), and you can't count on being able to run code you wrote yourself on an earlier Python version without a half-hour interruption to fix it. Which may break it on the older Python version.
Amusingly that links to https://peps.python.org/pep-0206/ from 14th July 2000 (25 years ago!) which, even back then, described the cgi package as "designed poorly and are now near-impossible to fix".
Looks like the https://github.com/jackrosenthal/legacy-cgi package provides a drop-in replacement for the standard library module.
There are certainly some suboptimal design choices in the cgi module's calling interface, things you did a much better job of in Django, but what made them "near-impossible to fix" was that at the time everyone reading and writing PEPs considered backwards compatibility to be not a bad thing, or even a mildly good thing, but an essential thing that was worth putting up with pain for. Fixing a badly designed interface is easy if you know what it should look like and aren't constrained by backwards compatibility.
I have bash CGI scripts too, though Shellshock and bash's general bug-proneness make me doubt that this was wise.
There are some advantages of having the CGI protocol implemented in a library. There are common input-handling bugs the library can avoid, it means that simple CGI programs can be really simple, and it lets you switch away from CGI when desired.
That said, XSS was a huge problem with really simple CGI programs, and an HTML output library that avoids that by default is more important than the input parsing—another thing absent from Python's standard library but done right by Django.
That policy, and the heinous character assassination the PSF carried out against Tim Peters, mean I can no longer recommend in good conscience that anyone adopt Python.
But I also understand that the world is not perfect. We all need to prioritize all the time. As they write in the rationale: "The team has limited resources, reduced maintenance cost frees development time for other improvements". And the cgi module is apparently even unmaintained.
I guess a "batteries included" philosophy sooner or later is caught up by reality.
What do you mean by "character assassination" carried out against Tim Peters? Not anything in the linked article I presume?
https://www.theregister.com/2024/08/09/core_python_developer...
https://tim-one.github.io/psf/ban
https://chrismcdonough.substack.com/p/the-shameful-defenestr...
As for prioritizing, I think the right choice is to deprioritize Python.
Also I used Python way before JS, and I still like JS's syntax better. Especially not using whitespace for scope, which makes even less sense in a scripting language since it's hard to type that into a REPL.
What Node.js had from the start was concurrency via asynchronous IO. And before Node.js was around to be JavaScript's async IO framework, there was a robust async IO framework for Python called Twisted. Node.js was influenced by Twisted[0], and this is particularly evident in the design of its Promise abstraction (which you had to use directly when it was first released, because JavaScript didn't add the async/await keywords until years later).
[0] https://nodejs.org/en/about
Jupyter fixes the REPL problem, and it's a major advance in REPLs in a number of other ways, but it has real problems of its own.
This is a bit like Apple firing Steve Jobs for wearing sneakers to work because it violates some dress code.
Since I learnt Python starting in version 1.6, it has mostly been for OS scripting stuff.
Too many hard learnt lessons with using Tcl in Apache and IIS modules, continuously rewriting modules in C, back in 1999 - 2003.
Lua barely has any stdlib to speak of, most notably in terms of OS interfaces. I'm not even talking about chmod or sockets; there's no setenv or readdir.
You have to install C modules for any of that, which kinda kills it for having a simple language for CGI or scripting.
Don't get me wrong, I love Lua, but you won't get far without scaffolding.
But my concern is mostly not about needing to bring my own batteries; it's about instability of interfaces resulting from evaporating batteries.
Jython no longer works with basically any current Python libraries because it never made the leap to Python 3, and the Python community stigmatizes maintaining Python 2 compatibility in your libraries. This basically killed Jython, and from my point of view, Jython was one of the best things about Java.
That said these days I'd rather use Go.
Admittedly Python is not great at this either (reload has interacted buggily with isinstance since the beginning), but it does attempt it.
I agree its not a rapid prototyping kind of language. AI assistance can help though.
Python has Werkzeug, Flask, or at the heavier end Django. With Werkzeug, you can translate your CGI business logic one small step at a time - it's pretty close to speaking raw HTTP, but has optional components like a router or debugger.
At the time Perl was the thing I used in the way I use Python now. I spent a couple of years after that working on a mod_perl codebase using an in-house ORM. I still occasionally reach for Perl for shell one-liners. So, it's not that I haven't considered it.
Lua is in a sense absolutely stable unless your C compiler changes under it, because projects just bundle whatever version of Lua they use. That's because new versions of Lua don't attempt backwards compatibility at all. But there isn't the kind of public shaming problem that the Python community has where people criticize you for using an old version.
JS is mostly very good at backwards compatibility, retaining compatibility with even very bad ideas like dynamically-typed `with` statements. I don't know if that will continue; browser vendors also seem to think that backwards compatibility with boring technology like FTP is harmful.
- `perl -de 0` provides a REPL. With a readline wrapper, it gives you history and command editing. (I use comint-mode forn this, but there are other alternatives.)
- syscalls can automatically raise exceptions if you `use autodie`.
Why is this not the default? Because Perl maintainers value backward compatible. Improvements will always sit behind a line of config, preventing your scripts from breaking if you accidentally rely on functionality that later turns out to be a mistake.
https://entropicthoughts.com/you-want-technology-with-warts
https://entropicthoughts.com/why-perl
Perl feels clumsy and bug-prone to me these days. I do miss things like autovivification from time to time, but it's definitely bug-prone, and there are a lot of DWIM features in Perl that usually do the wrong thing, and then I waste time debugging a bug that would have been automatically detected in Python. If the default Python traceback doesn't make the problem obvious, I use cgitb.enable(format='text') to get a verbose stack dump, which does. cgitb is being removed from the Python standard library, though, because the maintainers don't know it can do that.
Three years ago, a friend told me that a Perl CGI script I wrote last millennium was broken: http://canonical.org/~kragen/sw/rfc-index.cgi. I hadn't looked at the code in, I think, 20 years. I forget what the problem was, but in half an hour I fixed it and updated its parser to be able to use the updated format IETF uses for its source file. I was surprised that it was so easy, because I was worse at writing maintainable code then.
Maybe we could do a better job of designing a prototyping language today than Larry did in 01994, though? We have an additional 31 years of experience with Perl, Python, JS, Lua, Java, C#, R, Excel, Haskell, OCaml, TensorFlow, Tcl, Groovy, and HTML to draw lessons from.
One benefit Perl had that I think not many of the other languages do was being designed by a linguist. That makes it different -- hard to understand at first glance -- but also unusually suitable for prototyping.
Your app could add almost no latency beyond storage if you try.
I agree that tens of milliseconds of latency is significant to the user experience, but it's not always the single most important consideration. My ping time to news.ycombinator.com is 162–164ms because I'm in Argentina, and I do unfortunately regularly have the experience of web page loads taking 10 seconds or more because of client-side JS.
And we trading performance for what exactly? Code certainly didn't become any simpler.
In practise I'm not convinced -- but I would love to be. Reverse proxying a library-specific server or fiddling with FastCGI and alternatives always feels unnecessarily difficult to me.
Which is only a small proportion of sites out there.
Or if you don't want to pay for an 8/16 for the sort of throughput you can get on a VPS with half a core.
So you really do not have to be bothered by installation or anything of these lines. You install once and you are fine. You should check out the Wiki pages of Arch Linux, for example. It is pretty straightforward. As for upgrades, Arch Linux NEVER broke. Not on my servers, and not on my desktop.
That said, to each their own.
There is work-arounds but usually it a better idea to ditch PHP for a better technology more suited for modern web.
And for your information, you can have stateful whatnots in PHP. Hell, you can have it in CSS as I have demonstrated in my earlier comments.
This let's you drop .htaccess files anywhere and Apache will load them on each request for additional server config. https://httpd.apache.org/docs/2.4/howto/htaccess.html
One big reason to avoid them was performance; it required extra disk access on every request and it was always better to put the configuration in the main config file if possible.
But now? When most servers have an SSD and probably spare RAM that Linux will use to cache the file system?
Ok, performance is still slightly worse as Apache has to parse the config on every request as opposed to once, but again, now that most servers have more powerfull CPU's? In many use cases you can live with that.
[ Side project is very early version but I'm already using it: https://github.com/StaticPatch/StaticPatch/tree/main ]
> I'm not a real programmer. I throw together things until it works then I move on. The real programmers will say "Yeah it works but you're leaking memory everywhere. Perhaps we should fix that." I’ll just restart Apache every 10 requests.
PHP got a very long way since then, but a huge part of that was correcting the early mistakes.
> PHP 8 is significantly better because it contains a lot less of my code.
Another place this can be useful is for allowing customers to extend a local software with their own custom code. So instead of having to use say MCP to extend your AI tool they can just implement a certain request structure via CGI.
This makes me wonder if an MCP service couldn't be also implemented as CGI: an MCP framework might expose its feature as a program that supports both execution modes. I have to dig into the specs.
Well ackshually ... the technology here that was important was mod_php; PHP itself was no different to Perl in how it was run, but the design choice of mod_php as compared to mod_perl was why PHP scripts could just be dumped on the server and run fast, where you needed a small amount of thinking and magic to mod_perl working.
What almost brought us to tears the day we learned about PHP was how everything we had been painstakingly programming ourselves from scratch reading RFCs or reverse engineering HTTP was just a simple function call in PHP. No more debugging our scuffed urlencode implementation or losing a day to a stray carriage return in an HTTP header...
mod_perl2[0] provides the ability to incorporate Perl logic within Apache httpd, if not other web servers. I believe this is functionally equivalent to the cited PHP Apache module documentation:
0 - https://perl.apache.org/docs/2.0/index.htmlEDIT: I have managed to dig out slides from a talk I gave about this a million years ago with a good section that walks through history of how all this worked, CGIs, mod_perl, PSGI etc, for anyone who wants a brief history lesson: https://www.slideshare.net/slideshow/psgi-and-plack-from-fir...
I got into web dev in the tail end of perl and cgi-bin. I remember my first couple scripts which were just copy/paste from tutorials and what not, everyone knows how it goes. It was very magical to me how this "cgi-bin" worked. There was a "script kiddy hacking tool" I think named subseven (or similar) written partially in perl that you would trick your friends into running or you'd upload on filesharing. The perl part gave you your web based C&C to mess with people or open chats or whatever. I really got into programming trying to figure out how this all worked. I soon switched over to PHP and in my inexperience never realized the deployment model was so similar.
I do think this model of running the script once per request and then exiting really messed with my internal mental model of how programs and scripts worked. Once I was exposed to long running programs that could maintain state, their own internal data structures, handling individual requests in a loop, etc, was a real shock and took me awhile to conceptualize.
It’s strange thinking back to the days where persisting information as simple as a view counter required persisting data to a flatfile* or something involving a database.
These days with node and our modern languages like go and rust it’s immediately obvious how it’s done.
I think it’s both a mix of me learning and growing and the industry evolving and growing, which I think all of us experience over time.
* for years using flat files was viewed as bad practice or amateurish. fun to learn years later that is how many databases work.
It's possible that your experience with people switching was later, when performance was no longer such a pressing concern.
These were real issues on multi-user hosts, but as most of the time we don’t use shared hosting like that anymore it’s not an issue.
There were also some problems with libraries parsing the environment variables with the request data wrong, but that’s no different from a badly implemented http stack these days. I vaguely recall some issues with excessively log requests overflowing environment variables, but I can’t remember if that was a security problem or DoS.
The fork[0] system call has been a relatively quick operation for the entirety of its existence. Where latency is introduced is in the canonical use of the execve[1] equivalent in newly created child process.
> ... cgi bin works really well if you don’t have to pay for ssl or tcp connections to databases or other services, but you can maybe run something like istio if you need that.
Istio[2] is specific to Kubernetes and thus unrelated to CGI.
0 - https://man.freebsd.org/cgi/man.cgi?query=fork&apropos=0&sek...
1 - https://man.freebsd.org/cgi/man.cgi?query=execve&sektion=2&a...
2 - https://istio.io/
"A brief, incomplete and largely inaccurate history of dynamic webpages"
https://www.slideshare.net/slideshow/psgi-and-plack-from-fir...
That same go program can easily go over 10k reqs/sec without having to spawn a process for each incoming request.
CGI is insanely slow and insanely insecure.
EDIT: Looks like the way CGI works made it vulnerable to Shellshock in 2014: https://en.m.wikipedia.org/wiki/Shellshock_(software_bug)
I agree that there's probably not much of an argument to switch to it from the well established alternative mechanisms we are using already.
The one thing in its favor is that it makes it easier to have a polyglot web app, with different languages used for different paths. You can get the same thing using a proxy server though.
And a Go program reading from a network connection is immune from the same concerns how?
From your linked article: If the handler is a Bash script, or if it executes Bash...
But we are talking about Python not Bash.
I hesitate to suggest that you might be misremembering things that happened 30 years ago, but possibly you were using a very nonstandard setup?
You could configure the server to be insecure by, eg, allowing cgi execution from a directory where uploaded files are stored.
And in a similar vein, Postgres (which is generally well liked!) uses a new backend process per connection. (Of course this has limitations, and sometimes necessitates pgbouncer, but not always.)
However, through the years I learned:
- yes, forks and in general processes are fast - yes, it saves memory and CPU on low load sites - yes, it’s simple protocol and can be used even in shell
However,
- splitting functions (mimic serverless) as different binaries/scripts creates mess of cross scripts communication - deployment is not that simple - security wise, you need to run manager as root and use unique users for each script or use cgroups (or at least chroot). At that moment the main question is why not use containers asis
Also, compute wise, even huge Go app with hundreds endpoints can fit just few megabytes of RAM - there is no much sense to save so few memory.
At worst - just create single binary and run on demand for different endpoints
Uber famously switched from pg to mysql because their SWEs couldn't properly manage connections
I can totally see how the cgi-bin process-per-request model is viable in a lot of places, but when it isn't, the difference can be vast. I don't think we'd have benefited from the easier concurrency either, but that's probably just because it was all golang to begin with.