> When the client sends a hello, the firewall says “that looks like a TLS hello”, and then waits for the server’s response. It inspects the certificate and then applies any rules.
This kind of stuff is exactly why TLS 1.3 encrypts everything now.
rkagerer · 5h ago
In TLS 1.3, the server’s certificate is encrypted, which means the firewall can’t do this anymore. ... The firewall will open a second connection and do a TLS 1.2 handshake to retrieve the certificate in plaintext. If the rules allow the connection, the firewall then allow the connection to proceed.
Sounds like a cat & mouse game. How long before we get servers that respond with different certificates based on who"s asking or other cues from the connection, in efforts to bypass the firewall restrictions.
tialaramex · 4h ago
Sure, this already happens today as in practice pre-standard ECH is deployed, so client says "Hi I'm calling some.boring.example [encrypted: Actually clown-porn.example]" and this useless middlebox cannot decrypt the inner message so it retrieves the some.boring.example certificate but the client's encrypted connection reaches clown-porn.example which may have a different certificate.
For many years, long before TLS 1.3 the RFC explains how to do this properly, obviating the problem, but that's very expensive because you need to TLS proxy every single connection, you can impose oversight by - literally - adding an actual oversight layer, which your users can also see you did. The popular middlebox products say they're doing two things, they're cheaper (maybe you buy their $100k product instead of a $100M solution) and they're less "intrusive" (ie you needn't tell your users that you're spying on them)
In reality they're ineffective, which is why the RFC says not to do this - but they can either outright lie or hide this fact in an asterisked disclaimer somewhere, and most of their customers don't care whether it actually works, they want to tick a box.
kevvok · 6h ago
Years of middleboxes tampering with TCP and barfing on anything they didn’t recognize is why QUIC encrypts everything as well
silverwind · 2h ago
> QUIC encrypts everything
QUIC is built on top of TLS 1.3 where client hello encryption is not mandatory, so this is not strictly true.
tialaramex · 5h ago
Technically TLS 1.3 doesn't encrypt the Client Hello, and indeed the first stanza of the Server Hello isn't encrypted either, although it deliberately doesn't say anything you could use as a third party to make decisions and the rest of a TLS 1.3 Server Hello is encrypted.
The in-progress Encrypted Client Hello (was Encrypted Server Name Indication hence esni in the name of the document) fixes that and you probably have software which uses it though the work to tie up all the loose ends up and publish a document might take some time yet.
Because the middlebox vendors are _so_ incompetent not only does TLS 1.3 need to work around their nonsense, thus proving that it was never useful security (an attacker could always have just done what TLS 1.3 does and it would have bypassed this worthless garbage) some of them screwed up badly enough that the anti-downgrade feature trips, to their credit Google refused to ship a permanent workaround for this, the workaround they shipped in Chrome sunset in about a year & required an explicit key setting, so basically "I acknowledge that I have defective middleboxes and must remove or upgrade them soon" by the local administrators.
But yes, it's noticeable that it was much easier to sell some engineers on "Thanks to TLS 1.3 now this stupid middlebox product won't be able to make your service slower" than say, "Thanks to TLS 1.3 now this stupid middlebox product won't report to the government if you read a Wikipedia article which contradicts its dogma".
yardstick · 4h ago
Encrypted SNI was never about hiding which specific articles in Wikipedia you are looking at. It was about hiding the fact you are looking at Wikipedia at all.
There are valid use cases for TLS middleboxes. Anyone having to secure a networks outbound access to only essential services has run into the “I have to allow all of AWS/GCP/Azure/CloudFlare/…” for some critical tool to work.
Options are:
- Allow all out (nope, not secure)
- Allow all to the cloud providers IP range (still terrible)
- Filter on SNI but don’t inspect the payload (better than no security, and doesn’t require plaintext access).
- Full MITM TLS proxy (performance bottleneck, and now we have plaintext access to all your data, which we really don’t want and didn’t previously require to do the filtering).
- Try convince the third party service to run on a handful of static IPs that aren’t behind a global load balancer with access to the rest of the cloud providers customer domains. (Yeah right)
See: Hospitals. Payment Networks. IoT networks.
tialaramex · 3h ago
> There are valid use cases for TLS middleboxes
There are, and always have been, people who are sure that they "need" to do things which don't work because the correct solution would be inconvenient for them. Those people should instead suck it up and accept the inconvenience or, as is more likely, remember they don't "need" this after all considering how inconvenient it is.
It won't suddenly work just because that would be more convenient and it's very annoying technically that we have to keep having this conversation, it's not going to stop being true just because that would make your life easier.
silverwind · 2h ago
Imho, clients like browsers should just indicate to the user whether their traffic can be intercepted or not (fully encrypted including client hello).
And those clients should offer an option to downgrade the TLS connection to make traffic interceptable.
yardstick · 3h ago
Well, just give up using shared hosting for services that need to be locked down. Ain’t going to happen though, because a different and at times overlapping group of people would be inconvenienced.
silverwind · 2h ago
Unfortunately, Encrypted Client Hello isn't automatic and requires cumbersome DNS configuration, so it's likely not going to gain widespread adoption.
Here's hoping that the next TLS version will automatically encrypt everything. Firewall vendors will not like it, but it's the only way to truly hide everything from middleboxes.
tialaramex · 1h ago
Magic is impossible. In order to encrypt a message to Bob, Alice must first have Bob's key.
That's why ECH needs DNS records, Bob publishes a key, "Here's the key for all six thousand blog.example sites we publish" and Alice can then encrypt either "Hi Bob, give me clown-porn.blog.example" or "Hi Bob, give me trans-rights.blog.example" and the middlebox can't read it.
We can't stop the middlebox from saying well, Bob publishes a trans rights blog so now all Bob's blogs are inaccessible just in case. Nor can we prevent Bob from deciding to publish separate keys, "these are for the clown porn and lynching videos, we'll use separate keys for the trans-rights stuff because the Government says that is naughty". But we can make that Bob's choice.
To do that, Alice needs Bob's key and either Alice has to magically know everybody's keys (which doesn't scale) or it is published somewhere for Alice to find, in this case DNS.
If we just say OK we'll encrypt with no particular recipient in mind, that doesn't prevent the scenario we care about, Alice encrypts the message, the middlebox decrypts it (No reason it can't, Alice can't pick Bob as the only recipient) and the middlebox gets to inspect Alice's destination.
hypeatei · 2h ago
Encrypted client hello can be stripped easily by modern firewalls[0]. I know because my employers network enables this functionality and it forces a downgrade so that it can see what you're trying to access.
Direct TLS can speed up your postgreSQL connection
prussian · 1h ago
True, it can help Microsoft SQL Server as well. In SQL Server 2022, they finally added Strict Encryption.
I'm glad to see more databases are removing these strange STARTTLS like features.
aflukasz · 5h ago
And mostly if you are behind CISCO firewall during TLS Server Identity Discovery or some equivalent setup. 3 seconds mentioned in the article were coming mostly from that. From the text itself it's not clear how much gains come from sslnegotiation=direct itself (if we assume no other factors like those present in this case).
aflukasz · 25m ago
For those interested - I've checked and observed a difference of 0.2ms on average across 1000 connection attempts on localhost.
smallpipe · 5h ago
"Having a dumb firewall can slow down your connections"
sylware · 4h ago
Is there a conflict or significant redundancy between QUIC TLS and DTLS?
tialaramex · 3h ago
There will be (is? under standards development?) a DTLS replacement built on QUIC but today these are orthogonal technologies. QUIC makes you a TCP-but-encrypted while DTLS is more TLS-but-for-UDP
In a world with no legacy technology rusted in place, QUIC would be a new IP protocol, like TCP and UDP, but there's stuff rusted in place which can barely manage UDP and would not understand how there can possibly be a new protocol even though the entire network was designed to allow that, so that's why QUIC is spelled as UDP data.
mrb · 4h ago
Yeah, my take away is more "yet another example of a lousy firewall causing dumb issues".
This kind of stuff is exactly why TLS 1.3 encrypts everything now.
Sounds like a cat & mouse game. How long before we get servers that respond with different certificates based on who"s asking or other cues from the connection, in efforts to bypass the firewall restrictions.
For many years, long before TLS 1.3 the RFC explains how to do this properly, obviating the problem, but that's very expensive because you need to TLS proxy every single connection, you can impose oversight by - literally - adding an actual oversight layer, which your users can also see you did. The popular middlebox products say they're doing two things, they're cheaper (maybe you buy their $100k product instead of a $100M solution) and they're less "intrusive" (ie you needn't tell your users that you're spying on them)
In reality they're ineffective, which is why the RFC says not to do this - but they can either outright lie or hide this fact in an asterisked disclaimer somewhere, and most of their customers don't care whether it actually works, they want to tick a box.
QUIC is built on top of TLS 1.3 where client hello encryption is not mandatory, so this is not strictly true.
The in-progress Encrypted Client Hello (was Encrypted Server Name Indication hence esni in the name of the document) fixes that and you probably have software which uses it though the work to tie up all the loose ends up and publish a document might take some time yet.
Because the middlebox vendors are _so_ incompetent not only does TLS 1.3 need to work around their nonsense, thus proving that it was never useful security (an attacker could always have just done what TLS 1.3 does and it would have bypassed this worthless garbage) some of them screwed up badly enough that the anti-downgrade feature trips, to their credit Google refused to ship a permanent workaround for this, the workaround they shipped in Chrome sunset in about a year & required an explicit key setting, so basically "I acknowledge that I have defective middleboxes and must remove or upgrade them soon" by the local administrators.
But yes, it's noticeable that it was much easier to sell some engineers on "Thanks to TLS 1.3 now this stupid middlebox product won't be able to make your service slower" than say, "Thanks to TLS 1.3 now this stupid middlebox product won't report to the government if you read a Wikipedia article which contradicts its dogma".
There are valid use cases for TLS middleboxes. Anyone having to secure a networks outbound access to only essential services has run into the “I have to allow all of AWS/GCP/Azure/CloudFlare/…” for some critical tool to work.
Options are:
- Allow all out (nope, not secure)
- Allow all to the cloud providers IP range (still terrible)
- Filter on SNI but don’t inspect the payload (better than no security, and doesn’t require plaintext access).
- Full MITM TLS proxy (performance bottleneck, and now we have plaintext access to all your data, which we really don’t want and didn’t previously require to do the filtering).
- Try convince the third party service to run on a handful of static IPs that aren’t behind a global load balancer with access to the rest of the cloud providers customer domains. (Yeah right)
See: Hospitals. Payment Networks. IoT networks.
There are, and always have been, people who are sure that they "need" to do things which don't work because the correct solution would be inconvenient for them. Those people should instead suck it up and accept the inconvenience or, as is more likely, remember they don't "need" this after all considering how inconvenient it is.
It won't suddenly work just because that would be more convenient and it's very annoying technically that we have to keep having this conversation, it's not going to stop being true just because that would make your life easier.
And those clients should offer an option to downgrade the TLS connection to make traffic interceptable.
Here's hoping that the next TLS version will automatically encrypt everything. Firewall vendors will not like it, but it's the only way to truly hide everything from middleboxes.
That's why ECH needs DNS records, Bob publishes a key, "Here's the key for all six thousand blog.example sites we publish" and Alice can then encrypt either "Hi Bob, give me clown-porn.blog.example" or "Hi Bob, give me trans-rights.blog.example" and the middlebox can't read it.
We can't stop the middlebox from saying well, Bob publishes a trans rights blog so now all Bob's blogs are inaccessible just in case. Nor can we prevent Bob from deciding to publish separate keys, "these are for the clown porn and lynching videos, we'll use separate keys for the trans-rights stuff because the Government says that is naughty". But we can make that Bob's choice.
To do that, Alice needs Bob's key and either Alice has to magically know everybody's keys (which doesn't scale) or it is published somewhere for Alice to find, in this case DNS.
If we just say OK we'll encrypt with no particular recipient in mind, that doesn't prevent the scenario we care about, Alice encrypts the message, the middlebox decrypts it (No reason it can't, Alice can't pick Bob as the only recipient) and the middlebox gets to inspect Alice's destination.
[0] https://community.fortinet.com/t5/FortiGate/Technical-Tip-Ho...
In a world with no legacy technology rusted in place, QUIC would be a new IP protocol, like TCP and UDP, but there's stuff rusted in place which can barely manage UDP and would not understand how there can possibly be a new protocol even though the entire network was designed to allow that, so that's why QUIC is spelled as UDP data.