Compares emdashes per page in the bill vs the same from the average bill sent to Congress
jeffgreco · 10h ago
Incredibly stupid how this has become a leading AI “tell”.
rsynnott · 1h ago
If it works, it works.
It's a crude example, but pattern analysis to figure out who wrote a thing is an old, old technique; people have been doing it with Shakespeare stuff for centuries, in particular.
alexjplant · 9h ago
Different models have different quirks. I use 4o at work and it has an annoying habit of taking things absurdly literally and using emojis and formatting too liberally instead of giving me a useful answer. Claude is much more subdued and helpful because it's conversational when you ask something open-ended, succinct when you ask it something data-driven, and almost always asks a follow-up question.
The sci-fi trope of being able to tell "AIs" apart from one another is absolutely coming true in real-time.
anakaine · 10h ago
Might be stupid, but it works, for now. Over indexed tokens is another (common inclusions). Some grammar constructions, too, where over descriptiveness is present - though that's easy to read and probably a bit harder to code for.
IAmGraydon · 9h ago
Why is it stupid? As long as people are too lazy to find/replace them, it seems to work.
bglazer · 9h ago
First, I’m almost certain that this article was also partially written by AI. See for example this paragraph obviously copy pasted from Deep Research
“Overall, a more nuanced view of AI in government is necessary to create realistic expectations and mitigate risks (Toll et al., 2020)”
What a unique and human thought for a personal blogpost. Also who the fuck is Toll et al, there’s no bibliography.
Second the authors used Gemini to count em dashes. I know parsing PDF’s is not trivial but this is absurd.
rooftopzen · 6h ago
First, see below for Toll et al 2020 and I used autocorrect for grammar. Sorry you were dismissive before looking it up, is more a reflection of your bias.
Second, I noted all caveats with an LLM counting that - I actually presumed I undercounted, but it had been noted that a simple ctrl-f found 3.8 per page rather than 9.8 per page (counting only single emdashes not double). The actual number doesn’t matter so much, since low bound is absurd difference from baseline bills I checked from earlier this year and 2024, where they do not exist outside of the table of contents.
4.x emdashes per page (low bound) is absurd, and the implication of this is the point you (respectfully) missed.
umbra07 · 4h ago
And how do you know it wasn't just edited by someone who loves em-dashes?
comparing it to the average doesn't matter too much. Better evidence would be proving that there has never been a bill with anywhere close to the number of em-dashes used in this bill.
inverted_flag · 11h ago
This administration is too stupid to write coherently without the aid of AI. It’s a way for them to dress up their low IQ ideas.
IAmGraydon · 9h ago
It’s probably also an attempt to inflate the bill to the point that it’s difficult to read. It came in at over 900 pages.
rooftopzen · 6h ago
Exactly the point of the post (900 pages falls in line with “flooding the zone” mentality), but we don’t have data on the intent.
elijahwright_ · 11h ago
this article doesn't make any sense. the bill has a lot of em dashes because that's how bills are expressed and it's a large bill. bills in Congress aren't written with em dashes because it can be confusing with the bill syntax and there's not a reason to do it that way
anakaine · 10h ago
The author compares it to the average bill going through congress, where you expect 0.1 emdash per page, where this bill has 10. So 100x the historic average.
elijahwright_ · 9h ago
well, for one, it's more more than 0.1 em dashes per page. the SHARE IT Act has 10 on each page[0]. I don't know how many the 2017 tax cut bill had but it's more than 1,000 and that was over 185 pages[1], and obviously that was before LLMs like ChatGPT. so I don't really know why this is the measure of AI or not, especially because bills have always had a lot of em dashes to start. if you're not analyzing the text of the bill then it's just not going to be accurate
Share IT is from 2024, but the 2017 tax cut bill is interesting (lots of emdashes there that deviate from the avg) - you’re correct on the additional need for text analysis in this case. Bills I’d found from earlier in 2024 that are publicly available do not have emdashes outside of the table of contents, which is built into the average - curious how/why they are used so much in this bill from 2017, now wondering how they got into any potential templates (or not), and adds the confound of how much this is AI or template (or requirements, or something else) Thx!
CamperBob2 · 10h ago
Add it to the ever-growing pile of "Things that Would Get an SF Author Laughed out of the Industry."
rooftopzen · 6h ago
Lol can you elaborate
CamperBob2 · 5h ago
Author: "A crowd of intellectually-challenged legislators is swept into Congress to serve an equally-dimwitted but far more devious POTUS who has hijacked their entire party. Their election campaigns were backed by endless free news coverage by media moguls and hundreds of millions in cash from an erratic Hugo Drax-like billionaire, all of whom demand a quick return from their investments. They immediately start passing laws they haven't read and don't actually understand, handed down from AI models that are still under development and highly error-prone. Hilarity ensues when..."
If I’m following correctly, this drama is like a Netflix series (and I agree, crazy stuff we couldn’t make up). If it’s only the Trump admin policies here, yeah it’s crazy bold (and which I state the ethical implications of).
Science fiction plot twist could involve anything completely crazy in the bill no one notices by having to use an LLM to read and that is open to interpretation enough to be only decided by the courts later.. I didn’t look for anything hidden and vague; but how would one really know lol.
It's a crude example, but pattern analysis to figure out who wrote a thing is an old, old technique; people have been doing it with Shakespeare stuff for centuries, in particular.
The sci-fi trope of being able to tell "AIs" apart from one another is absolutely coming true in real-time.
“Overall, a more nuanced view of AI in government is necessary to create realistic expectations and mitigate risks (Toll et al., 2020)”
What a unique and human thought for a personal blogpost. Also who the fuck is Toll et al, there’s no bibliography.
Second the authors used Gemini to count em dashes. I know parsing PDF’s is not trivial but this is absurd.
https://liu.diva-portal.org/smash/get/diva2:1591409/FULLTEXT...
Second, I noted all caveats with an LLM counting that - I actually presumed I undercounted, but it had been noted that a simple ctrl-f found 3.8 per page rather than 9.8 per page (counting only single emdashes not double). The actual number doesn’t matter so much, since low bound is absurd difference from baseline bills I checked from earlier this year and 2024, where they do not exist outside of the table of contents.
4.x emdashes per page (low bound) is absurd, and the implication of this is the point you (respectfully) missed.
comparing it to the average doesn't matter too much. Better evidence would be proving that there has never been a bill with anywhere close to the number of em-dashes used in this bill.
[0] https://www.congress.gov/118/plaws/publ187/PLAW-118publ187.p...
[1] https://www.congress.gov/115/statute/STATUTE-131/STATUTE-131...
Editor: "Don't call us, we'll call you^H^H^H security."
Science fiction plot twist could involve anything completely crazy in the bill no one notices by having to use an LLM to read and that is open to interpretation enough to be only decided by the courts later.. I didn’t look for anything hidden and vague; but how would one really know lol.