The Bluesky Dictionary

For what it's worth, there's 1.7bn posts on Bluesky according to this: https://bsky.jazco.dev/stats

The dictionary site has only checked 4,920,000 posts, which is 0.28% of all messages.

Hey! this is my site - it's not all that complex, i'm just using a sqlite db with two tables - one for stats, the other for all the words that's just word | count | first use | last use | post.

I... did not expect this to be so popular

You can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words...

Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie.

The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.

Probably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set.

250k words at a generous 100 bytes per word is only 25MB of memory...

I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.

Maybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.

You may need to allow scripts from the domain avibagla.com, it shows 0 when the scripts are blocked.

For me it took a minute to start loading data and switch from just showing 0.

Same... maybe you need a Bluesky account, which I don't have.

It's working fine for me on Firefox

> slowly moving towards

Depends what we accept as norm.

From what they say, it is a lot, but it's generally on the order of a few hundreds of connections total at the moment

Not an answer to your question, but I suspect most people don't -- my bot (a pi searcher bot, of course) just runs on Jetstream, which is pretty lightweight and heavily compressed.

(The website in question uses jetstream also.)

GeologySky will get to it soon enough.

I checked out the author's other projects and this is common issue. For example, he has a "lean checker" for bluesky that claims it is right-leaning simply because of all the people saying "That's right," "He was right," etc. None of the supposed right-leaning posts were actually conservative in nature. They just used to word right to mean correct.

I've found content for all of my future skeets.