This project creates a "compact, machine-optimized format designed for efficient AI parsing rather than human readability" called "Structured Knowledge Format (SKF)".
It's not obvious to me that this is a good idea. LLMs are trained on human-readable text.
The author notes that reasoning LLMs do much better with these SKFs. Maybe that's a hint that human readable summaries would perform better? Just a guess.
iandanforth · 2h ago
I applaud this effort, however the "Does it work?" section answers the wrong question. Anyone can write a trivial doc compressor and show a graph saying "The compressed version is smaller!"
For this to "work" you need to have a metric that shows that AIs perform as well, or nearly as well, as with the uncompressed documentation on a wide range of tasks.
marv1nnnnn · 2h ago
I totally agreed with your critic. To be honest, it's even hard for myself to evaluate.
What I do is select several packages that current LLM failed to handle, which are in the sample folder, `crawl4ai`, `google-genai` and `svelte`. And try some tricky prompt to see if it works.
But even that evaluation is hard. LLM could hallucinate. I would say most time it works, but there are always few runs that failed to deliver.
I actually prepared a comparison, cursor vs cursor + internet vs cursor + context7 vs cursor + llm-min.txt. But I thought it was stochastic, so I didn't put it here. Will consider add to repo as well
timhigins · 12m ago
> LLM could hallucinate
The job of any context retrieval system is to retrieve the relevant info for the task so the LLM doesn't hallucinate. Maybe build a benchmark based on less-known external libraries with test cases that can check the output is correct (or with a mocking layer to know that the LLM-generated code calls roughly the correct functions).
ricardobeat · 56m ago
> But even that evaluation is hard. LLM could hallucinate. I would say most time it works, but there are always few runs that failed to deliver
You can use success rate % over N runs for a set of problems, which is something you can compare to other systems. A separate model does the evaluation. There are existing frameworks like DeepEval that facilitate this.
rybosome · 31m ago
To be honest with you, it being stochastic is exactly why you should post it.
Having data is how we learn and build intuition. If your experiments showed that modern LLMs were able to succeed more often when given the llm-min file, then that’s an interesting result even if all that was measured was “did the LLM do the task”.
Such a result would raise a lot of interesting questions and ideas, like about the possibility of SKF increasing the model’s ability to apply new information.
SparkyMcUnicorn · 2h ago
It's also missing the documentation part. Without additional context, method/type definitions with a short description will only go so far.
Cherry picking a tiny example, this wouldn't capture the fact that cloudflare durable objects can only have one alarm at a time and each set overwrites the old one. The model will happily architect something with a single object, expecting to be able to set a bunch of alarms on it. Maybe I'm wrong and this tool would document it correctly into a description. But this is just a small example.
For much of a framework or library, maybe this works. But I feel like (in order for this to be most effective) the proposed spec possibly needs an update to include little more context.
I hope this matures and works well. And there's nothing stopping me from filling in gaps with additional docs, so I'll be giving it a shot.
enjoylife · 1h ago
Was going to point this out too. One suggestion would be to try this on libraries having recent major semvar bumps. See if the compressed docs do better on the backwards incompatible changes.
rco8786 · 57m ago
Yea I was disappointed to see that they just punted (or opted not to show?) on benchmarks.
gk1 · 3h ago
92% reduction is amazing. I often write product marketing materials for devtool companies and load llms.txt into whatever AI I’m using to get accurate details and even example code snippets. But that instantly adds 60k+ tokens which, at least in Google AI Studio, annoyingly slows things down. I’ll be trying this.
Edit: After a longer look, this needs more polish. In addition to key question raised by someone else about quality, there are signs of rushed work here. For example the critical llm_min_guideline.md file, which tells the LLM how to interpret the compressed version, was lazily copy-pasted from an LLM response without even removing the LLM's commentary:
"You are absolutely right! My apologies. I was focused on refining the detail of each section and overlooked that key change in your pipeline: the Glossary (G) section is no longer part of the final file..."
Doesn't exactly instill confidence.
Really nice idea. I hope you keep going with this as it would be a very useful utility.
marv1nnnnn · 2h ago
Oof, you nailed it. Thanks for the sharp eyes on llm_min_guideline.md. That's a clear sign of me pushing this out too quickly to get feedback on the core concept, and I didn't give the supporting docs the attention they deserve. My bad.
Cleaning that up, and generally adding more polish, is a top priority.
Really appreciate you taking the time to look deeper and for the encouragement to keep going. It's very helpful!
thegeomaster · 2h ago
What is absolutely essential to present here, but is missing, is a rigorous evaluation of task completion effectiveness between an agent using this format vs the original format. It has to be done on a new library which is guaranteed not to be present in the training set.
As it stands, there is nothing demonstrating that this lossy compression doesn't destroy essential information that an LLM would need.
I also have a gut feeling that the average LLM will actually have more trouble with the dense format + the instructions to decode it than a huge human-readable file. Remember, LLMs are trained on internet content, which contains terabytes of textual technical documentation but 0 bytes of this ad-hoc format.
I am happy to be proven wrong on both points (LLMs are also very unpredictable!), but the burden of proof for an extravagant scheme like this lies solely on the author.
marv1nnnnn · 2h ago
Agree, actually this approach isn't even possible without the birth of reasoning LLM. In my test, reasoning LLM perform much better than non-reasoning LLM in interpreting the compressed file. Those LLMs are really good at understanding abstraction.
thegeomaster · 1h ago
My point still stands --- the reasoning tokens being consumed to interpret the abstracted llms.txt could have been used for solving the problem at hand.
Again, I'm not saying the solution doesn't work well (my intuition on LLMs has been wrong enough times), but it would be really helpful/assuring to see some hard data.
ianbicking · 3h ago
This mentions an SKF format for knowledge representation... but looking it up, I'm assuming it was invented just for this project?
Which is fine, but is there a description of the format distinct from this particular use? (I'm playing around with these same knowledge representation and compression ideas but for a different domain, so I'm curious about the ideas behind this format)
We've done some experimentation when using Claude Code and taken to just creating a "vendor" folder under our "docs" section of each of our repos and just pull down the readme file for every library we use. Then when I'm prompting Claude to figure something out, I'll remind it to go check "docs/vendor/awesomelib" or whatever and it does a fine job of checking the docs out before it starts formulating an answer.
This has done wonders for improving our results when working with TanStack Start or shadcn/ui or whatever.
I guess there's pieces of this that would be helpful to us, but there's too much setup work for me to mess with it right now, I don't feel like generating a Gemini api key, installing puppeteer, etc.
I already have all the docs pulled down, but reducing the number of tokens used for my LLM to pull up the doc files I'm referencing is interesting.
Is there a command line tool anyone has had luck with that just trims down a .md file but still leaves it in a state that the LLM can understand it?
TheTaytay · 18m ago
I’ve been creating a doc for each of my primary libs (using Claude Code of course). I like your vendor/readme idea. Do you find Claude going and reading more docs if it needs to?
eric-burel · 40m ago
You'd want to make this tool domain specific - language, type of docs, perhaps target a specific documentation framework/format that is common and standardized enough. I fail to buy a content-agnostic summarization method, though I recognize it could be better than nothing.
Also benchmark or it doesn't exist.
dmos62 · 2h ago
I would really like a benchmark showing that AIs can use this. Just the possibility that AI can understand the compressed format similarly well as the original excites me. How did you come up with the format?
marv1nnnnn · 2h ago
Honestly it's really funny. I have the initial idea, and then brainstorm with gemini 2.5 pro a lot, let it design the system. (And in prompt let it think like Jeff Dean and John Carmack )
But most version fails. Then somehow realized I can't let it design from scratch, I give gemini a structure I think is reasonable and efficient after seeing all those versions, let it polish based on that and it works much better.
ramoz · 1h ago
I've been using Gemini to shorten llms.txt and stick those in my repo.
But what I find best - cloning doc sites directly into my repos in a root context folder. I have bash scripts for managing those, and I instruct Claude how to use them. Context7 I dont like for the same reasons I don't hook up any MCP to Claude Code.
tough · 36m ago
I made a little bun app / script that takes a config.toml file with tasks and downloads the files using fetch or git clone
it aint much but its simple and i control it
obviyus · 3h ago
I recently upgraded a project from Remix to React Router 7 but unfortunately all AI assistants still try to "fix" my code with the Remix imports / conventions. I've had to add a bunch of custom rules but that doesn't seem enough.
This seems super useful though. I'll try it out with the RR7 docs and see how well it works.
jsmith99 · 37m ago
I'm also using RR7 and Gemini 2.5 Pro just refused to believe me that I could import Link from react-router. Just ignored my instructions and went down a rabbit hold in copilot agent mode, deeper and deeper, trying every possible package name (none of which were installed). I've now created a copilot instructions file into which I've copied most of the RR7 migration docs.
corytheboyd · 2h ago
FWIW sounds like a great use case for some rules files. I’ve only worked with Cursor and Roo but they both support it.
This of course only works for the “stop recommending X” part of your problem, but maybe something like the project here helps fill in with up-to-date documentation too?
Both Cursor and Roo also support URL context additions, which downloads the page and converts it to a machine readable format to include in context. I throw documentation links into my tasks with that all the time, which works out because I know that I am going to be sanity checking generated code against documentation anyway.
cluckindan · 2h ago
Maybe they have read that RR and Remix are now the same thing.
fcoury · 3h ago
Does it work with any technical doc? I see the CLI claims it's Python specific?
> $ llm-min --help
Usage: llm-min [OPTIONS]
Generates LLM context by scraping and summarizing documentation for Python libraries.
dmos62 · 2h ago
There's a sample for svelte in the repo.
fcoury · 1h ago
Guess I missed it. Thank you.
theturtle32 · 1h ago
I would love to see an example of a full transcript of the generation process for a small-ish library, including all the instructions given to the LLM, its reasoning steps, and its output, for each step in the generation flow.
This is one of those rare ideas that's obvious in hindsight. Instead of trying to stuff full docs into an AI, you give it only what matters. Clear structure, minimal noise, and no magic. Nice!
k__ · 3h ago
Pretty cool.
Seems like it could be a nice addition to Aider to supplement repo maps.
It's not obvious to me that this is a good idea. LLMs are trained on human-readable text.
The author notes that reasoning LLMs do much better with these SKFs. Maybe that's a hint that human readable summaries would perform better? Just a guess.
For this to "work" you need to have a metric that shows that AIs perform as well, or nearly as well, as with the uncompressed documentation on a wide range of tasks.
The job of any context retrieval system is to retrieve the relevant info for the task so the LLM doesn't hallucinate. Maybe build a benchmark based on less-known external libraries with test cases that can check the output is correct (or with a mocking layer to know that the LLM-generated code calls roughly the correct functions).
You can use success rate % over N runs for a set of problems, which is something you can compare to other systems. A separate model does the evaluation. There are existing frameworks like DeepEval that facilitate this.
Having data is how we learn and build intuition. If your experiments showed that modern LLMs were able to succeed more often when given the llm-min file, then that’s an interesting result even if all that was measured was “did the LLM do the task”.
Such a result would raise a lot of interesting questions and ideas, like about the possibility of SKF increasing the model’s ability to apply new information.
Cherry picking a tiny example, this wouldn't capture the fact that cloudflare durable objects can only have one alarm at a time and each set overwrites the old one. The model will happily architect something with a single object, expecting to be able to set a bunch of alarms on it. Maybe I'm wrong and this tool would document it correctly into a description. But this is just a small example.
For much of a framework or library, maybe this works. But I feel like (in order for this to be most effective) the proposed spec possibly needs an update to include little more context.
I hope this matures and works well. And there's nothing stopping me from filling in gaps with additional docs, so I'll be giving it a shot.
Edit: After a longer look, this needs more polish. In addition to key question raised by someone else about quality, there are signs of rushed work here. For example the critical llm_min_guideline.md file, which tells the LLM how to interpret the compressed version, was lazily copy-pasted from an LLM response without even removing the LLM's commentary:
"You are absolutely right! My apologies. I was focused on refining the detail of each section and overlooked that key change in your pipeline: the Glossary (G) section is no longer part of the final file..."
Doesn't exactly instill confidence.
Really nice idea. I hope you keep going with this as it would be a very useful utility.
As it stands, there is nothing demonstrating that this lossy compression doesn't destroy essential information that an LLM would need.
I also have a gut feeling that the average LLM will actually have more trouble with the dense format + the instructions to decode it than a huge human-readable file. Remember, LLMs are trained on internet content, which contains terabytes of textual technical documentation but 0 bytes of this ad-hoc format.
I am happy to be proven wrong on both points (LLMs are also very unpredictable!), but the burden of proof for an extravagant scheme like this lies solely on the author.
Again, I'm not saying the solution doesn't work well (my intuition on LLMs has been wrong enough times), but it would be really helpful/assuring to see some hard data.
Which is fine, but is there a description of the format distinct from this particular use? (I'm playing around with these same knowledge representation and compression ideas but for a different domain, so I'm curious about the ideas behind this format)
This has done wonders for improving our results when working with TanStack Start or shadcn/ui or whatever.
I guess there's pieces of this that would be helpful to us, but there's too much setup work for me to mess with it right now, I don't feel like generating a Gemini api key, installing puppeteer, etc.
I already have all the docs pulled down, but reducing the number of tokens used for my LLM to pull up the doc files I'm referencing is interesting.
Is there a command line tool anyone has had luck with that just trims down a .md file but still leaves it in a state that the LLM can understand it?
But what I find best - cloning doc sites directly into my repos in a root context folder. I have bash scripts for managing those, and I instruct Claude how to use them. Context7 I dont like for the same reasons I don't hook up any MCP to Claude Code.
it aint much but its simple and i control it
This seems super useful though. I'll try it out with the RR7 docs and see how well it works.
This of course only works for the “stop recommending X” part of your problem, but maybe something like the project here helps fill in with up-to-date documentation too?
Both Cursor and Roo also support URL context additions, which downloads the page and converts it to a machine readable format to include in context. I throw documentation links into my tasks with that all the time, which works out because I know that I am going to be sanity checking generated code against documentation anyway.
Edit: not quite.
Seems like it could be a nice addition to Aider to supplement repo maps.