We’ve been working on this problem off and on for over a year now. Many models bake knowledge of particular tools/libraries/patterns into their weights very well and others quite poorly. In my experience Claude is quite good at integrating the dog.ceo API and noticeably ignorant when it comes to Postgres features, and it knows gcloud commands enough to very confidently and consistently hallucinate arguments.
We’ve baked a solution to this into our product, so if anybody is working on an API/SDK/etc feel free to contact me if your users are running into problems using LLMs to integrate them.
One thing we’ve noticed is that subtle changes to library/api integration prompts’ context can be surprisingly impactful. LLMs do very well with example commands and explicit instructions to consider X, Y, and Z. If you just dump an API reference and information that implicitly suggests that X, Y, and Z might be beneficial, they won’t reliably make the logical leaps you want them to unless you let them iterate or “think” (spend more tokens) more. But you can’t as easily provide an example for everything, and the ones you do will bias the models towards them, so you may need a bit of both.
EGreg · 13m ago
I made a provisional patent this year, about how exactly I would solve this problem. Imagine hiring a "team of developers" who can learn your library and iterate 24/7, improving things, doing support, even letting the pointy-haired boss turn his ideas into reality in a forked sandbox on the weekend.
Normally I don't like software patents, but in the case of AI, I have made an exception. I have also rethought how I am going to do open source vs closed source in my AI business.
We’re trying to build a similar kind of experience but for both “sides” of the problem: software provider and software users/integrators.
rikroots · 51m ago
I've done a lot of work recently to make my library more "LLM Friendly", but I'm not willing at this time to sign up to a service which I don't know I'd ever use again just to run a test on your behalf. If you want to run the test on my library then its GitHub can be found here: https://github.com/KaliedaRik/Scrawl-canvas
richardblythman · 2h ago
If coding agents are the new entry point to your library, how sure are you that they’re using it well?
I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries:
• Takes your library’s docs
• Automatically extracts usage examples
• Tasks AI agents (like Claude Code) with generating those examples from scratch
• Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
bdhcuidbebe · 46m ago
> If coding agents are the new entry point to your library, how sure are you that they’re using it well?
> I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Why should they even bother to answer such a loaded and hypothetical question?
justonceokay · 57m ago
If making dev tooling is selling shovels to the miners, then this is like selling sheet metal to the shovel makers.
weitendorf · 16m ago
Let’s meet and see if it might make sense for us to team up. We’re working on this from the agent/library-specific-task side, and we might be better than chatgpt at marketing your product :)
dotancohen · 1h ago
Note that this comment is not hijacking. The author of this comment is also the author of the post.
add-sub-mul-div · 26m ago
That's the more likely assumption. Accounts with only self-promotion spam activity have become more of a rule here than an exception.
spankalee · 43m ago
Why do we need to log in?
metadat · 58m ago
The skip-to-the-end answer: Context7 MCP is so good it seems like magic, even to many well-informed, highly capable hackers. Simply wildly good for libraries and SDKs. All it takes to start using it is to add the MCP provider to your agent config and save your arms, "Use Context7 for this".
I'm confused a bit by this. For instance, Gemini was struggling to write proper Java code for using Firebase Admin SDK. It would write Java code using methods that only exist in the JavaScript SDK. And when I would correct it, it would give other options that also were only in the JavaScript SDK or were invalid.
We’ve baked a solution to this into our product, so if anybody is working on an API/SDK/etc feel free to contact me if your users are running into problems using LLMs to integrate them.
One thing we’ve noticed is that subtle changes to library/api integration prompts’ context can be surprisingly impactful. LLMs do very well with example commands and explicit instructions to consider X, Y, and Z. If you just dump an API reference and information that implicitly suggests that X, Y, and Z might be beneficial, they won’t reliably make the logical leaps you want them to unless you let them iterate or “think” (spend more tokens) more. But you can’t as easily provide an example for everything, and the ones you do will bias the models towards them, so you may need a bit of both.
Normally I don't like software patents, but in the case of AI, I have made an exception. I have also rethought how I am going to do open source vs closed source in my AI business.
(If anyone wants to work with me on this, hit me up, email is in my profile) https://grokers.ai/patent.pdf
We’re trying to build a similar kind of experience but for both “sides” of the problem: software provider and software users/integrators.
I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries: • Takes your library’s docs • Automatically extracts usage examples • Tasks AI agents (like Claude Code) with generating those examples from scratch • Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
> I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Why should they even bother to answer such a loaded and hypothetical question?
https://context7.com/
So I thought this is where context7 would be useful, but I'm confused what I'm looking at in the detail page: https://context7.com/firebase/firebase-admin-java
I was expecting some sort of dump of all the admin methods, but it gives a single example of one library function and info on how to build javadoc.