Show HN: LLMs suck at writing integration code… for now

13 sfaist 10 7/24/2025, 3:28:07 PM github.com ↗
Hi HN! Stefan here from superglue and today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs.

We gave LLMs API documentation and asked them to write code that makes actual API calls. Things like "create a Stripe customer" or "send a Slack message". We're not testing if they can use SDKs; we're testing if they can write raw HTTP requests (with proper auth, headers, body formatting) that actually work when executed against real API endpoints and can extract relevant information from that response.

tl:dr: LLMs suck at writing code to use APIs.

We ran 630 integration tests across 21 common APIs (Stripe, Slack, GitHub, etc.) using 6 different LLMs. Here are our key findings:

- Best general LLM: 68% success rate. That's 1 in 3 API calls failing, which most would agree isn’t viable in production

- Our integration layer scored a 91% success rate, showing us that just throwing bigger/better LLMs at the problem won't solve it.

- Only 6 out of 21 APIs worked 100% of the time, every other API had failures.

- Anthropic’s models are significantly better at building API integrations than other providers.

Here is the results chart: https://superglue.ai/files/performance.png

What made LLMs fail:

- Lack of context (LLMs are just not great at understanding what API endpoints exist and what they do, even if you give them documentation which we did)

- Multi-step workflows (chaining API calls)

- Complex API design: APIs like Square, PostHog, Asana (Forcing project selection among other things trips llms over)

We've open-sourced the benchmark so you can test any API and see where it ranks: https://github.com/superglue-ai/superglue/tree/main/packages...

Check out the repo, consider giving it a star, or see the full ranking at https://superglue.ai/api-ranking/.

If you're building agents that need reliable API access, we'd love to hear your approach, or you can try our integration layer at superglue.ai.

Next up: benchmarking MCP.

Comments (10)

adinagoerres · 3h ago
Hey HN, I'm Adina, Stefan's co-founder at superglue. When we started working on LLM-powered integrations about a year ago, the models were barely good enough to handle simple mappings. We started benchmarking our performance as an internal evals project and thought it would be fun to open source it, to create more transparency around LLM performance. Our goal here is to understand how we can make agents production-ready and improve reliability across the board.
hoerzu · 3h ago
Love the benchmarks. Is better to use single LLM for performance or would always advise to add a self reflection step
adinagoerres · 2h ago
self-reflection is very important for both humans and LLMs, indeed
ThomasMin · 1h ago
Awesome work Stefan, this is super insightful! Really appreciate the transparency and open-sourcing the benchmark. The 68% success rate is a wake-up call for anyone building with LLMs. Your 91% integration layer result is impressive, shows tooling matters. Excited to see what you uncover next with MCP!
iamflimflam1 · 2h ago
I would expect most developers to fail at this challenge. Here’s the doc - you’ve got one chance to get the API to do this.

I can’t tell from the description if the LLMs are allowed to try and then correct based on any errors received.

Though it would be surprising if that helped. Most APIs don’t tell you what you’ve done wrong…

sfaist · 2h ago
We would've assumed that the llms are much better at writing working code since it's not random APIs but rather established API patterns which they should be able to one-shot (e.g. Stripe). Bad error messages are a problem indeed. We will release another one with retries very soon.
ForzaAaRon · 2h ago
Fascinating read. Interesting how opus performs worse compared to sonnet
sfaist · 2h ago
Quite interesting actually. not sure why, I assume it just overthinks. What suprised me even more is how bad o4-mini performed, after taking up hours of evaluation time and more credits than all other llms combined. More thinking != better (integration) coding performance
hoerzu · 3h ago
What's the hello world of super glue?
danmeier · 2h ago
very interesting! curious to see the benchmarks for MCP!