Launch HN: Design Arena (YC S25) – Head-to-head AI benchmark for aesthetics

34 grace77 23 8/12/2025, 4:10:03 PM
Hi HN, I’m Grace from Design Arena (https://www.designarena.ai/) - we’re building a crowdsourced benchmark for AI-generated visuals (websites, images, video, and more). We put AI models and builder tools in head-to-head comparisons that get voted on by real users from around the world. Think “Hot or Not” for the AI era :)

(Btw, when we say real users we mean real users, so you may get a captcha on the site. Sorry, but we have to use every bot protection available! We only want human ratings, for obvious reasons.)

Here’s a demo video: https://www.youtube.com/watch?v=vPyEQnuVgeI

We didn’t set out to build this - we were actually working on an AI game engine. But we found that models sucked at look-and-feel. Even when the output code was usually functional, most visual aspects lacked the soul that makes great graphics feel alive.

So we built a this-or-that game, just for ourselves, to figure out which generated outputs had the best graphics. To our surprise, that turned out to be more exciting than the original idea—it turns out this is a widespread problem! We did a Show HN a month ago (https://news.ycombinator.com/item?id=44542578) and that was partly what convinced us to make this benchmark thing our actual product.

State-of-the-art models might be winning IMO gold, but they are still putting white text on a white background. There needs to be some measurement of what’s good and what isn’t (yes, there is such a thing as good design!), and it sure isn’t going to come from LLMs.

We come from engineering backgrounds (Apple and Nvidia) with a love for design; we know when we like or dislike something, even when we can’t say why. This-or-that / hot-or-not games are made for domains like this: Design Arena’s goal is to make everything stupidly simple so humans can just do the easy part: like-vs.-dislike. Which also turns out to be the valuable part, because what’s easiest for humans is actually the part that the AIs can’t currently do.

Since our Show HN, we’ve extended our initial set of ~25 LLM models to 54 LLM models, 12 image models, 4 video models, 22 audio models, and 22 vibe-coding tools (like Lovable, Bolt, v0, Firebase Studio, and more). In this last category, we’ve been surprised to find that agentic tools that were not specifically marketed as vibe-coders like Devin performed exceedingly well in the builder category, outperforming dedicated builder tools like Lovable, v0, and Bolt.

Our users are mostly devs who want to spin up a frontend, or designers who want to spin up design variants faster. In both cases, Design Arena provides a quick way to find out which options are better than others. Dev-or-designer needs to make the final calls, because there’s no substitute for good judgment. But this type of formatting can really help.

We plan to make money by offering version testing as a service to companies that need to quantify improvements in their product between builds.

This is the first time we’ve ever worked on something like this! We’d love to learn from you all and look forward to your feedback.

Comments (23)

henriquegodoy · 1h ago
This is actually really needed, current ai design tools are so predictable and formulaic, like every output feels like the same purple gradients with rounded corners and that one specific sans serif font that every model seems obsessed with, it's gotten to the point where you can spot ai-generated designs from a mile away because they all have this weird sterile aesthetic that screams "made by a model"
grace77 · 1h ago
Exactly - we think the ai design tools are in the equivalent of the 'uncanny valley' territory that a lot of the diffusion models were stuck in just 1-2 months ago; most average diffusion models are still in this local optimum, but the best of the best seem to have escaped it.
Michelangelo11 · 1h ago
This is interesting but, speaking frankly, I see many seemingly insurmountable issues. Here are some:

- Contests will often be won not by the entry that best adhered to the prompt, but the best-looking one. This happened in the contest "Input Prompt Build a brutalist website to a typeface maker," which I got as a recent example. The winning entry had megawatt-bright magenta and yellow, which shouldn't appear anywhere near brutalism, and in other design aspects had almost no connection to brutalism either -- but it was the most attractive of the bunch.

- The approach only gets you to a local maximum. Current LLMs aren't very good designers, as you say, so contests will involve picking between mostly middling entries. You'd want a design that's, say, a 9 or a 10 on a 10-point scale -- but some 95% of the entry distribution will probably be between 5.5 and 7.5 or so, and that's what users will get to pick from.

j_da · 1h ago
All great points. A limitation with human feedback is that once you start asking for more than binary preferences (e.g. multiple rankings or written feedback), the quality of the feedback does decrease. For instance, many times humans can give a quick answer on preference, but when asked "why" they prefer one thing over the other, they might not be able to full explain it in language. This in general is very much an open area of research on collecting and incorporating the most optimal types of feedback.

I definitely agree with your second point. One idea we're experimenting with is adding a human baseline, in which the models are benchmarked against human generated designs as well.

grace77 · 1h ago
yes! to the second point, someone in our show HN proposed encouraging human designers to compete in submissions as well - we tried implementing this and found that, at least right now, LLMs are still so bad at design that asking a human to beat them is trivial - our plan right now is to focus more on this once it becomes more of challenges and therefore hopefully more interesting/entertaining
refrigerator · 2h ago
Great concept — definitely needed and will hopefully push the labs to improve design abilities of models!
j_da · 1h ago
Yes, exactly. We want to be a forcing function for better design models and agents.
transformi · 2h ago
Cool - do you train model that will be the proxy from the votes of persons?
grace77 · 2h ago
we're not training models or proxying human votes with models
ryhanshannon · 2h ago
Is this an area that is not yet covered by other user rating benchmark sites like LLMarena?
grace77 · 2h ago
yes! LMArena recently started pushing "webdev" arena, but there was no explicit emphasis on design or aesthetics, just web-based content
doctorpangloss · 1h ago
Can you write what you imagine is a good “game dev” prompt?
grace77 · 46m ago
We keep our system prompts across the board as bare bones as possible: https://www.designarena.ai/system-prompts

As for good game dev prompts, here's one from a user that made a pretty fun game: Make asteroids with 2 computers playing against each other on one screen. There should be asteroids flying and 2 ships being controlled by 2 computers. Pay attention to thoroughly implementing the logic to make the ships avoid asteroids at all costs. Absolutely no user input should be necessary, no click to start, no click to restart. The game starts automatically on load and automatically restarts when either computer is dead. The ships should survive as long as possible. The ships should fly around, avoid asteroids as a priority, but also shoot asteroids and each other. Make ships and asteroids positions random each time. Asteroids should split when shot. The goal is to create a robust algorithm for ships so they can survive as long as possible. The game should be playable at 500x500 screen resolution.

doctorpangloss · 10m ago
Do you see why this is an interesting question? It's not meant to criticize.

Here's another one. Game out for me, what would happen if I developed an LLM benchmark that was scientifically robust, but, it showed that some small open-weights model outperformed Anthropic, Google or OpenAI's best offerings? Like from the POV of a startup - let's say that happened.

andrewstuart · 2h ago
AI is terrible at making nice looking design layout with font selections.

Sure it can make great looking images but nothing can make a nice looking poster or basic page layout.

I’m waiting for someone to solve this. I’m not even sure it takes AI it might just be programmatic.

rovmut · 1h ago
You've perfectly articulated the gap in the market. The solution isn't just a better image generator. I built a tool called LayoutCraft to solve this exact problem. It focuses entirely on creating a great layout with good font choices automatically. It uses AI to understand the request, but then applies a structured, programmatic 'blueprint' to build the layout. This is how it handles fonts and spacing properly, resulting in a clean design, not a chaotic image.
grace77 · 2h ago
yes - we're trying to figure out why that is
ChrisArchitect · 1h ago
So this went from a Show HN: to a Launch HN in a month?

(Show HN: https://news.ycombinator.com/item?id=44542578)

grace77 · 43m ago
yes - the changes since Show HN have been builders (https://www.designarena.ai/builder), audio (https://www.designarena.ai/audio), video (https://www.designarena.ai/diffusion), and compare (https://www.designarena.ai/studio). We also had a feed at some point, but ripped that out because it looked messy
nextworddev · 46m ago
Vibe launch
KaoruAoiShiho · 2h ago
Curious if you guys got into YC for this idea or something else?
j_da · 2h ago
We started out building a platform to one-shot games (single-player and multi-player), but realized that the model you used under the hood really made a difference in functionality and graphics. We started out building the benchmark as an internal tool for ourselves to see which model was the best, but found that benchmarking models on visual "taste" was something that people were generally interested in currently.
neonate · 2h ago
Post says they were making an AI game engine, so that's probably what they got in with.