Top model scores may be skewed by Git history leaks in SWE-bench (github.com)

For over half our lives, we've been developers and we've done tons of user-facing projects like a coding competition I built called CerealCodes or freelancing projects on Upwork. The biggest problem we faced was that we shipped bugs in edge cases we didn't test, and the process of testing was annoying to do everytime we shipped a new feature. We tried automated testing tools, but those were flaky and couldn't adapt to feature changes. They also were really annoying to set up.

Our solution is to use browser agents to help you find bugs in your web app by clicking through your product like users would. You'd enter in your URL, describe what a user would do, and Ghostship would go through and try finding bugs by going through the user journey and extrapolating edge cases by visually seeing where else to click as it goes through each step in the user journey. We then show session replays of our agents going through your web app and list out all the steps it took.

We're able to find edge cases with almost no prompting. All you need to do is enter in one URL and one user journey (if you have login credentials on your web app, enter in some test credentials).

One bug we were able to find with Ghostship was on the YC application page. Apparently you could add your education dates in reverse chronological order (April 2022 to January 2021, which makes no sense).

Another bug we were able to find was a crypto smart contract CRM dashboard we vibe coded where we found a bug involving data corruption when you tried editing a draft contract multiple times.

You can sign up here: https://playground.tryghostship.dev/ for a limited number of credits. We'd love to hear from the HN community, whether you're building a web app for fun or a developer shipping a cool user-facing product to customers. We'd love to see what bugs we can find in your web app with Ghostship!

p.s. If you want Ghostship directly in your CI/CD pipeline and run after every PR, book a demo with us.

Comments (12)

tarasyarema · 12m ago

Completely agree with some of the comments, actually the hardest part is to execute these tests in a fast an reliable manner paralelizing the most of it, for your regression tests.

As we all know, in large orgs the form issue has been solved by using trusted components with unit tests. What worries me is how somebody were actually to provide valuable bugs, those that keep you up at night trying to fix them.

I worked with exploratory testing companies that would report hundreds of bug that nobody ever cared about. Does YC really care about the bug that they found, or have a business impact?

(Actually coming from building a company in this area now https://desplega.ai, we have an demo video in the landing)

asdev · 2h ago

I've seen other YC companies do this exact thing and pivot. What's your differentiation?

minajevs · 33m ago

UI tests, and any automated tests in general, must be reproducible and deterministic to be useful. Do you position this product as an automated test tool, or useful “chaos monkey”?

Also a nitpick, but there are profanities on your landing page and it kinda puts me off.

rgbrgb · 2h ago

This is awesome and an idea i've thought a lot about. In my experience, people either have flaky e2e tests or no e2e tests. How do you solve cost / margin? Is this an order of magnitude more expensive than my existing CI running playwright tests?

lovich · 44m ago

Man this is a tough crowd.

The product looks interesting. My only question would be what does a common successful description of a user journey look like?

Is it pages and pages of prompting to get the agents to understand or do you see success with less verbose descriptions?

m90 · 1h ago

Typo on your landing page: "GitHub PR Intergration" -> "GitHub PR Integration"

handfuloflight · 1h ago

Price?

Mountain_Skies · 13m ago

I'm in no way affiliated with Ghostship so I'm not trying to talk on their behalf but my experience in this space is that token credits are sold by negotiated private contracts. Similar products and companies typically don't advertise standard fees for token credits, so there's very little price transparency. Maybe Ghostship will be different, which would be refreshing.

just_steve_h · 51m ago

You know “Ghostship” was the name of an incredible East Bay artists’ community that was destroyed in a fire which killed multiple people. Is this really the name you’re going with? It’s not ancient history, either - less than 10 years.

lovich · 46m ago

Never heard of them. I have heard of the shitty 2002 film by that name.

I don’t know why someone would be expected to be aware of a niche, local, artist community or their history

fakedang · 38m ago

I haven't lived in the Bay Area ever, although I had plans to move there at the time (plans which were sidelined because I didn't get the H1B then), but the fire was pretty big news even halfway across the world.

https://en.m.wikipedia.org/wiki/Ghost_Ship_warehouse_fire

It wasn't the community that sparked attention, but the shoddy living conditions and that people had to resort to living in a warehouse in the Bay Area because of unaffordable housing (and which indirectly led to the conditions for the fire to happen).

But again I wouldn't expect recent YC people who just moved to the Bay Area to know about it.

lovich · 30m ago

I’ve never lived in the area, didn’t even know what city he was referencing with “East Bay” and I have been on hacker news since before this fire according to Wikipedia.

I have never heard of this event.

Also according to Wikipedia 36 people died. 31 people have died in Nepal due to their political shenanigans the past few days and I highly doubt anyone not involved in the region is going to remember in 10 years.

It seems really weird to me to be calling out random people for naming collisions with incredibly local, niche, news events from a decade ago