I worked on this! Happy to answer any questions anyone has.
atlas_mugged · 1d ago
What are the limitations are there in terms of tasks this can handle? How does this compare with the other products out there? There are plenty of options...
sdspurrier · 1d ago
Depends on your set of tasks but we use Engine for the bottom ~50% of issues by complexity. We have a pretty good swe-bench score from a while back but it's got much better since!
We have also focused on workflow integrations so you can assign issues from Linear, Jira, Trello etc which makes it more useful for teams.
jackmpcollins · 1d ago
Seems to me that integrations will be the most important component of tools like this. As an engineer I get my context from video calls with customers and other engineers, slack messages, emails, docs online, using the product myself, etc. So an auto-engineer should do the same.
diminikolaou · 1d ago
This is cool. I can see the anti-monopoly of OpenAI argument, but apart from that is there a strong argument of being multi-LLM for a Codex-like agent?
sdspurrier · 1d ago
We often find that some models perform better on certain types of repo. For example Claude 3.5/7 is typically much better at frontends. That's why we let you switch up the model for each repo.
jackmpcollins · 1d ago
I've already merged my first Engine PR! Being able to review PRs like normal and it updates its work is very cool.
julvo · 1d ago
Looks great! What's your experience of using this for working on real world production code?
sdspurrier · 1d ago
60% of the time, it works every time
simvirdi · 1d ago
Looks cool - do you have any benchmarks? How do you compare to other products out there?
sudb · 1d ago
We last submitted a SWE-Bench verified result in November 2024 - at the time I believe we were in the top 5 entrants.
We expect Engine to be as good as the other code-writing agents out there at the moment - we understand almost everyone in the space to be using very similar base models and agent scaffolding.
RHSman2 · 1d ago
Demo’s this 6 months ago. Super excited to see how far it has come since!!!
ca508 · 1d ago
been following engine from afar for a while, super cool to see it on HN.
didn't see it had a free plan, will try it out.
We have also focused on workflow integrations so you can assign issues from Linear, Jira, Trello etc which makes it more useful for teams.
We expect Engine to be as good as the other code-writing agents out there at the moment - we understand almost everyone in the space to be using very similar base models and agent scaffolding.