Does anyone use Skyvern to build their websites? I’m wondering how I might benefit from using an agentic browser workflow instead of a playwright MCP server for building a web UI?
neveroddoreven · 15h ago
I had no idea WebVoyager only spanned 15 websites lol... the 452 figure you have still seems a little low though - do you have plans to expand it? It seems like you'd want as many sites as possible to improve the real-world accuracy of agents due to the long tail nature of website traffic
suchintan · 14h ago
We definitely plan to expand it. I want to get to ~10,000 for a reasonable benchmark.
15 blew my mind -- it's too easy to overfit that dataset
vasusen · 8h ago
Thank you so much for creating this folks! A browser navigation agent is key part of our AI QA setup at Donobu (https://donobu.com/). We found the WebVoyager benchmarks severely lacking for complex e2e test cases like logged-in dashboards, onboarding forms, etc.
While the extraction/2fa flows aren't super relevant to us, this saves us time from building our own set of benchmarks. Really appreciate it and hope we can contribute to make this a really large set.
suchintan · 5h ago
That would be amazing!!
pants2 · 7h ago
Great work! Big fan of Skyvern.
Looking forward to the benchmarks on Claude 4 (and o3 CUA when that's released)
gitmagic · 9h ago
Would love to see how Nelly [0] performs on this benchmark.
15 blew my mind -- it's too easy to overfit that dataset
While the extraction/2fa flows aren't super relevant to us, this saves us time from building our own set of benchmarks. Really appreciate it and hope we can contribute to make this a really large set.
Looking forward to the benchmarks on Claude 4 (and o3 CUA when that's released)
[0] https://nelly.is