Hi! We just released SWE-rebench – an extended and improved version of our previous dataset with GitHub issue-solving tasks.
One common limitation in such datasets is that they usually don’t have many tasks, and they come from only a small number of repositories. For example, in the original SWE-bench there are 2,000+ tasks from just 18 repos. This mostly happens because researchers install each project manually and then collect the tasks.
We automated and scaled this process, so we were able to collect 21,000+ tasks from over 3,400 repositories.
One common limitation in such datasets is that they usually don’t have many tasks, and they come from only a small number of repositories. For example, in the original SWE-bench there are 2,000+ tasks from just 18 repos. This mostly happens because researchers install each project manually and then collect the tasks.
We automated and scaled this process, so we were able to collect 21,000+ tasks from over 3,400 repositories.