Origin of SWE Bench

The idea for SWE Bench emerged when John Yang and Carlos Jimenez, both from prestigious universities, found themselves at Princeton during the summer, having completed their projects. They realized that utilizing GitHub pull requests could serve as a rich data source for benchmarking AI models in software engineering tasks. This led to brainstorming and collaboration, where their combined insights facilitated the development of SWE Bench, which benchmarks the performance of compound AI systems on real-world software engineering scenarios. The initiative represents a shift from basic coding challenges towards actual contributions to repositories, demonstrating the practical application of AI in software development.