New Benchmark Real-SWE Evaluates AI Models on Real-World Enterprise Codebases

Real-SWE evaluates frontier AI models on tasks derived from licensed private production codebases of real companies.

The benchmark aims to replicate the complexities faced by software engineers in actual work environments, contrasting with expert-generated or synthetic tasks.

Tasks are designed to reflect real engineering challenges, requiring models to understand existing business logic and coding patterns while making changes within operational constraints.

The benchmark includes 11 files, with a typical instruction averaging 1,742 characters. It reports a failure rate of 71.4% for rollouts under 10 minutes and 73.4% for longer rollouts.

Each task is inspired by or directly taken from real-world codebases, and the evaluation process involves running agents in isolated sandboxes with verifiers based on existing test suites.