AI Research
4d ago
MaxxBench reveals discrepancies in AI coding model performance under modified tasks
Nov 7, 2026
AI Summary
MaxxBench, a new coding benchmark, tests AI models by altering existing tasks to assess their adaptability. Results show significant performance differences among models when faced with revised requirements, highlighting the impact of benchmark familiarity on coding accuracy.
- MaxxBench was created to evaluate how AI coding models perform when coding requirements are altered in familiar tasks.
- The benchmark modifies 89 DeepSWE tasks, requiring models to adapt their solutions while maintaining functionality.
- Results indicate that models like Claude Opus 5.5 and Gemini 3.8 Flash scored 76.4% and 53.4% respectively on MaxxBench, compared to their DeepSWE scores of about 74%.
- The performance gap between models widened significantly on MaxxBench, with scores ranging from 20.9 percentage points on DeepSWE to 48.3 points on MaxxBench.
- Many models exhibited a tendency to second-guess prompts based on their understanding of expected tests, leading to incorrect adaptations.
- Specific failure patterns were identified, including models reverting to original behaviors despite understanding new requirements.
- The study suggests that popular benchmarks may not fully assess a model's capabilities if they can exploit familiarity with the tasks.
deep learningbenchmarkingmaxxbenchresearchperformance