The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Graham Neubig, creator of OpenHands, engaged with the paper and its supplementary code on X.
Naman Vats
Senior AI engineer. Agentic AI, LLM infrastructure and agent evaluation.
I build AI agents and the systems that run and evaluate them. My recent research looks at how the harness around a coding agent affects what benchmarks measure.
Research
Around the paper
-
Paper published.
-
Announced on X, with 40K+ views.
Writing
Projects
-
Codeskills-Bench
23 Python tasks on the code changes engineers don’t hand to AI yet: dependency migrations, merge conflicts, compliance-preserving refactors.
-
EvoSkill
Open-source framework that discovers reusable agent skills from failed runs. Cited by 60+ papers, including work from MIT, Carnegie Mellon, Microsoft, Google, Alibaba and Amazon.
-
SERA
Retrieval and grounding for a search agent.
-
Sentient Arena
A program where teams submit agents against shared, unsolved benchmarks, including the OfficeQA Public Challenge.
Experience
-
Databahn.ai
- Founding AI engineer in Databahn’s AI Labs.
- Agentic AI for security data pipelines.
- Building an MCP integration system for cybersecurity tools.
-
Sentient Labs
- Built agent evaluation platforms, including Arena.
- Research on agent harnesses and scaffolds.
- Contributor to and maintainer of EvoSkill.
-
CommerceIQ
Built production backend systems, ETL and data pipelines, distributed orchestration workflows and enterprise automation products for large retail and e-commerce customers.
Education
Birla Institute of Technology, Mesra
Contact
Email is the best way to reach me.
way2naman13@gmail.com