GitHub - facebookresearch/ProgramBench: Can Language Models Rebuild Programs From Scratch? (original) (raw)

Can Language Models Rebuild Programs From Scratch?

Given only a compiled binary and its documentation, AI agents must architect and implement a complete codebase that reproduces the original program's behavior.

Links

• Website
• Paper
• HuggingFace
• Leaderboard
• Usage Guide
• mini-swe-agent baseline

Quickstart

We recommend uv for managing Python environments.

Run without installing

uvx programbench --help

Or install into a project

uv pip install programbench

Or with pip

pip install programbench

For development:

git clone https://github.com/facebookresearch/programbench.git cd programbench uv sync # installs editable + dev dependencies

To run our baseline:

uvx --from mini-swe-agent mini-extra programbench --help

or

pip install mini-swe-agent && mini-extra programbench --help

See the mini-swe-agent baseline docs for details.

Citation

If our work was useful for you, please cite it:

@misc{yang2026programbenchlanguagemodelsrebuild, title={ProgramBench: Can Language Models Rebuild Programs From Scratch?}, author={John Yang and Kilian Lieret and Jeffrey Ma and Parth Thakkar and Dmitrii Pedchenko and Sten Sootla and Emily McMilin and Pengcheng Yin and Rui Hou and Gabriel Synnaeve and Diyi Yang and Ofir Press}, year={2026}, eprint={2605.03546}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2605.03546}, }

License

ProgramBench is licensed under the terms of the license found in LICENSE.