InterCode (original) (raw)

News

10.19.2023 InterCode (v1.0.2) released, new IC-SWE!

10.12.2023 Lemur sets new highs on IC-[Bash, CTF, SQL]!

09.22.2023 InterCode accepted to 2023 NeurIPS Datasets & Benchmark track!

08.15.2023 InterCode (v1.0.1) released, new IC-CTF, IC-Python!

07.01.2023 InterCode now available on PyPI

06.27.2023 InterCode (v1.0.0) available on GitHub

What is InterCode?

InterCode is a benchmark for evaluating language models on the interactive coding task. Given a natural language request, an agent is asked to interact with a software system (e.g., database, terminal) with code to resolve the issue.

InterCode currently features 5 different code environments: IC-Bash, IC-CTF, IC-Python, IC-SQL, IC-SWE. You can learn more about each on the Environments page!

Question & Contributing

If you have any questions or would like to contribute to InterCode, you can post an issue on the InterCode GitHub issues page. Also, please feel free to contactJohn Yang directly.

Acknowledgements

We would like to thank thePrinceton NLP group for their support towards building InterCode. In particularly, we'd like to thankCarlos E. Jimenez and Yuhan Liu for testing InterCode and providing valuable feedback. In addition, our thanks toProf. Pranav Rajpurkar for giving us permission to use the SQuAD template for this website.

Citing

If you found InterCode helpful for your work, please cite us!

@misc{yang2023intercode, title={InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback}, author={John Yang and Akshara Prabhakar and Karthik Narasimhan and Shunyu Yao}, year={2023}, eprint={2306.14898}, archivePrefix={arXiv}, primaryClass={cs.CL} }

Leaderboard

The Success Rate metric refers to the percentage of tasks that were resolved by the model (received a score of 1.0).

0 - Denotes zero shot evaluation (no interaction)