Paper page - Textbooks Are All You Need (original) (raw)

Published on Jun 20, 2023

Submitted by AK

on Jun 21, 2023

Authors:

Abstract

A new compact Transformer-based large language model for code, phi-1, achieves high accuracy on coding benchmarks despite having fewer parameters than competing models.

We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2306.11644

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash