Apple-Newton Handwriting Recognition (original) (raw)
Despite the abysmal recognition accuracy in the first generation Newton, most Newton afficianados or people interested in handwriting recognition will tell you that the second generation, "Print Recognizer" in Newton OS 2.x was a vast improvement, offering fast and surprisingly accurate recognition. Unlike the first generation software, this second generation recognition engine was developed in-house at Apple, in the Advanced Technology Group (ATG), later (and briefly) renamed the Apple Research Laboratories (ARL). I served as Technical Lead for the project, and together with a core team of three Apple engineers and two contractors, plus a host of other contributors (most of whom are listed in the slides mentioned below), we managed to produce what many have called the first genuinely usable handwriting recognition system. The technical papers, articles, and slides below document a lot of the key technological hurdles that were overcome and the innovations that were made in order to make this possible.
The core recognition technology from the Newton has gained a new lease on life in the Jaguar release of Mac OS X (10.2). Together with a different team of engineers I have helped integrate handwriting recognition into Mac OS X in such a way that it just works with all existing apps; i.e., applications are not required to rev in order to support ink and the routine input of text by a pen and graphics tablet. This technology has been dubbed "Inkwell". (Partly it just seemed like a good name, plus I have a long-standing fondness for the Fleischer Brothers' animations, including their "Out of the Inkwell" series.) The Apple Computer page on Inkwell is here: http://www.apple.com/macosx/jaguar/inkwell.html
For a silly paragraph concocted entirely out of words in the original Newton's limited dictionaries (for benchmarking our recognizer against the old one), check out Test Drive1.
For detailed technical info, please refer to:
- Our most complete paper, "Combining Neural Networks and Context-Driven Search for On-Line, Printed Handwriting Recognition in the Newton", by Larry Yaeger, Brandyn Webb, and Richard Lyon, in the Spring 1998 issue of AAAI's AI Magazine, containing a discussion of the complete recognition system. The most up-to-date pre-print version is only available in (PDF 120KB).
- A plain text, online copy of my article for IEEE Expert, providing a somewhat more informal overview of the entire system.
- A reprint of our NIPS '96 paper, "Effective Training of a Neural Network Character Classifier for Word Recognition" (PDF 32KB), by Larry Yaeger, Richard Lyon, and Brandyn Webb, containing a subset of the neural network issues related to the benefits of prior bias reduction during training.
- A reprint of our invited technical paper from MicroNeuro '96 that concentrates on both some neural network issues and the Geometric Context model, "On-Line Hand-Printing Recognition with Neural Networks (PDF 128KB)", by Richard F. Lyon and Larry S. Yaeger.
- My slides from the '96 "Machines that Learn" workshop at Snowbird, in a PostScript Type 2, Binary format either plain/uncompressed (1.8MB) or unix-compressed (.Z) (733KB).
[Note: there was an extended abstract for the IWFHR5 workshop here for a while, but it was removed when we withdrew our paper from that conference.]
If you have problems printing the PostScript Type 2, Binary format (in which the MicroNeuro '96 paper is provided), email me about obtaining a Type 1, ASCII format.
I've now added PDF format versions of all but the Machines that Learn slides, so most of the papers can be read online if desired (or downloaded in the nice, compact PDF format). I may get around to puting a PDF version of the slides up, but if anyone wants that in a hurry, email me.
Though many people contributed to this effort, the core group consists of:
- Larry Yaeger, Technical Lead, ARL (nee ATG)
- Brandyn Webb, Contractor
- Richard F. Lyon, Manager and Distinguished Scientist, ARL (nee ATG)
- Bill Stafford, Engineer, ARL (nee ATG)
- Les Vogel, Contractor