Camera 2.0: New computing platforms for computational photography (original) (raw)

3/15/12 - The FCam API is now available on the Nokia N9 Linux-based smartphone. See below for details.

Marc Levoy has retired from Stanford University to lead a team at Google. This project is no longer active, and the Stanford Frankencamera is not available for purchase. However, the FCam API is available on cell phones and tablets that run Android version "L" (Lollipop), as theCamera2 API (Application Programming Interface) andHAL v3 (Hardware Abstraction Layer) for the cameras on these devices. See also this presentation by former Stanford PhD student Eddy Talvala atGoogle I/O 2014, or this summary byAndroid Police. Apple offers a similar API in iOS.

Overview

Computational photography refers broadly to sensing strategies and algorithmic techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. Representative techniques include high dynamic range imaging, flash-noflash imaging, coded aperture and coded exposure imaging, photography under structured illumination, multi-perspective and panoramic stitching, digital photomontage, all-focus imaging, and light field imaging. Although interest in computational photography has steadily increased among graphics and vision researchers, progress has been hampered by the lack of a portable, programmable camera platform with enough image quality and computing power to be used for everyday photography. To address this problem, we have been pursuing two subprojects:

**Computational Photography on Cell Phones.**Over the past five years, the cameras in cell phones have improved dramatically in resolution, optical quality, and photographic functionality. (See 1st and 2nd images above.) Moreover, camera phones offer features that dedicated cameras do not: wireless connectivity, a high-resolution display, 3D graphics, and high-quality audio. Finally and perhaps most importantly, these platforms run real operating systems, which vendors have begun opening to third-party developers. We have therefore begun developing computational photography applications for commerical cell phones. Our Eurographics 2008 paper on viewfinder alignment is an example.
**The Stanford Frankencamera.**Despite these encouraging trends, there are computational photography experiments that cannot be implemented on today's cell phones. either because the cameras' sensor or optics aren't good enough, the computing resources aren't powerful enough, or the APIs connecting the camera to the computing are too restrictive. We have therefore built a custom camera that runs Linux, is fully programmable (including its digital signal processor) and connected to the Internet, and accommodates SLR lenses and (soon) SLR-quality sensors. Our current prototype (3rd and 4th images above) is constructed from off-the-shelf parts, in some cases borrowed from dead cameras. It's also ugly - hence the name. After pursuing these two subprojects independently for two years, we decided that the best approach was to define a single open architecture for programmable cameras. Our architecture consists of a hardware specification, a software stack based on Linux, and an API with bindings for C++. The architecture permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes. To demonstrate the viability of this architecture, we have built two reference implementations: a Nokia N900 smartphone with a modified software stack, and the Frankencamera F2 pictured above. Click here for ourSIGGRAPH 2010 paper about this architecture.

Going public

On July 21, 2010 we released the source code for our implementation of the Frankencamera architecture andFCam APIon theNokia N900 smartphone. A binary executable was also released, which can be downloaded to any retail N900 without bricking the phone, thereby making its camera programmable. To get started turning your N900 into a Frankencamera, clickhere. FCam is now also available on the Nokia N9 as part of the Meego/Harmattan 1.2 operating system update. The Harmattan documentation is here. The N9 is not sold by carriers in the U.S., but unlocked N9's can be purchased on Amazon, and we believe they work with AT&T and T-Mobile.

In addition to our implementation on the N900, we plan to make the more powerful custom-built Frankencamera F's available to the research and teaching community. However, the Frankencamera F2 (pictured above) is not worth buying, since its sensor is only cell-phone sized. (Hence, it has a 6x crop factor relative to a 35mm sensor. The only Canon lens that makes sense given this crop factor is a 10-22mm set to 10mm. This configuration is pictured on the web page of ourSIGGRAPH 2010 paper.) Instead, we are actively working on the Frankencamera F3, which will have a 24mm x 24mm sensor, hence a 1x crop factor (i.e. full-frame), at least in the vertical direction. We don't have a release date for the F3, but we are shooting for Spring 2012. Our plan is to distribute the camera "at cost" (might be $8K) to anyone who wishes to purchase one.

Using FCam in university courses

Are you a university instructor teaching a course in computational photography? If so, then we want to talk to you. We are currently assembling packages to help teach computational photography at the university level. Each package will consist of 20-30 lectures on computational photography written byMarc Levoy (Stanford) andFredo Durand (MIT), a box containing 10 Nokia N900 smartphones, and one Frankencamera F3. The lecture material will be free, available to anyone and downloadable from the Internet. The N900s will also be free, but as loaners only, and subject to available supply. This loan program is supported by a generous grant from Nokia. The Frankencamera F3s will be sold "at cost", as mentioned earlier.

As of Summer 2010, this package has been used in two courses - at Stanford University (co-taught by Marc Levoy and Fredo Durand) and at ETH Zurich (taught byMarc Pollefeys). In both courses we loaned Nokia N900s to every student, and assignment #1 was to replace its autofocus algorithm. (The phone's camera, while small, has a movable lens. For more details look at the "Hello Camera" link on thecourse web site.) We graded the assignment on the accuracy they could focus on a test scene we provided, as well as focusing speed in milliseconds. An assignment like this would have been impossible before FCam. Two of the students submitted algorithms that were better than Nokia's; these were presented in Finland at the conclusion of the course. The students also did interesting projects for the course, some of which might be submitted for publication over the coming year.

At present this program is designed mainly for universities in North America and Europe. However, we will entertain all reasonable requests. In addition, for universities in the United States, the Frankencamera F3s will be free, supported by a grant from the National Science Foundation. Want to find out more about this program? If you're a U.S. university looking to join the NSF-supported program, send email toMarc Levoy. If you're any other university looking for loaner N900s, contactTimo Ahonen at Nokia.

Funding sources

The Camera 2.0 project, which began as a collaboration between theStanford Computer Graphics Laboratoryand theNokia Research Center Palo Alto Laboratory, has also received funding from Adobe Systems, Kodak, Hewlett-Packard, The Walt Disney Company, Intel, Texas Instruments, Google, NVIDIA, and Sony. The NSF portion of the project is in collaboration withFredo Durand andWilliam Freeman of MIT.

People

Andrew Adams < abadams@mit.csail.edu >
Jongmin Baek < [first initial][last name]@[cs].[school name].[school domain] >
Jennifer Dolson < jldolson [at] stanford [dot] edu >
Mark Horowitz < horowitz@ee.stanford.edu >
David Jacobs < dejacobs [at] cs [full-stop] stanford [full-stop] edu >
Marc Levoy
Wojciech Matusik < wojciech@csail.mit.edu >
Sung Hee Park < shpark7 [at] stanford [dot] edu >
Kari Pulli (now at NVIDIA)
Eino-Ville (Eddy) Talvala (now at Google)

Collaborators

Alumni

Abe Davis < abedavis@cs.stanford.edu >

Recent papers in this area:

Decoupling algorithms from schedules for easy optimization of image processing pipelines

Jonathan Ragan-Kelley,Andrew Adams,Sylvain Paris,Marc Levoy,Saman Amarasinghe,Fredo Durand

ACM Transactions on Graphics 31(4) (Proc. SIGGRAPH 2012)

Experimental Platforms for Computational Photography

Marc Levoy

IEEE Computer Graphics and Applications, Vol. 30, No. 5, September/October, 2010, pp. 81-87.

The Frankencamera: An Experimental Platform for Computational Photography

Andrew Adams, Eino-Ville (Eddy) Talvala,Sung Hee Park,David E. Jacobs,Boris Ajdin, Natasha Gelfand,Jennifer Dolson,Daniel Vaquero,Jongmin Baek,Marius Tico, Henrik P.A. Lensch,Wojciech Matusik,Kari Pulli,Mark Horowitz,Marc Levoy

Proc. SIGGRAPH 2010

Reprinted in CACM, November 2012, with an introductory technical perspective by Richard Szeliski

Gaussian KD-Trees for Fast High-Dimensional Filtering

Andrew Adams Natasha Gelfand,Jennifer Dolson,Marc Levoy

Proc. SIGGRAPH 2009

Spatially Adaptive Photographic Flash

Rolf Adelsberger, Remo Ziegler,Marc Levoy, Markus Gross

Technical Report 612, ETH Zurich, Institute of Visual Computing, December 2008.

Viewfinder Alignment

Andrew Adams,Natasha Gelfand,Kari Pulli

Proc. Eurographics 2008

Publicity about the project

We'd love to hear from you, but first check out our list offrequently asked questions.

Left: Marc Levoy.Right: Andrew Adams with camera.Behind them is theStanford Multi-Camera Array. Another picture of the camera, used in the July 2010 Stanford Report article. Images by Linda Cicero.

Stanford Engineering Research Profile (August 2009), superceded by Stanford Report (below) Stanford Report (August 31), picked up bySlashdot (September 3),CrunchGear,Physorg.com,ScienceDaily,Popular Science,Make:,Hack A Day,Gizmag,cnet news,ZDNet,ZDNet UK,engadget,Linux Foundation,Linux Magazine,Coolest Gadgets,LittleAbout,SlipperyBrick,TG Daily,TechShout!,GizmoWatch,HotHardware,Shuttertalk,ePHOTOzine,This Week in Photography (TWIP),Neowin,HardOCP,techradar,Heritage Key,Science Codex,Malaysia Sun,ALT1040 (Spanish),NeoTeo (Spanish),MuyComputer (Spanish),Tendencias Informáticas (Spanish),Quesabesde (Spanish),Mobiunaute (French),Le monde de la photo (French),RosInvest.Com (Russian),Ferra (Russian),ITC Online (Ukrainian),akam.no (Norwegian),tweakers.net (Dutch),La Repubblica (Italian),Universy.it (Italian),Digital Focus (Italian),PuntoInformatico (Italian),Tecnocino (Italian),dphoto (Italian),Wissenschaft aktuell (German),Chip Online (German),Spiegel Online (German),News1 (Hebrew),Internet Watch (Japanese),United Daily News (Chinese),cnBeta (Chinese),Sina.com (Chinese), etc. NPR's All Things Considered (October 11) New Scientist (November 11) We have gaven up tracking these articles; there are too many of them.Caveat: We have found articles illustrated by "alternative Frankencameras" that are not related to our project. Some are quiteamusing, while others look likeRussian constructivist art projects.The real Stanford Frankencamera is pictured at left and at the top of this web page.

Left: The Frankencamera F2 with two Canon flash units attached.Right: A picture taken by this Franken-arrangement. The smaller 430EX flash was programmed using ourFCam APIto strobe repeatedly, and the larger 580EX was programmed to fire once at the end of the exposure.

Stanford Report (July 21, 2010), picked up byPhysOrg.com,TechNewsDaily,Scientific Computing,Adorama Photo News. Here is ahigh-resolution versionof the photograph used in the Stanford Report article. Nokia Conversations (Nokia's official blog, July 21), picked up byUnwiredView.com,YouMobile.org SlashGear,intomobile,recombu.com,Softpedia,Mobiletor.com,TechGadgets.in,PluGGd.in,TusEquipos.com (Spanish),MovilZona (Spanish),tuexpertoMovil.com (Spanish),Celularis.com (Spanish),Canarias7.es (Spanish),mobile blog.it (Italian),Golem.de (German),Pro-Linux.de (German),Mobilfunk-Talk.de (German),XaLuan (Vietnamese),mforum.ru (Russian),TechnoPortal.ua (Ukraine) Nokia Conversations (July 27) MIT Technology Review (July 30) Imaging Executive Podcast (September 15 audio interview by Paul Worthington of the 6Sight Report), also available atDaily Photo News Stanford Magazine (September/October issue) New York Times (December 18), reprinted in the San Francisco Chronicle (December 19) and elsewhere Stanford Daily (January 6, 2011) MIT Technology Review (June 2011 issue) The Economist (September 3, 2011, see also this blog in the Economist on June 29) Science News (January 28, 2012)