Marc Levoy's Home Page (original) (raw)

Biographical sketch

Marc Levoy is the VMware Founders Professor of Computer Science (Emeritus) at Stanford University and a Vice President and Fellow at Adobe. In previous lives he worked on computer-assisted cartoon animation (1970s), volume rendering (1980s), 3D scanning (1990s), light field imaging (2000s), and computational photography (2010s). At Stanford he taught computer graphics, digital photography, and the science of art. At Google he launched Street View, co-designed the library book scanner, and led the team that created HDR+, Portrait Mode, and Night Sight for Pixel smartphones. These phones won DPReview's Innovation of the Year (2017 and 2018) and Smartphone Camera of the Year (2019), and Mobile World Congress's Disruptive Innovation Award (2019). Levoy's awards include Cornell University's Charles Goodwin Sands Medal for best undergraduate thesis (1976) and the ACM SIGGRAPH Computer Graphics Achievement Award (1996). He is an ACM Fellow (2007) and member of the National Academy of Engineering (2022).

My Stanford career

Selected highlights

(If you don't see 8 images, open your browser window wider!)
Avolume rendering from my dissertation at UNC The Stanford Bunny, fromour3D scanning repository Volumetric scan and 3D fax ofa happy buddha (by Brian Curless) Logo of theDigital Michelangelo Project Fragment from the giant marble puzzle of the Forma Urbis Romae Project Light Field Rendering, my most-cited paper (withPat Hanrahan) Stanford multi-camera array, andrelatives Light Field Microscope Project

Links to the rest

And some photo essays

I love teaching. After becoming full-time at Google, and finding people there who wanted to know more about photography, I decided to teach a revised but nearly full version of my Stanford courseCS 178 (Digital Photography) at Google. These lectures were recorded and edited to remove proprietary material, at which point Google permitted me to make them public. Here is a link to this course, which I called Lectures on Digital Photography. I also uploaded the lecture videos as aYouTube playlist, where to my surprise they went viral in September 2016. A Googler made a word cloud (at left) algorithmically from the comments on those videos and sent it to me as a gift. In a word cloud the size of each word is proportional to the number of times it appears in the text being processed by the algorithm. It's one of the nicest gifts I have ever received.
For several years my research focused onmaking cameras programmable. One concrete outcome of this project was our Frankencamera architecture, published in thisSIGGRAPH 2010 paper and commercialized in Android'sCamera2/HAL3 APIs. To help me understand the challenges of building photographic applications for a mobile platform, I tried writing an iPhone app myself. The result was SynthCam. By capturing, tracking, aligning, and blending a sequence of video frames, the app made the near-pinhole aperture on an iPhone camera act like the large aperture of a single-lens-reflex (SLR) camera. This includes the SLR's shallow depth of field and resistance to noise in low light. The app launched in January 2011, and seeing it appear in the App Store was a thrill. I originally charged $0.99, but eventually made it free. Here are a few of my favorite reviews of the app:MIT Technology Review,WiReD,The Economist. Unfortunately, with my growing involvement at Google I stopped maintaining the app, and it stopped working with iOS 11. That said, SynthCam's ability toSeeInTheDark inspired the development ofNight Sight, described in more detail below under My Google career.
In 1999 the National Academies publishedFunding a Revolution: Government Support for Computing Research. This landmark study, sometimes called the Brooks-Sutherland report, argued that research in computer science often takes 15 years to pay off. Theiconic illustrationfrom that report is reproduced at right. In 1996Pat Hanrahan and I begin working onlight fields and synthetic focusing, supported by the National Science Foundation. In 2005Ren Ng, a PhD student in our lab, worked out an optical design that allowed dense light fields to be captured using a handheld camera. This design enabled everyday photographs to be refocused after they are captured. I worked on this technology alongside Ren in the Stanford Graphics Laboratory, and later applied it tomicroscopy, but the key ideas behind the light field camera were his. (Ren's doctoral dissertation, titledDigital Light Field Photography, received the 2006 ACM Doctoral Dissertation Award.) In the same year Ren started a company called Refocus Imaging to commercialize this technology. In 2011 that company, renamed Lytro, announced its first camera. So 15 years from initial idea to first product. An exciting ride, but a long wait. At left is the Lytro 1 camera, and below it the first picture I took using the camera, refocused after capture at a near distance, then a far distance. Unfortunately, aside from being refocusable Lytro cameras didn't take great pictures, and refocusability wasn't a strong enough differenting feature to drive consumer demand, so the product (and the follow-on Lytro Illum camera) weren't successful. Eventually,portrait modeson mobile cameras provided similar refocusing and shallow depth of field, but without sacrificing spatial resolution as microlens-based light field cameras do. While thesynthetically defocused pictures produced by mobile cameras are not physically correct, it's hard to tell the difference. BTW,Ren Ngis now a professor in EECS at UC Berkeley.
What do the two images at left have in common? In the mid-1990s, three students in my lab (Andrew Beers, Brian Freyburger, and Apostolos Lerios) left the PhD program to co-found Invisalign. The company's genesis is described in thisInteresting Engineering article, from which the upper image at left was taken. Initially they planned to use 3D laser scanning (3rd image from left above). They later pivoted to taking a mold of your teeth, slicing the mold finely using a milling machine, and photographing each slice to build a 3D model. (They may have since pivoted to another technology.) This gave me an idea - why not slice everyday objects thinly, photograph each slice, and volume render the resulting stack of images? As an experiment, I glued together blocks of wood (lower image at left), sliced the block thinly using Invisalign'smilling machine, andphotographed each slice. Here's a grayscale video of the resulting image stack. The objects at lower right are pine cones embedded in resin. Isn't it interesting how wood grain changes shape over depth? Watch the knots move around in thisblock of maple. Knots are twigs from an early stage of tree growth, which eventually become embedded in the thickening branch from which they sprang. My plan was to volume render this and other natural objects (marble veining?), and write a book (or create a web site) of visualizations. I would call the book_Volumegraphica_ (or Volumegraphia, inspired by Robert Hooke's 1665 bookMicrographia, which contains beautiful drawings of what he could see through a microscope). An early attempt at volume rendering that block of maple is shown at right. Click on it for a video; you can clearly see the twig buried inside the block. Unfortunately, I never found time to puruse this sabbatical-type project, becoming involved instead in multi-camera arrays and light field microscopy (see images above). I also found it challenging to fix natural objects so they don't fall apart in the milling machine, especially if they contain cavities (like conch shells). For more details, see thisslide deck. And for some related projects that use 3D medical imaging technologies instead of physical slicing, see the Visible Human Project, or this stunningvolume rendering of a mouse tail from Resolution Sciences.

Note: I retired from Stanford in 2015. As the note in the box at the top of this page says, I am not accepting new PhD students, postdocs, visiting scholars, or summer interns. Please don't email me asking about opportunities in my research group at Stanford; there is no such group, and there hasn't been one since 2015. Ask me instead about Adobe!

My Google career

Selected highlights

(If you don't see 7 images, open your browser window wider!)
Gaurav Garg taking video for CityBlock,later productized as Google StreetView Figure 1 from US patent 7,586,655on the design of a book scanner Wearing Glass at a public photo walk,photo by Chris Chabot (2012) Portrait Mode on Pixel 2,photo by Marc Levoy (2017) Night Sight on Pixel 3,photo by Diego Perez (2018) Astrophotography on Pixel 4Florian Kainz (2019) My portion of the Keynote addressat the launch of Pixel 4 (2019)

Chronology

In 1993, while as Assistant Professor at Stanford, I lent a few disk drives to PhD students David Filo and Jerry Yang so they could build a table of contents of the internet. This later becameYahoo. In 1996 Larry Page and Sergey Brin, also PhD students at Stanford, began assembling a pile of disk drives two doors down from my office to build an index of the internet. This later becameGoogle. Hmmm, table of contents versus index... At the time both ideas made some sense, but history shows which approach scaled better as the internet grew. (While Larry shared an office with my grad students, I also got to know Sergey well, as a student in one of my computer graphics courses.)

In 2002 I launched a Google-funded Stanford research project calledCityBlock, which Google later commercialized asStreetView. For an overview of the Stanford project, see thistalk I gave at the 2004 Stanford Computer Science Department retreat. This article in TechCrunchoutlines the project's later history at Google, except that the video Larry Page and Marissa Meyer captured while driving around San Francisco predated my project at Stanford, not the other way around as the article says. (I still have his video in my attic.)

In 2003, working as a consultant for Google, I co-designed the book scanner for Google'sProject Ocean, whose goal was to non-destructively digitize millions of books from six of the world's largest libraries: the University of Michigan, Harvard (Harvard University Library), Stanford (Green Library), Oxford (Bodleian Library), and the New York Public Library. While the scanner design actually used in the project is a Google trade secret, the patent linked above is representative of the technologies we were exploring. See alsothis CNET article, or thiscoverage from National Public Radio. (Francois-Marie Lefevere, the co-inventor on this second patent, was formerly one of my students.)

From 2011 through 2020 I built and led a team at Google that worked broadly on cameras and photography, serving first as visiting faculty inGoogleX, and later as a full-time Principal Engineer and Distinguished Engineer inGoogle Research. The internal team name was "Gcam", which became public when thisGoogleX blog article appeared. Our first project was burst-mode photography forhigh dynamic range (HDR) imaging, which we launched in the Explorer edition of Google Glass. See also this publictalk. This work was later extended and applied to mobile photography,launching asHDR+ modein theGoogle Camera app on multiple generations ofNexus andPixel smartphones. The French agencyDxOgave the 2016 Pixel thehighest rating ever given to a smartphone camera, and aneven higher rating to the Pixel 2 in 2017. Here are some albums of photos I shot with the Pixel XL:Desolation Wilderness,Cologne and Paris,New York City,Fort Ross and Sonoma County. Press "i" ("Info") for per-picture captions.

In 2017 we branched out from HDR imaging to synthetic shallow depth of field, based loosely on my 2011SynthCamapp for iPhones. The first version of this technology launched asPortrait Mode on Pixel 2 (see example above). Here is an album of portrait mode shots ofpeople, and another ofsmall objectslike flowers. See also thisinterview in The Verge, this cute explanatory video about the Pixel 2's camera, and these papers inSIGGRAPH Asia 2016 on HDR+ and SIGGRAPH 2018 on portrait mode. We later extended portrait mode to usemachine learning to estimate depth (published inICCV 2019), and on Pixel 4 to use bothdual-pixels and dual-cameras.

In 2018 my team developed (or collaborated with other teams on) several additional technologies:Super Res Zoom (paper inSIGGRAPH 2019), synthetic fill-flash, learning-based white balancing (papers inCVPR 2015 andCVPR 2017), andNight Sight (paper inSIGGRAPH 2019). These technologies launched on Pixel 3. Night Sight was inspired by my prototypeSeeInTheDark app, which is described in thispublic talk, but was never released as a public app. Night Sight has won numerous awards, including DP Review's 2018Innovation of the Year and theDisruptive Device Innovation Awardat Mobile World Congress 2019. See also these inteviews byDP Review andCNET, and this lay person'svideo tutorial by the Washington Posts's Geoffrey Fowler.

In 2019 we launched a real-time version of HDR+ called "Live HDR+", which made Pixel 4's viewfinder WYSIWYG (What You See Is What You Get),and"Dual Exposure Controls", which allowed separate control over image brightness and tone mapping interactively at time of capture - a first for any camera. In Pixel 4 we also extended Night Sight toastrophotography. My team had been exploring this problem for a while; see this2017 article by Google team member Florian Kainz. However, Florian's experiments required a custom-written camera app and manual post-processing. On Pixel 4 consumers could capture similar pictures with a single button press, either handheld or stabilized, such as Florian's stunning tripod photograph (see above) of the Milky Way over Haleakala in Hawaii. Note that pictures like this still require aligning and merging a burst of frames, because over a 4-minute capture the stars do move. Is there art as well as science in our work on Pixel phones? Lots of it; check out this short video by Google on howItalian art influenced the look of Pixel's photography.

Finally, my team also worked on underlying technologies forProject Jump, alight field camerathat captures stereo panoramic videos for VR headsets such asGoogle Cardboard. See also this May 2015 presentation atGoogle I/O. A side project I did at Google in 2016 was to record a modified version of my quarter-long Stanford courseCS 178 (Digital Photography) in front of a live audience (of Googlers). The videos and PDFs of these lectures are available to the public for freehere. See also this YouTubeplaylist.

As an epilogue to this chronology, the technologies my team launched in Pixel phones from 2014 to 2019 were quickly adopted by other smartphone vendors. It seems fair to say that my team's work, and our publications, accelerated the use of smartphones for everyday photography, and disrupted the camera industry. (See also my sidebar below about cell phones versus SLRs.) I believe it was partly for this reason that I was inducted into theNational Academy of Engineering in 2022. However, it's important to note that these technologies were developed by scrappy subteams of smart and creative people (led bySam Hasinoff andYael Pritch), not by me acting as some kind of "hero inventor". Our impact was also facilitated by Google executives (starting with Dave Burke) who believed in the team and were willing to risk their flagship product on our strange ideas.

Oh yes, and then there is the story about how Google got its name, due to a spelling mistake made by one of my Stanford graduate students.
To hear the full story, you'll have to treat me to a glass of wine.

Will cell phones replace SLRs?

Although I've largely moved from SLRs to mirrorless cameras for my big-camera needs, I'm frankly using big cameras less and less, because they have poor dynamic range unless I use bracketing and post-processing, and they take poor pictures at night unless I use a tripod.Both of these use cases have become easy and reliable on cell phones, due mainly to computational photography.HDR+ mode on Nexus and Pixel smartphones is one example. These pictures were taken with the 2015-era Nexus 6P: the left image with HDR+ mode off, and the right image with HDR+ mode on. Click on the thumbnails to see them at full resolution. Look how much cleaner, brighter, and sharper the HDR+ image is. Also, the stained-glass window at the end of the nave is not over-exposed, and there is more detail in the side arches.

Taking a step back, the tech press is fond of saying that computational photography on cell phones is making big cameras obsolete. Let's think this through. It is true that sales of interchangeable-lens cameras (ILCs), which include SLRs and mirrorless cameras, have dropped by two-thirds since 2013, andpoint-and-shoot cameras have largely disappeared, in lockstep with the growth ofmobile phones. Similarly,Photokina, the world's largest photography trade show, shrank by half from 2015 to 2020, and was finally canceled forever in 2021, having been partly supplanted byMobile World Congress. There are confounding factors at play here, including the decimation of professional photography niches like photojournalism, due to shifts in how news is gathered and disseminated. But improvements in the quality of mobile photography over the last decade is certainly a contributor to these changes. Rishi Sanyal, Science Editor for DP Review, who has written extensively aboutPixel 3 andPixel 4, once estimated that the signal-to-noise ratio (SNR) of merged-RAW files from Pixel 4 matches that of RAW files from a micro-four-thirds camera. I think Pixel's dynamic range is actually better - roughly matching that of a full-frame camera, due to low read noise on Sony's recent mobile sensors, but in either case the trend is clear. I also note that as we move from SLRs to mirrorless cameras, hence from optical to electronic viewfinders, the dynamic range of the viewfinder becomes more important. In this respect the Pixel 4, with its"Live HDR+", firmly beats the electronic viewfinders on big mirrorless cameras. (The SNR of cell phone viewfinders still lags, but I expect this to become better in the future.) When Pixel 2 won DP Review'sInnovation of the Year in 2017, comments in the user forums were running 60/40 in favor of "This isn't real photography." By the time Pixel 3 wonthe same award again in 2018, user forum comments were running 80/20 in favor of "Why aren't the SLR makers doing this?" Actually, a few mirrorless cameras do offer single-shutter-press burst capture and automatic merging of frames. But as of 2020 these modes produce only JPEGs, not merged RAW files, and their merging algorithms use only global homographies, not robust tile-by-tile alignment. As a result they work poorly when the camera is handheld or there is motion in the scene.

It's hard to resist concluding that this is a classic case of "disruptive technology", as described by Clayton Christensen inThe Innovator's Dilemma. Cell phones entered at the bottom of the market, so traditional camera makers didn't see them as a threat to their business. Slowly but inexorably, mobile photography improved, due to better sensors and optics, and to algorithmic innovations by Google and its competitors. Computational photography played a role, and so did machine learning. So also did Google's culture ofpublication, which allowed other companies to become "fast-followers". By the time traditional camera companies realized that their market was in danger it was too late. Even now (in 2020), they seem slow to respond. I analyzed their slowness in this2010 article, the same year we published ourFrankencamera architecture(commercialized by Google in itsCamera2/HAL3 API). Little has changed since then, except that cell phones have become more capable. Full disclosure: one factor I missed in my 2010 article was that while large cameras have powerful special-purpose image processing engines, they have relatively weakprogrammable processors. At the same time, they have more pixels to process than cell phones, because their sensors are larger. These two factors made innovation difficult.

In the end, it's hard to avoid thinking it's "Game Over" for ILCs, except in niches likesports and wildlife photography, where big glass is mandated by the laws of physics. Even Annie Leibovitz thinks this. For the rest of us, it's not only that"the best camera is the one that's with you"; it may also be true that the camera that's with you(meaning your cell phone)is your best camera!

ChatGPT's version of this sidebar, rewritten in hip-hop style.

My Adobe career

A history yet to be written. 😉

But we'rehiring! Especially MS and PhD graduates with expertise in computer vision, computational photography, and/or AI.

Meanwhile, see this 1-hour, in-depthpodcast / interviewby Nilay Patel of The Verge, on camera hardware, software, and artistic expression at Google and Adobe.
And this 10-minute talk at Adobe MAX 2020 aboutcomputational photography at the point of capture.
In March 2022Adobe interviewed meabout my work in photography, following my election to theNational Academy of Engineering. See also this coverage inPetapixel.
Here's a nice July 2022interview by Atila Iamarino of Brazilian public television, about computational photography in general, presented in a mix of English and Portuguese.
Here is my Adobe team's gift to me of asuperhero backstory.

TheSneak Peek my team showed at Adobe MAX 2023 for removing reflections from windows (see the first row of photos above) is now available to photographers as a feature inAdobe Camera Raw. (It will eventually be available in Lightroom.) The technology is designed to address only one kind of reflection — from plate glass windows that cover most or all of your field of view. It's not designed to remove reflections from windows that are small or far away, or where the window frame is within the field of view, or reflections from objects like wine glasses, car bodies, or clouds reflected in a lake. The goal of this feature is to help you turn a photo you might otherwise delete (first image at left) into one that is good enough to share (second image). By the way, the feature also predicts the image that was reflected in the window (third image), a backyard scene in this example. The technology is powered by AI, but it's not generative AI, so it will never create objects that weren’t present in the original photograph. For more details, see thisAdobe blog and thisarXiv paper(to appear in Proc. CVPR 2025). The feature has evidently resonated with the public; here is some coverage byPetapixel,DiYPhotography,DP Review, andThe Verge. See also this tutorial by Adobe Photoshop evangelistJulieane Kost, demos by vloggersBrian Matiash andColin Smith, and this interview of me inCNET.

Photography

My 5-year journey from SLRs to cell phones

Turkey, June 2015 Myanmar, December 2016 Antarctica, December 2017 Italy, Switzerland, Germany, 2018 Barcelona, February 2019 New York, March 2020

Other favorite Pixel 3 and Pixel 4 albums (with a Google Photos parlour trick in the last album)
(Mostly) nature shots New York, January 2020 Pixel 4 launch event Prague to Vienna by bicycle Eastern Sierras, August 2019 Portrait Blur on famous paintings

Selected older travel albums

India, December 2008 Thailand/Cambodia, December 2013 Chile and Patagonia, December 2015 Van trip across EU, June 2016 Croatia and Italy, June 2017 Tour of the Baltics, June 2018

Autobiography

In 2012 I was invited to give the commencement address at the2012 Doctoral Hooding Ceremony of the University of North Carolina (from which I graduated with a doctorate in 1989).After the ceremony a number of people asked for the text of my address. Here it is, retrospectively titled"Where do disruptive ideas come from?". Or here isUNC's version, with more photos like the one at left. And here is thevideo.
Portrait photographer Louis Fabian Bachrach took some nice photographs (here andhere) of me in 1997 for the Computer Museum in Boston (now closed). I do occasionally wear something besides blue dress shirts. Here are shots with other shirts, fromAugust 2001 andJuly 2003. Yes, that's a Death Ride T-shirt in the last shot. I also rode in 2005, and yes, I finished all 5 passes - 15,000 feet of climbing. That's why I'm smiling in the official ride photograph (shown at left), taken at the top of Carson Pass after 12 hours of cycling.
During a 1998-99 sabbatical in Italy, my students and I digitized 10 of Michelangelo's statues in Florence. We called this theDigital Michelangelo Project. Here are some[photographic essays](/projects/mich/mich.html#private collection) about personal aspects of the sabbatical. In particular, I spent the yearlearning to carve in marble. At left is my first piece - a mortar with decorated supports. Here are some sculptures bymy mother, who unlike me had real talent.
The Digital Michelangelo Project was not my first foray into measuring and rendering 3D objects. Here are some drawings I made in college for theHistoric American Buildings Survey. In this project the measuring was done by hand - using rulers, architect's combs, and similar devices.
I still like finding and measuring old objects, especially if it involves getting dirty. This photographic essay describes a week I spent on anarchaeological dig in the Roman Forum. The image at left, taken during the dig, graces the front cover ofThe Bluffer's Guide to Archaeology.
My favorite radio interview, by Noah Adams of National Public Radio's All Things Considered - about the diverging gaze directions of the two eyes in Michelangelo's David (June 13, 2000). (Click to hear the interview using RealAudio at14.4Kbs or28.8Kbs, or as a[.wav file](/projects/mich/publicity/ npr-atc-13jun00/npr-atc-13jun00.WAV).)
This interview, by Guy Raz, weekend host of All Things Considered, runs a close second. It's about theFrankencamera (October 11, 2009), pictured at left. (Click here for NPR'sweb page containing the story and pictures, and here for a direct link to the audio as an.mp3 file.)Finally, here's a text-only interview at SIGGRAPH 2003, by Wendy Ju, with reminiscences of my early mentors in computer graphics.
Speaking of computer graphics, I'm fond of the front cover of theSiggraph 2001 proceedings. The image is from a paper (in the proceedings) onsubsurface scattering, co-authored with Henrik Wann Jensen, Steve Marschner, and Pat Hanrahan. And check out this milk. This paper won a Technical Academy Award in 2004. Subsurface scattering is now ubiquitous in CG-intensive movies.
However, not everything went smoothly at Siggraph 2001. A more-strenuous-than-expected after-Siggraph hike inspired Pat Hanrahan's and my students to create this humorous movie poster. Click here for the innocent version of this story. And here for the real story.

Genealogy

A lot of my early Stanford research related to volume data. The cause may be genetic. My mother's cousin David Chesler is credited with the first demonstration of filtered backprojection, the dominant method used in computed tomography (CT) and positron emission tomography (PET) for combining multiple projections to yield 3D medical data. Here is a descriptionof his contributions.
My father's genes also seem to be guiding my research tastes. Optics has been in my family for four generations. My father Barton Monroe Levoy and my grandfather Monroe Benjamin Levoy were opticians and sellers of eyeglasses through their company, Tura. At left is an earlybrochure. Tura is still alive and well, with corporate headquarters in New York City, although it is no longer in the family. Here is a marvelously illustratedtimelinethey assembled about the history of the company; it includes the image at left.
Going back further, my great-grandfather Benjamin Monroe Levoy sold eyeglasses, cameras, microscopes, and other optical instruments in New York City a century ago. Here is a piece of stationery from his store. He later moved to 42nd street, as evidenced by the address on the case of theseeyeglasses, remade as pince-nez with a retractor. (At left is a closeup of the embossed address.) And here is awooden boxhe used to mail eyeglasses to customers. The stamp is dated 1902.
In the drawing (at left) illustrating that stationary, I believe you can see a microscope. In any case I have an old microscope from his store. This specimen of a silkworm mouth, which accompanied the microscope, appears in our SIGGRAPH 2006 paper on light field microscopy.
My great-grandfather apparently also sold binoculars from the store. This pair, made by Jena Glass about 80 years ago and inscribed with the name B.M. Levoy, New York, was recovered by a SWAT team during a drug raid in South Florida in 2008. They have undoubtedly passed through many hands during their long and storied life. It would be fascinating to watch a video of everything these lenses have seen.
When my Stanford students worked with me at the optical bench, they were perplexed by my arcane knowledge of mirror technology. Returning to my mother's side of the family, my great-grandfather Jacob Chesler built afactory in Brooklyn (pictured at left) that made hardware. The building passed to my grandfather Nathan Chesler, who converted it to making decoratively bevelled mirrors. I spent many pleasant hours studying the factory's machinery, designed by my uncle Bertram Chesler, for silvering large plate-glass mirrors.