(original) (raw)

Computational Photography - CSCI-GA.3033-012

Semester: Fall 2013. Time and Location: Thursday 5:00-6:50pm, Room 1221, 715 Broadway. TA: To be decided Instructor: Rob Fergus Office hours: Thursday 4-5pm, Room 1226, 12th floor, 715 Broadway.

Overview

Computational Photography is an exciting new area at the intersection of Computer Graphics and Computer Vision. Through the use of computation, its goal is to move beyond the limitations of conventional photography to produce enhanced and novel imagery of the world around us. The main focus of the course will be on software-based methods for producing visually compelling pictures. However, it will also cover novel camera designs, for which computation is integral to their operation. The course will explain the principles behind many of the advanced tools that can be found in Adobe Photoshop, although the use of this package itself is outside the scope of the course. The course will be suitable for advanced undergraduates, masters and PhD students. A reasonable knowledge of linear algebra is required and familiarity with Matlab is desirable. Assessment will be through coursework and a course project.

Schedule

Acknowledgements

The instuctor would like to thank the large number of people who have made their slides available for use in this course. In particular, I'd like to thank Fredo Durand and Bill Freeman for letting me use their wonderful slides. Thanks also to: Eero Simoncelli, Ravi Ramamoorthi, Rick Szeliski, Steve Seitz, whose material I've used (and credited) in the above slides.

Course project ideas

1. Camera spectral response calibration . This project involves determining the sensitivity vs wavelength curve for each color channel in the camera. This is a hardware-oriented project. The software component involves the use of simple least-squares minimization techniques in Matlab. The DSLR camera will be provided.

2. Seam Carving implementation . The initial part of this project involves re-implementing the Seam Carving paper by Avidan and Shamir PDF. This is a software-only project, requiring a C-based implementation to give real-time performance. The latter part of the project involves applying this method to a variety of settings and imagery, such as text images, web pages, Possible

3. 2-D Panoramas . This project involves the re-implementation the panorama creation paper of Brown et al PDF. This is a software-only project that can be performed in Matlab. Executables will be provided for certain parts of the algorithm, such as feature detection.

4. Creation of David Hockney Joiners. . The artist David Hockney has created images, known as "joiners", that consist of a collection of photos of scene, taken from a wide range of viewpoints, arranged in a collage to give a photo-collage yielding a non-realistic but compelling representation of the 3-D scene. Creating these currently involves a lot of manual alignment and artistic skill. The project would involve creating a tool that would let a user create these images much more quickly, while retaining artistic control over the process. This is a software-only project which an artistic and human-compute-interface flavor.

5. Hot or not. . This project explores ways to parametrize our notion of beauty. First, a large collection of images of people from websites such as Hot Or Not should be collected, along with metadata such as viewers ratings, hair color, eye color etc. After alignment of the faces, you should try and build simple models that allow a user to traverse the space of faces in a logical manner, e.g. blonder hair, prettier, thinner face etc. Some machine learning may be required, so is more suitable for someone who took Prof. LeCun's machine learning class last semester.

6. Cleaning up surveillance footage. . Surveillance cameras are now a common feature of many big cities in the US and Europe. However, the quality of footage that the record varies enormously. Older cameras record at modest resolutions and are further compromised by heavy compression artifacts. This project involves taking some real surveillance camera footage of a crime taking place and trying to remove the compression artifacts that prevent clear identification of the suspect. The key to this will be modeling the compression process and using image priors to infer what the uncompressed image would have looked like. This is a software-only project that can be performed in Matlab.

7. Image Analogies. . Implement the Image Analogies paper by Aaron Hertzmann (former NYU student) and co-authors. See the project page here

Additional Material

Matlab guides

Matlab tutorial by Hany Farid and Eero Simoncelli Link

A more comprehensive Matlab tutorial by David Griffiths Link

Further documentation on Matlab can be found here Link

Books

Forsyth, David A., and Ponce, J.Computer Vision: A Modern Approach, Prentice Hall, 2003.

Goldberg, Norman.Camera Technology: the Dark Side of the Lens, Academic Press, 2002.

Hartley, R. and Zisserman, A.Multiple View Geometry in Computer Vision, Academic Press, 2002.

Palmer, Stephen E. Vision Science: Photos to Phenomenology, MIT Press, 1999.

Ray, Sidney F.Applied Photographic Optics, Focal Press, 3/e, 2002.

Strang, Gilbert.Linear Algebra and Its Applications 2/e, Academic Press, 1980.

Wandell, Brian A.Foundations of Vision, Sinauer, 1995.

Further References

Adelson, Edward H. and Bergen, James R.The Plenoptic Function and the Elements of Early Vision. In Computational Models of Visual Processing, ed. M. Landy and J. A. Movshon, MIT Press, 1991, 3-20. Week 15

Adelson, Edward H. and Wang, John Y. A.Single Lens Stereo with a Plenoptic Camera, IEEE PAMI, Vol 14, No. 2, 1992Week 15

Avidan, Shai and Shamir, Ariel.Seam Carving for Content-Aware Image Resizing, ACM Transactions on Graphics 26(3), SIGGRAPH 2007. Week 9

Brown, M. and Szeliski, R. and Winder, S.Multi-image matching using multi-scale oriented patches, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, 510-517. Week 7

Burt, P. J., and Adelson, E. H.The Laplacian Pyramid as a Compact Image Code, IEEE Transactions On Communications, Vol. COM-31(4):532-540, 1983.Week 5

Cathey, W. T. and Dowski, E. _A new paradigm for imaging systems_Applied Optics, 1995. Week 13

Criminisi, A., and Perez, P. and Toyama, K.Object Removal by Exemplar-based Inpainting, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Week 10

Debevic, Paul E. and Malik, Jitendra_Recovering High Dynamic Range Radiance Maps from Photographs_, ACM SIGGRAPH, 1997. Week 12

Dowski, E. and Cathey, W. T. _Single-lens single-image incoherent passive-ranging systems_Applied Optics, 1994. Week 13

Efros, Alexei A., and Leung, Thomas K.Texture Synthesis by Non-parametric Sampling, IEEE ICCV, 1999. Week 10

Efros, Alexei A, and Freeman, William T.Image Quilting for Texture Synthesis and Transfer, Proceedings of SIGGRAPH'01, Los Angeles, California, August, 2001.Week 10

Fergus, R. and Singh, B. and Hertzmann, A. and Roweis, S. T. and Freeman, W.T.Removing Camera Shake From A Single Photograph, ACM SIGGRAPH, 2006. Week 12

Field, D. What is the goal of sensory coding? Neural Computation, 1994. Week 5

Fischler, M. and Bolles, R.Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, 24(6):381-395, 1981. Week 7

Kolmogorov, V. and Zabih, R. What Energy Functions can be Minimized via Graph Cuts?, PAMI Feb 2004. Weeks 7,14

Kundur, D. and Hatzinakos, D., Blind Image Deconvolution IEEE Signal Processign, May 1996. Week 12

Levin, A.Blind Motion Deblurring using Image Statistics, NIPS 2006 Week 12

Levin, A. and Fergus, R. and Durand, F. and Freeman, W. T.Image and Depth from a Conventional Camera with a Coded Aperture, ACM SIGGRAPH 2007 Week 13

Levoy, M. and Hanrahan, P. Light Field Rendering ACM SIGGRAPH, 1996. Week 15

Lowe, David G.Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, 60(2):91-110, 2004. Week 7

Lucas, Bruce D., and Kanade, Takeo.An iterative image registration technique with an application to stereo vision, Proceedings of the Image Understanding Workshop, 121-130, 1981.Week 7

Ng, R. and Levoy, M. and Bredif, M. and Duval, G. and Horowitz, M. and Hanrahan, P.Light Field Photography with a Hand-Held Plenoptic Camera, Stanford University Computer Science Tech Report CSTR 2005-02, 2005.Week 15

Portilla, J. and Strela, V. and Wainwright, M. and Simoncelli, E. P.Image denoising using scale mixtures of Gaussians in the wavelet domain IEEE TIP 2003.Week 6

Simoncelli, E. P.Statistical modeling of photographic images Chapter 4, Handbook of Image and Video Processing, 2005. Week 5

Raskar, R. and Agrawal, A. and Tubmlin, J. Coded Exposure Photography: Motion Deblurring using Fluttered Shutter ACM SIGGRAPH 2006.Week 5

Roth, S. and Black, M. J.,Fields of Experts: A framework for learning image priors CVPR 2005. Week 6