CSE 274 Selected Topics in Graphics: Sampling and Reconstruction of Visual Appearance (original) (raw)

CSE 274 Topics in Computer Graphics, Spring 2018, Prof. Ravi Ramamoorthi Time and Place: Tu/Th 9:30-10:50, EBU3B 4140

Overview

There are a number of topics in computer graphics which today require sampling and reconstruction of high-dimensional visual appearance datasets, such as high-quality real-time rendering with precomputed light transport, Monte Carlo rendering, light transport acquisition, and light field imaging for virtual reality. Perhaps the most significant recent developments have arisen in the area of Monte Carlo image synthesis via path tracing and related methods. Indeed, one of the perennial goals of computer graphics is creating high quality images which are indistinguishable from photographs: a goal referred to as photorealism. (Please note that since CSE 168 on Computer Graphics II: Rendering is not currently offered, this course also aims to provide basic background on rendering and the first 3 weeks cover the essential topics of CSE 168, especially physically-based path tracing. )

In essence, one is sampling a high-dimensional function (8D for image coordinates, soft shadows on an area light, global illumination from incident directions, and depth of field on a lens), then reconstructing or denoising it to produce the final image. The industry has recently moved to physically-based path tracing as its preferred solution, with a low sample count, followed by advanced reconstruction and denoising methods. These advances have reduced the computation time by one to two orders of magnitude, enabling production rendering with physically-based global illumination, and this approach is beginning to be considered for real-time rendering applications. This course surveys these recent developments over the past 5-10 years, in many of which UCSD faculty have played a prominent role. (Note that no background on Monte Carlo rendering is required, since most algorithms take the actual rendering process as a black box).

Beyond traditional rendering, sampling and reconstruction of visual appearance is critical for many other applications, such as acquisition of light transport from real-world scenes, light field cameras that capture both spatial and angular distributions of light, and real-time methods based on precomputed light transport. All of these problems involve 4D-8D functions, which are prohibitive to sample by brute force, and where coherence or sparsity needs to be exploited to enable sparse sampling and sophisticated reconstrcution. The course surveys recent advances, within the last decade, in all of these areas.

This body of work is built on mathematical techniques, some of which are recent developments, including multi-dimensional Fourier analysis of light transport spectra, compressive sensing for sparse datasets, and deep learning for feature-based image denoising. While no prior background on these areas is required, the course will provide a brief high-level introduction, to the extent these topics are needed for the practical applications we consider.

Please note that CSE 274 is a topics course, covering current topics in computer graphics. Course content may change every year or two years. This course material was first offered last year (Winter 2018), and you are allowed to enrol and get full credit even if you have taken CSE 274 on another occasion (apart from Winter 2018) with me or other faculty. Below are a few example images produced using some of the algorithms and systems we will be discussing.

Prerequisites

The course is targeted towards MS and PhD students with a knowledge of and interest in computer graphics (at the level of the introductory CSE 167 computer graphics course, or equivalent at another university.) Earlier, we used to also teach CSE 168 on Computer Graphics II: Rendering, one of whose main goals is to cover physically-based Monte Carlo path tracing. CSE 274 covers the core part of CSE 168 as background in the first 3 weeks and will likely also be of interest to those who want a substitute for CSE 168. In fact, we provide an optional path tracing assignment for those who are interested. Note however, that no prior knowledge of rendering or CSE 168 is required, and that the path tracing assignment is strictly optional/for your own benefit and not graded... you can also use an existing off-the-shelf black box path tracer (such as PBRT, Mitsuba or OptiX) for your project, focusing on sampling and reconstruction. We welcome all PhD students working in graphics, vision, and robotics. We are willing to consider students who do not completely fulfil the pre-requisites in exceptional cases, if they have a strong interest in the material. Undergraduates who have taken CSE 167 (and/or other advanced courses such as CSE 163/169) are also welcome, space permitting (space is not usually a problem).

Relationship to Other Courses

The course is an integral part of the vision and graphics track for undergraduate, MS and PhD students (The instructor can help in getting any relevant permissions for it to count for credit towards the track as needed). It builds on the undergraduate graphics classes CSE 167 [taught by Profs. Schulze/myself in the fall/winter] and CSE 163/169/190. If you like this course, you may also be interested in CSE 291s in winter and spring (this year is our strongest ever sequence of graduate graphics courses): machine learning for 3D geometry taught in the winter by Prof. Hao Su and spring courses on computational photography taught by Ben Ochoa, and physical simulation taught by Steve Rotenberg.

Course Format and Requirements

The course will consist of lectures on the relevant topics by the instructor, student presentations of papers covering current research in the area, and student projects. A syllabus/schedule is noted below. The grading will depend primarily on the final project. (Loosely, we note this as approximately 60% for the project, 30% for paper presentation and 10% for class participation.) Students are expected to come to class regularly and participate in discussions, since this is an advanced graduate course.

In general, roughly (depends on the number of students in the course), 1 or 2 paper presentations will be required (if the number of students in the course exceeds the number of presentations, some may be done jointly by a group of two students). A project is not required for students taking the course S/U or P/NP; we leave it to your discretion if you want to register for 2 or 4 units in this case. An S/U registration is a good option for PhD students and others to read papers on this exciting topic and learn about the area without committing too much effort into a course project, and we encourage you to sign up. Auditors, who simply want to sit in on the course are also welcome; however, we prefer if you sign up for the course pass/fail instead [this just involves doing one or two paper presentations].

Students taking the course for a letter grade are required to do a project [this may be in groups of 2], give a presentation in class regarding their results, and also submit a final written report. Wide flexibility is available with respect to project topics, provided they relate loosely to the subject matter of the course. We expect that most projects will implement one or more of the algorithms or papers discussed in the course, showing results on sampling and reconstruction of visual appearance. We welcome suggestions from the students on alternative project ideas. The best projects will go beyond the published work in some way, such as trying out an alternative or better approach or trying to develop some variant or more general version of the technique. However, this is not essential; in general, students who fulfil all course requirements including a well-executed project will easily receive an A in the course.

As a potentially easier alternative to the project, we will also accept a well-written summary or tutorial, covering 3 or 4 papers. The best summaries will point out links between the papers not noticed by the original authors and suggest improvements or directions for future research. However, this option is recommended only as a last resort and will generally receive a slightly lower score; we prefer that you do a good project (which may involve understanding a few papers in any case).

The optional path tracing assignment is not explicitly graded, but if you do it as part of your project, we encourage you to include documentation in your final project (or survey if doing a survey of papers) report so we can consider it when grading. Note that this is not required and no penalty will be levied/the assignment is strictly optional. You are also welcome and encouraged to use existing path tracers (PBRT, Mitsuba, OptiX) as black boxes and focus on sampling and reconstruction, if that is most suitable for your project. Indeed, the philosophy of recent work and the course is to treat the path tracer as a black box and focus on sampling/reconstruction.

Lectures and Office Hours

The lectures will be in EBU3B (CSE Building) 4140 from 9:30-10:50 on Tuesdays and Thursdays. Students are expected to come to the lectures and participate in discussion. Office hours will be immediately after class from 11-12 in the Professor's office, EBU3B 4118. You may also e-mail for another time to meet if that is not convenient.

Teaching Assistant

We have a teaching assistant for the course, Jiyang Yu. His e-mail is jiy173@eng.ucsd.edu. His office hours will be in 4150 (CS bldg) Mondays and Wednesdays from 11-noon, or you can e-mail him to schedule an appointment.

Topics

Topics to be covered include

Resources

Outline

The (tentative) course schedule is as follows. In general, you will likely benefit from doing the reading (i.e. the papers assigned for a particular date) before class; it will at least make for more lively discussion.

Ravi Ramamoorthi

Last modified: Thu Oct 11 08:39:44 PDT 2018