CSE 274 Selected Topics in Graphics: Sampling and Reconstruction of Visual Appearance: From Denoising to View Synthesis (original) (raw)

CSE 274 Topics in Computer Graphics, Fall 2022, Prof. Ravi Ramamoorthi Time and Place: Tu/Th 9:30-10:50, EBU3B 4140

Overview

There are a number of topics in computer graphics which today require sampling and reconstruction of high-dimensional visual appearance datasets, such as high-quality real-time rendering with precomputed light transport, Monte Carlo rendering, light transport acquisition, and view synthesis for virtual reality. Perhaps the most significant recent developments have arisen in the area of Monte Carlo image synthesis via path tracing and related methods. Indeed, one of the perennial goals of computer graphics is creating high quality images which are indistinguishable from photographs: a goal referred to as photorealism. (While we formally only require CSE 167 or an equivalent introduction to computer graphics as a pre-requisite and do provide background on modern Monte Carlo rendering, I would also encourage you to look at CSE 168 if you have not taken it. The UCSD Online public version of the course is available at: Link to Sign up for free).

In essence, one is sampling a high-dimensional function (8D for image coordinates, soft shadows on an area light, global illumination from incident directions, and depth of field on a lens), then reconstructing or denoising it to produce the final image. The industry has recently moved to physically-based path tracing as its preferred solution, with a low sample count, followed by advanced reconstruction and denoising methods. These advances have reduced the computation time by one to two orders of magnitude, enabling production rendering with physically-based global illumination, and this approach is now becoming commonplace for real-time rendering applications, arguably accelerating the use of physically-based rendering in real-time by a decade or more. This course surveys these recent developments over the past 10+ years, in many of which UCSD faculty have played a prominent role. (Note that relatively little background on Monte Carlo rendering is required, since most algorithms take the actual rendering process as a black box).

Beyond traditional rendering, sampling and reconstruction of visual appearance is critical for many other applications, such as acquisition of light transport from real-world scenes, light field cameras that capture both spatial and angular distributions of light, and real-time methods based on precomputed light transport. All of these problems involve 4D-8D functions, which are prohibitive to sample by brute force, and where coherence or sparsity needs to be exploited to enable sparse sampling and sophisticated reconstrcution. The course surveys recent advances, within the last decade, in all of these areas. In particular, we focus on problems of view synthesis or appearance acquisition, where dramatic strides have been made in the last few years.

This body of work is built on mathematical techniques, some of which are recent developments, including multi-dimensional Fourier analysis of light transport spectra, compressive sensing for sparse datasets, and deep learning for feature-based image denoising and neural radiance fields. While no prior background on these areas is required, the course will provide a brief high-level introduction, to the extent these topics are needed for the practical applications we consider. Note that almost all papers within the last 5 years on these topics have involved deep learning, and we encourage taking other deep learning courses in the department. We are hoping for this course to have access to the instructional machine learning cluster, and we may have some older GPUs or AWS credits available, please ask. Beyond this, you are on your own in terms of infrastructure if you seek to do a deep learning project. You are of course also welcome to implement or build off one of the earlier papers that does not require deep learning.

Please note that CSE 274 is a topics course, covering current topics in computer graphics. Course content may change every year or two years. This course material was first offered in Winter and Fall 2018, and you are allowed to enrol and get full credit even if you have taken CSE 274 on another occasion with me or other faculty. In fact, the course material this year has also been updated to include considerable emphasis on newer methods for view synthesis including neural radiance fields, beyond Monte Carlo denoising. The course is basically the same as last year (Fall 2021), but does include a few papers published within the past 12 months. Below are a few example images produced using some of the algorithms and systems we will be discussing.

Prerequisites

The course is targeted towards MS and PhD students with a knowledge of and interest in computer graphics (at the level of the introductory CSE 167 computer graphics course, or equivalent at another university). CSE 168 is helpful but not required (see link at the top of the page for UCSD online to sign up for the CSE 168 MOOC). We will cover some of the basis of physically-based Monte Carlo path tracing in this course for those who have not taken CSE 168, but that material is much more detailed and thorough. We encourage you to do the CSE 168 assignments until at least the basic path tracer (homework 3, and ideally homework 4) in a systematic fashion. Note however, that no prior knowledge of rendering or CSE 168 is required, and that the path tracing assignment is strictly optional/for your own benefit and not graded... you can also use an existing off-the-shelf black box path tracer (such as PBRT, Mitsuba or OptiX) for your project, focusing on sampling and reconstruction, or on other topics such as view synthesis. However, if you have not written a path tracer or taken CSE 168 before, and you do end up building the path tracer, please include a link in your final project submission, to your full resolution grader feedback results. We welcome all PhD students working in graphics, vision, and robotics. We are willing to consider students who do not completely fulfil the pre-requisites in exceptional cases, if they have a strong interest in the material. Undergraduates who have taken CSE 167 (and/or other advanced courses such as CSE 168/169) are also most welcome, space permitting (space is not usually a problem).

Relationship to Other Courses

The course is an integral part of the vision and graphics track for undergraduate, MS and PhD students (The instructor can help in getting any relevant permissions for it to count for credit towards the track as needed). It builds on the undergraduate graphics classes CSE 167 [taught by Profs. Chern/myself in the fall/winter] and CSE 168/169/190. If you like this course, you may also be interested in CSE 291s in winter and spring (this year is our strongest ever sequence of graduate graphics courses): machine learning for 3D geometry taught in the winter by Prof. Hao Su and spring courses on computational photography taught by Ben Ochoa (CSE 273), among others.

Course Format and Requirements

The course will consist of lectures on the relevant topics by the instructor, student presentations of papers covering current research in the area, and student projects. A syllabus/schedule is noted below. The grading will depend primarily on the final project. (Loosely, we note this as approximately 60% for the project, 30% for paper presentation and 10% for class participation, with the final project being the most critical aspect.) Students are expected to come to class regularly and participate in discussions, since this is an advanced graduate course.

In general, roughly (depends on the number of students in the course), 1 or 2 paper presentations will be required (if the number of students in the course exceeds the number of presentations, some may be done jointly by a group of two students). A project is not required for students taking the course S/U or P/NP; we leave it to your discretion if you want to register for 2 or 4 units in this case. An S/U registration is a good option for PhD students and others to read papers on this exciting topic and learn about the area without committing too much effort into a course project, and we encourage you to sign up. Auditors, who simply want to sit in on the course are also welcome; however, we prefer if you sign up for the course pass/fail instead [this just involves doing one or two paper presentations].

Students taking the course for a letter grade are required to do a project [this may be in groups of 2], give a presentation in class regarding their results, and also submit a final written report. Wide flexibility is available with respect to project topics, provided they relate loosely to the subject matter of the course. We expect that most projects will implement one or more of the algorithms or papers discussed in the course, showing results on sampling and reconstruction of visual appearance. We welcome suggestions from the students on alternative project ideas. The best projects will go beyond the published work in some way, such as trying out an alternative or better approach or trying to develop some variant or more general version of the technique. However, this is not essential; in general, students who fulfil all course requirements including a well-executed project will easily receive an A in the course.

As a potentially easier alternative to the project, we will also accept a well-written summary or tutorial, covering 3 or 4 papers. The best summaries will point out links between the papers not noticed by the original authors and suggest improvements or directions for future research. However, this option is recommended only as a last resort and will generally receive a slightly lower score; we prefer that you do a good project (which may involve understanding a few papers in any case).

Writing a path tracer as per CSE 168, if you have not done so already, is not explicitly graded. But if you do it as part of your project, we encourage you to include documentation in your final project (or survey if doing a survey of papers) report so we can consider it when grading. Note that this is not required and no penalty will be levied/the assignment is strictly optional. You are also welcome and encouraged to use existing path tracers (PBRT, Mitsuba, OptiX) as black boxes and focus on sampling and reconstruction, if that is most suitable for your project. Indeed, the philosophy of recent work and the course is to treat the path tracer as a black box and focus on sampling/reconstruction.

Lectures and Office Hours

The lectures will be in EBU3B (CSE Building) 4140 from 9:30-10:50 on Tuesdays and Thursdays. Students are expected to come to the lectures and participate in discussion. Office hours will be immediately after class from 11-12 in the Professor's office, EBU3B 4118. You may also e-mail for another time to meet if that is not convenient. We also have a Piazza discussion board setup for this course if you are interested. It can be accessed at: http://piazza.com/ucsd/fall2022/cse274

Teaching Assistant

We have a teaching assistant for the course, Kai-En Lin. His e-mail is k2lin@eng.ucsd.edu. His office hours will be in 4150 (CS bldg) Wednesdays and Thursdays from noon-1pm, or you can e-mail him to schedule an appointment.

Topics

Topics to be covered include

Resources

Outline

The (tentative) course schedule is as follows. In general, you will likely benefit from doing the reading (i.e. the papers assigned for a particular date) before class; it will at least make for more lively discussion.

Ravi Ramamoorthi

Last modified: Thu Oct 6 16:39:20 PDT 2022