772: Real-Time Computer Graphics (original) (raw)

Also known as 15-472/672/772. Tuesdays and Thursdays from 14:00-15:20 in Posner Hall 151 during Spring 2024. Taught by Jim McCann. Office hours after class in Smith 229 or by appointment.

Real-time computer graphics is about building systems that leverage modern CPUs and GPUs to produce detailed, interactive, immersive, and high-frame-rate imagery. Students will build a state-of-the-art renderer using C++ and the Vulkan API. Topics explored will include efficient data handling strategies; culling and scene traversal; multi-threaded rendering; post-processing, depth of field, screen-space reflections; volumetric rendering; sample distribution, spatial and temporal sharing, and anti-aliasing; stereo view synthesis; physical simulation and collision detection; dynamic lights and shadows; global illumination, accelerated raytracing; dynamic resolution, "AI" upsampling; compute shaders; parallax occlusion mapping; tessellation, displacement; skinning, transform feedback; and debugging, profiling, and accelerating graphics algorithms.

Principles

This course is based on four principles:

Do Things for Reasons -- if you don't understand why you are doing something, look deeper. (Corr: if you don't understand why someone else is doing something, ask.)
No Magic -- (as much as possible) avoid black boxes and needless helper libraries.
Test and Improve -- make things efficient through real-world testing. (Corr: asymptotic complexity is not the only factor; constants matter in the real world.)
Go Big -- push systems to their limits.

Work

(Note: subject to revision based on class timing.)

Your grade in this course will be determined by four assignments that cumulatively build a high-performance real-time renderer (60%); a project that adds a state-of-the-art feature to that renderer (30%); and participation in in-class discussions and activities (10%).

Don't Copy and Don't Steal

You will write a lot of code in this class, and I require that you actually type this code. (It is easy to copy-paste faster than you can think.)

You may use code from the internet, but make sure the code is not covered by a license that prohibits copying (copyright violation -- illegal!), be sure to include a comment citing your source (plagiarism -- immoral!), and make sure it passes through your brain on its way to your fingers (e.g., "fix" the coding style, re-work awkward parts, look up and document parts you don't understand).

You may work together, but take extra care to not copy code. Rule of thumb: if you are writing code, you should not be doing it where you can see your classmates' code.

Examples:

GOOD: re-writing a documentation example in your own coding style (including a comment + URL citing your source).
BAD: copy-pasting a stackoverflow answer or code snippet from the documentation, then re-working it in place. (You must type the code you use.)
GOOD: working with a classmate to develop an design, then writing the code separately (and including a comment crediting the discussion in your code).
GOOD: helping a classmate to debug their code, being inspired by their approach, and -- with your classmate's permission and a citation comment -- using a similar idea in your code.
BAD: using copilot. (You aren't typing and you aren't thinking.)
BAD: ChatGPT and similar. (Just bad in general. A copyright [or at least plagiarism] laundry.)
GOOD: copy-pasting code you already wrote for this class.
DISCOURAGED (but not forbidden): using autocomplete / codesense.

Resources

This course does not have a textbook; but you will likely find the following web resources helpful:

C++ Reference
Compiler Explorer -- for playing with how code might get optimized
MDN's JavaScript reference
Node.js documentation
Blender Python API
Vulkan
- Vulkan Specification
- Vulkan Guide (easier reading than the spec!)
- Vulkan Tutorial (you will work through this for A0)
- (something about SPIR-V?)
Inspiration
- Pouët -- a demoscene archive
- Shadertoy -- fun fragment shader hacks
Advances in Real-Time Rendering -- notes from SIGGRAPH courses
GPU Gems -- the classic GPU programming books (now free!)

See also: the "extra readings" links in each section below.

Schedule

(Perpetually subject to revision -- and, at present, still very much a draft!)

Real-Time Graphics

Tuesday, January 16th

What it means to be real-time and general techniques for getting there.

T Jan 16
Course Overview; Core Principles
Interactive, Real-Time, and Beyond
Big Ideas
CPU Benchmarking and Fast Code Intuition
A0a Timing, Math, and Intrinsics

The Graphics Pipeline

Thursday, January 18th
-Thursday, January 25th

How we get pixels onto the screen quickly. From attributes to framebuffers.

R Jan 18
Vulkan Intro
A Picture of Vulkan
Uniforms, Attributes, Vertices, Primitives, Framebuffers
Opening a Window
A0b Vulkan Tutorial
T Jan 23
Framework Style
Vulkan Debugging Session
A0a Scoreboard Check-In
R Jan 25
Javascript
Build systems

Data Wrangling

Tuesday, January 30th Image from the paper 'Bisection Based Triangulation of Catmull Clark Subdivision' by Dupuy and Deloit.

Getting data into our renderer.

T Jan 30
Scene graphs
Instancing
Data inflation: subdivision surfaces
Sub-polygon detail: textures, normals, displacements
Mesh compression?
Texture compression
least squares vertex baking(?)
Creating (and finding) Bottlenecks
A1 statement walkthrough
Export scripts
A1 Scene Viewer out

Culling and Occlusion

Thursday, February 1st
-Thursday, February 8th Image from the talk 'GPU-Driven Rendering Pipelines' by Haar.

What you don't render can't slow you down.

R Feb 1
Frustum culling
Portal rendering
Level-of-detail
Occlusion queries
Geometry images
T Feb 6
Bottlenecks
R Feb 8: No Class / A1 Work Time

Materials

Tuesday, February 13th
-Thursday, February 22nd Illustration from SIGGRAPH Real-Time Rendering course about UE5's 'substrate' materials system

Matching the real world, or doing better.

T Feb 13
detail with textures
R Feb 15
physical materials
procedural materials
approximations and ground truth
material capture
antialiasing
glossy surfaces
precomputation
A2 Materials out
T Feb 20
Interesting Mesh Ideas
A1 Scene Viewer due (before class)
R Feb 22
A1 debrief + Code Share

Color

Tuesday, February 27th placeholder image for colorspace lecture.

Primaries, white points, and encodings oh my.

T Feb 27
color spaces
demo: P3 vs sRGB primaries
demo: HDR
vulkan formats and color spaces

Going Big

Thursday, February 29th
-Saturday, March 9th Megatextures visualized in id Software's 'Rage'.

Moving beyond what can fit in GPU memory.

R Feb 29
Mega-textures
Terrains
LOD streaming
T Mar 5: Spring Break
R Mar 7: Spring Break
S Mar 9
A2 Materials due (end of day)

Direct Lighting

Tuesday, March 12th Image from the talk 'The Light Pre-Pass Renderer: Renderer Design for Efficient Support of Multiple Lights' by Engel

The first bounce of lighting is always the brightest.

T Mar 12
sphere, spot, and sun lights
lambertian contribution from a spherical area light
the "representative point" approximation
light loops

Shadows

Thursday, March 14th
-Friday, March 15th Illustration from 'Compact Precomputed Voxelized Shadows' Sintorn, et al. 2014

Solving the visibility problem.

R Mar 14
shadow maps
cubic shadow maps
percentage-closer filtering
soft shadows with PCSS
F Mar 15
A3 Lights out

Real-time Raytracing

Tuesday, March 19th Image from the talk 'Real-Time rendering’s next frontier: Adopting lessons from offline ray tracing to real-time ray tracing for practical pipelines' by Pharr.