(original) (raw)

Stanford CS348K, Spring 2022

VISUAL COMPUTING SYSTEMS

Visual computing tasks such as computational imaging, image/video understanding, and real-time 3D graphics are key responsibilities of modern computer systems ranging from sensor-rich smart phones, autonomous robots, and large datacenters. These workloads demand exceptional system efficiency and this course examines the key ideas, techniques, and challenges associated with the design of parallel, heterogeneous systems that accelerate visual computing applications. This course is intended for systems students interested in architecting efficient graphics, image processing, and computer vision platforms (both new hardware architectures and domain-optimized programming frameworks for these platforms) and for graphics, vision, and machine learning students that wish to understand throughput computing principles to design new algorithms that map efficiently to these machines.

Basic Info

Tues/Thurs 1:30-3:00pm

Location: 540-108

Welcome to CS348K Spring 2022. Please see the course info page for more info on policies and logistics, and well as answers to common questions like "Am I prepared to take this class?" This course is a paper-reading and in-class discussion-based course, so live attendence is expected of all participants.

Spring 2022 Schedule

| Mar 29 | | Introduction + Throughput Computing Review Review of multi-core, multi-threading, SIMD, caches, and the value of hardware specialization | | ------ | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Mar 31 | | Digital Camera Processing Pipeline (Part I) Algorithms for taking raw sensor pixels to an RGB image: demosaicing, sharpening, correcting lens aberrations, multi-shot alignment/merging, image filtering | | Apr 05 | | Digital Camera Processing Pipeline (Part II) Multi-scale processing with Gaussian and Laplacian pyramids, HDR (local tone mapping), portrait mode | | Apr 07 | | Digital Camera Processing Pipeline (Part III) + Camera Processing Abstractions Autofocus, autoexposure, use of ML in advanced camera operations, the Frankencamera. | | Apr 12 | | Scheduling Image Processing Algorithms to Parallel Hardware Balancing locality, parallelism, and work, fusion and tiling, design of the Halide domain-specific language, automatically scheduling image processing pipelines | | Apr 14 | | Efficient DNN Inference Popular DNN trunks and topologies, where the compute lies in modern networks, data layout optimizations, scheduling decisions, modern code generation frameworks | | Apr 19 | | DNN Hardware Accelerators GPUs, TPUs, special instructions for DNN evaluation (and their efficiency vs custom ASIC), choice of precision in arithmetic, modern commercial DNN accelerators, flexibility vs efficiency trade-offs | | Apr 21 | | (The Lack of) System Support for Generating Supervision If the most important step of ML is acquiring labeled data for training and validation, why don't we have better systems for it? | | Apr 26 | | Raising the Level of Abstraction for Building ML Models Systems for specifying models at a higher level of abstraction than DNN architecture graphs (Overton, Ludwig). Goal: removing the need for a low-level ML engineer. | | Apr 28 | | Video Compression: Traditional and Learned H.264 video representation/encoding, parallel encoding, motivations for ASIC acceleration, ML-based compression methods, emerging opportunities for compression when machines, not humans, will observe most images | | May 03 | | The Present and Future of Videoconferencing Systems System design issues for building a video conferencing system: reducing latency, bandwidth, etc. How real-time video analysis will enable richer video-based applications. | | May 05 | | The Light Field (NeRF Preliminaries) The light field, initial discussion of NeRF algorithms | | May 10 | | NeRF Training and Rendering + Real-time Rendering Algorithms Review Discussion of the render arc of NeRF papers + review of the 3D rasterization pipeline (so we can talk about performance challenges next.) | | May 12 | | Scheduling the 3D Graphics Pipeline Scheduling graphics pipeline onto parallel GPUs, key optimizations for modern, power-optimized mobile GPUs. | | May 17 | | Hardware Accelerated Real-Time Ray Tracing Motivations of ray tracing, ray tracing workload characteristics, memory coherence and SIMD execution challenges of ray tracing, modern hardware acceleration on NVIDIA RTX GPUs) | | May 19 | | Real Time Ray Tracing 2 Efficient BVH construction, hardware-accelerated ray tracing APIs, neural denoising and post-processing techniques. | | May 24 | | Epic’s Nanite Renderer (Guest Lecture: Brian Karis - Epic Games) Brian Karis will talk about the design of Epic's Nanite renderer (efficient real-time rendering of very high-resolution geometry) | | May 26 | | Rendering and Simulation for Model Training How might systems for rendering and simulating virtual worlds be architected differently to support the needs of training machines instead of playing video games? (a.k.a. rendering for machine eyes, not human eyes) | | May 31 | | The Slang Shading Language (Guest Lecture: Yong He and Teresa Foley - NVIDIA) The design and implementation of Slang, discussion about transferring academic systems research into industry efforts. | | Jun 02 | | Compute Challenges at Waymo (Guest Lecture: Daniel Rosenband and Solomon Boulos - Waymo) On-vehicle sensing technologies and CPU/GPU/ML accelerator compute challenges in the real-time AV setting, challenges of AV simulation, such as accuracy vs. compute cost tradeoffs. |

Assignments

In addition to expectation that all students attend and participate in discussions in live lecture, there will be two short programming assignments and a self-selected term project.