(original) (raw)

Stanford CS348K, Spring 2021

VISUAL COMPUTING SYSTEMS

Visual computing tasks such as computational imaging, image/video understanding, and real-time 3D graphics are key responsibilities of modern computer systems ranging from sensor-rich smart phones, autonomous robots, and large datacenters. These workloads demand exceptional system efficiency and this course examines the key ideas, techniques, and challenges associated with the design of parallel, heterogeneous systems that accelerate visual computing applications. This course is intended for systems students interested in architecting efficient graphics, image processing, and computer vision platforms (both new hardware architectures and domain-optimized programming frameworks for these platforms) and for graphics, vision, and machine learning students that wish to understand throughput computing principles to design new algorithms that map efficiently to these machines.

Basic Info

Tues/Thurs 2:30-3:50pm

Virtual Course Only (held on Ohyay, see Piazza for details)

Spring 2021 Schedule

| Mar 30 | | Course Introduction + Computer Architecture Review Review superscalar, multi-core, SIMD, and multi-threaded CPU/GPU designs, + understanding latency and bandwidth | | ------ | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Apr 01 | | Digital Camera Processing Pipeline (Part I) Algorithms for taking raw sensor pixels to an RGB image: demosaicing, sharpening, correcting lens aberrations, multi-shot alignment/merging, image filtering | | Apr 06 | | Digital Camera Processing Pipeline (Part II) Multi-scale processing with Gaussian and Laplacian pyramids, HDR (local tone mapping), portrait mode | | Apr 08 | | Digital Camera Processing Pipeline (Part III) Finishing and reviewing concepts from prior lectures (see prior slides) | | Apr 13 | | Scheduling Image Processing Algorithms to Parallel Hardware Balancing locality, parallelism, and work, fusion and tiling, design of the Halide domain-specific language, automatically scheduling image processing pipelines | | Apr 15 | | Efficient DNN Inference (for Image Analysis) Popular DNN trunks and topologies, efficient topologies like MobileNet, where the compute lies in modern networks, DNN pruning, neural architecture search | | Apr 20 | | Hardware Accelerators for DNN Workloads GPUs, TPUs, special instructions for DNN evaluation (and their efficiency vs custom ASIC), choice of precision in arithmetic, recent ISCA/MICRO papers on DNN acceleration, flexibility vs efficiency trade-offs | | Apr 22 | | Parallel DNN Training at Scale Footprint challenges of training, model vs. data parallelism, asynchronous vs. synchronous training debate, parameter server designs, key systems optimizations for parallel training | | Apr 27 | | (The Lack of) System Support for Generating Supervision If the most important step of ML is acquiring labeled data for training and validation, why don't we have better systems for it? | | Apr 29 | | Raising the Level of Abstraction in Machine Learning Systems for specifying models at a higher level of abstraction than DNN architecture graphs (Overton, Ludwig). Goal: removing the need for a low-level ML engineer. | | May 04 | | Specializing Models to Video Streams (for Accuracy and Efficiency) Exploiting temporal coherence in video, specialization to camera viewpoint, scene appearance, and task. | | May 06 | | Video Compression (Traditional and Learned) H.264 video representation/encoding, parallel encoding, motivations for ASIC acceleration, emerging opportunities for compression when machines, not humans, will observe most images | | May 11 | | Video Conferencing Systems System design issues for building a video conferencing system: reducing latency, bandwidth, etc. | | May 13 | | The Real-Time Ray Tracing Workload Motivations for ray tracing vs. rasterization. Ray coherence during traversal, parallelizing BVH build, multi-level BVHs. | | May 18 | | Real-Time Ray Tracing Systems (HW Acceleration + Neural Denoising) Modern hardware acceleration (RTX GPUs), conversing noisy images to clean images using neural techniques. | | May 20 | | Real-Time Ray Tracing Systems (Part II) Rendering realistic scenes with many lights | | May 25 | | Rendering and Simulation for Training Agents How might systems for rendering and simulating virtual worlds be architected differently to support the needs of training machines instead of video games? (a.k.a. rendering for machine eyes, not human eyes) | | May 27 | | Rendering for Virtual Reality Key rendering issues for VR and AR devices | | Jun 01 | | Guest Lecture (Chris Wyman, NVIDIA) algorithmic innovation (and future challenges) for real-time ray tracing | | Jun 03 | | Project Presentations student project presentations |

Assignments

In addition to expectation that all students attend and participate in discussions in live lecture, there will be two short programming assignments and a self-selected term project.