Jay Busch - Academia.edu (original) (raw)
Papers by Jay Busch
ACM SIGGRAPH 2009 Emerging Technologies on - SIGGRAPH '09, 2009
Abstract When people communicate in person, numerous cues of attention, eye contact, and gaze dir... more Abstract When people communicate in person, numerous cues of attention, eye contact, and gaze direction provide important additional channels of information, making in-person meetings more efficient and effective than telephone conversations and 2D teleconferences. Two-dimensional video teleconferencing precludes the impression of accurate eye contact: when a participant looks into the camera, everyone seeing their video stream sees the participant looking toward them; when the participant looks away from the camera (for ...
ACM SIGGRAPH 2012 Posters on - SIGGRAPH '12, 2012
Abstract We propose a new head-mounted camera system based on stereo cell phone cameras. These ca... more Abstract We propose a new head-mounted camera system based on stereo cell phone cameras. These cameras have the advantage of being extremely small, light-weight, and programmable. We provide step-by-step details on how to recreate this apparatus and also how to apply this data to multiple applications in facial tracking and reconstruction. Our system is based on the LG Thrill, a 3D enabled cell phone that provides two synchronized stereo cameras in a tiny 4.2 gram module. We use two phones for a total of four cameras. ...
ACM SIGGRAPH 2010 Emerging Technologies on - SIGGRAPH '10, 2010
Abstract Head-mounted cameras are an increasingly important tool for capturing facial performance... more Abstract Head-mounted cameras are an increasingly important tool for capturing facial performances to drive virtual characters. They provide a fixed, unoccluded view of the face, useful for observing motion capture dots or as input to video analysis. However, the 2D imagery captured with these systems is typically affected by ambient light and generally fails to record subtle 3D shape changes as the face performs. We have developed a system that augments a head-mounted camera with LED-based photometric stereo. The system ...
We demonstrate a novel light probe which can estimate the full dynamic range of a scene with mult... more We demonstrate a novel light probe which can estimate the full dynamic range of a scene with multiple bright light sources. It places diffuse strips between mirrored spherical quadrants, effectively co-locating diffuse and mirrored probes to record the full dynamic range of illumination in a single exposure. From this image, we estimate the intensity of multiple saturated light sources by
ACM SIGGRAPH 2013 Posters on - SIGGRAPH '13, 2013
Figure 1: (Left) Three of eight high-res (0.1mm) light stage scans of the actor in static express... more Figure 1: (Left) Three of eight high-res (0.1mm) light stage scans of the actor in static expressions. (Middle) Seven-camera HD performance recording. (Right) 180Hz video-driven blendshape model with screen-space subsurface scattering and advanced eye shading effects.
Current scanning techniques record facial mesostructure with sub-millimeter precision showing por... more Current scanning techniques record facial mesostructure with sub-millimeter precision showing pores, wrinkles, and creases. However, surface roughness continues to shape specular reflection at the level of microstructure: micron scale structures. Here, we present an approach to increase the resolution of mesostructure-level facial scans using microstructure examples digitized about the face. We digitize the skin patches using polarized gradient illumination and
Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation - SCA '11, 2011
We present a semi-automatic technique for computing surface correspondences between 3D facial sca... more We present a semi-automatic technique for computing surface correspondences between 3D facial scans in different expressions, such that scan data can be mapped into a common domain for facial animation. The technique can accurately correspond high-resolution scans of widely differing expressions -without requiring intermediate pose sequences -such that they can be used, together with reflectance maps, to create high-quality blendshape-based facial animation. We optimize correspondences through a combination of Image, Shape, and Internal forces, as well as Directable forces to allow a user to interactively guide and refine the solution. Key to our method is a novel representation, called an Active Visage, that balances the advantages of both deformable templates and correspondence computation in a 2D canonical domain. We show that our semi-automatic technique achieves more robust results than automated correspondence alone, and is more precise than is practical with unaided manual input.
SIGGRAPH 2009: Talks on - SIGGRAPH '09, 2009
Abstract Capturing facial geometry that is high-resolution, yet easy to animate, remains a diffic... more Abstract Capturing facial geometry that is high-resolution, yet easy to animate, remains a difficult challenge. While a single scanned geometry may be straightforward to animate smoothly, it may not always yield realistic fine scale detail when deformed into different facial expressions. Combining scans of multiple facial expressions, however, is only practical if geometrical correspondences between the different scanned expressions are available. Correspondences obtained based on locations of facial landmarks or of placed markers ...
Computer Graphics Forum, 2013
a) Rendering using 4K (b) Rendering using 4K (c) Rendering using synthesized (d) Comparison photo... more a) Rendering using 4K (b) Rendering using 4K (c) Rendering using synthesized (d) Comparison photograph geometry geometry and microscale BRDF microstructure Figure 1: (a) Rendering with scanned mesostructure (4K displacement map). (b) Rendering with scanned mesostructure (4K displacement map) and microstructure measured BRDF. (c) Rendering with synthesized microstructure (16K displacement map). (d) Photograph under flash illumination. See Fig. 3, Subject 2 for the skin patches used in microstructure synthesis.
In this work, we develop a real-time geometry capture approach to digital face replacement for a ... more In this work, we develop a real-time geometry capture approach to digital face replacement for a dynamic performance. Digital face replacement has major applications in visual effects for motion pictures as well as interactive applications such as video games and simulation and training environments. Our approach looks into extending the 3D face scanning technology developed at the ICT Graphics Lab [Ma 2008] to support seamless face replacement along with separated diffuse and specular albedo textures and surface ...
We demonstrate a novel light probe which can estimate the full dynamic range of a scene with mult... more We demonstrate a novel light probe which can estimate the full dynamic range of a scene with multiple bright light sources. It places diffuse strips between mirrored spherical quadrants, effectively co-locating diffuse and mirrored probes to record the full dynamic range of illumination in a single exposure. From this image, we estimate the intensity of multiple saturated light sources by solving a linear system.
ACM Transactions on Graphics, 2010
Abstract We present a novel technique for acquiring detailed facial geometry of a dynamic perform... more Abstract We present a novel technique for acquiring detailed facial geometry of a dynamic performance using extended spherical gradient illumination. Key to our method is a new algorithm for jointly aligning two photographs, under a gradient illumination condition and its complement, to a full-on tracking frame, providing dense temporal correspondences under changing lighting conditions. We employ a two-step algorithm to reconstruct detailed geometry for every captured frame. In the first step, we coalesce information from the ...
ACM Transactions on Graphics, 2009
Abstract We present a set of algorithms and an associated display system capable of producing cor... more Abstract We present a set of algorithms and an associated display system capable of producing correctly rendered eye contact between a three-dimensionally transmitted remote participant and a group of observers in a 3D teleconferencing system. The participant's face is scanned in 3D at 30Hz and transmitted in real time to an autostereoscopic horizontal-parallax 3D display, displaying him or her over more than a 180 field of view observable to multiple observers. To render the geometry with correct perspective, we create a fast ...
ACM Transactions on Graphics, 2011
Abstract We present a novel process for acquiring detailed facial geometry with high resolution d... more Abstract We present a novel process for acquiring detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints using polarized spherical gradient illumination. Key to our method is a new pair of linearly polarized lighting patterns which enables multiview diffuse-specular separation under a given spherical illumination condition from just two photographs. The patterns--one following lines of latitude and one following lines of longitude--allow the use of fixed linear ...
Blendshape interpolation is one of the most successful techniques for creating emotive digital ch... more Blendshape interpolation is one of the most successful techniques for creating emotive digital characters for entertainment and simulation purposes. The quest for ever more realistic digital characters has pushed for higher resolution blendshapes. Many existing correspondence techniques, however, have difficulty establishing correspondences between very detailed blendshapes. To further aggravate the problem, in certain cases (such as pronounced wrinkles) exact 1:1
ACM SIGGRAPH 2009 Emerging Technologies on - SIGGRAPH '09, 2009
Abstract When people communicate in person, numerous cues of attention, eye contact, and gaze dir... more Abstract When people communicate in person, numerous cues of attention, eye contact, and gaze direction provide important additional channels of information, making in-person meetings more efficient and effective than telephone conversations and 2D teleconferences. Two-dimensional video teleconferencing precludes the impression of accurate eye contact: when a participant looks into the camera, everyone seeing their video stream sees the participant looking toward them; when the participant looks away from the camera (for ...
ACM SIGGRAPH 2012 Posters on - SIGGRAPH '12, 2012
Abstract We propose a new head-mounted camera system based on stereo cell phone cameras. These ca... more Abstract We propose a new head-mounted camera system based on stereo cell phone cameras. These cameras have the advantage of being extremely small, light-weight, and programmable. We provide step-by-step details on how to recreate this apparatus and also how to apply this data to multiple applications in facial tracking and reconstruction. Our system is based on the LG Thrill, a 3D enabled cell phone that provides two synchronized stereo cameras in a tiny 4.2 gram module. We use two phones for a total of four cameras. ...
ACM SIGGRAPH 2010 Emerging Technologies on - SIGGRAPH '10, 2010
Abstract Head-mounted cameras are an increasingly important tool for capturing facial performance... more Abstract Head-mounted cameras are an increasingly important tool for capturing facial performances to drive virtual characters. They provide a fixed, unoccluded view of the face, useful for observing motion capture dots or as input to video analysis. However, the 2D imagery captured with these systems is typically affected by ambient light and generally fails to record subtle 3D shape changes as the face performs. We have developed a system that augments a head-mounted camera with LED-based photometric stereo. The system ...
We demonstrate a novel light probe which can estimate the full dynamic range of a scene with mult... more We demonstrate a novel light probe which can estimate the full dynamic range of a scene with multiple bright light sources. It places diffuse strips between mirrored spherical quadrants, effectively co-locating diffuse and mirrored probes to record the full dynamic range of illumination in a single exposure. From this image, we estimate the intensity of multiple saturated light sources by
ACM SIGGRAPH 2013 Posters on - SIGGRAPH '13, 2013
Figure 1: (Left) Three of eight high-res (0.1mm) light stage scans of the actor in static express... more Figure 1: (Left) Three of eight high-res (0.1mm) light stage scans of the actor in static expressions. (Middle) Seven-camera HD performance recording. (Right) 180Hz video-driven blendshape model with screen-space subsurface scattering and advanced eye shading effects.
Current scanning techniques record facial mesostructure with sub-millimeter precision showing por... more Current scanning techniques record facial mesostructure with sub-millimeter precision showing pores, wrinkles, and creases. However, surface roughness continues to shape specular reflection at the level of microstructure: micron scale structures. Here, we present an approach to increase the resolution of mesostructure-level facial scans using microstructure examples digitized about the face. We digitize the skin patches using polarized gradient illumination and
Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation - SCA '11, 2011
We present a semi-automatic technique for computing surface correspondences between 3D facial sca... more We present a semi-automatic technique for computing surface correspondences between 3D facial scans in different expressions, such that scan data can be mapped into a common domain for facial animation. The technique can accurately correspond high-resolution scans of widely differing expressions -without requiring intermediate pose sequences -such that they can be used, together with reflectance maps, to create high-quality blendshape-based facial animation. We optimize correspondences through a combination of Image, Shape, and Internal forces, as well as Directable forces to allow a user to interactively guide and refine the solution. Key to our method is a novel representation, called an Active Visage, that balances the advantages of both deformable templates and correspondence computation in a 2D canonical domain. We show that our semi-automatic technique achieves more robust results than automated correspondence alone, and is more precise than is practical with unaided manual input.
SIGGRAPH 2009: Talks on - SIGGRAPH '09, 2009
Abstract Capturing facial geometry that is high-resolution, yet easy to animate, remains a diffic... more Abstract Capturing facial geometry that is high-resolution, yet easy to animate, remains a difficult challenge. While a single scanned geometry may be straightforward to animate smoothly, it may not always yield realistic fine scale detail when deformed into different facial expressions. Combining scans of multiple facial expressions, however, is only practical if geometrical correspondences between the different scanned expressions are available. Correspondences obtained based on locations of facial landmarks or of placed markers ...
Computer Graphics Forum, 2013
a) Rendering using 4K (b) Rendering using 4K (c) Rendering using synthesized (d) Comparison photo... more a) Rendering using 4K (b) Rendering using 4K (c) Rendering using synthesized (d) Comparison photograph geometry geometry and microscale BRDF microstructure Figure 1: (a) Rendering with scanned mesostructure (4K displacement map). (b) Rendering with scanned mesostructure (4K displacement map) and microstructure measured BRDF. (c) Rendering with synthesized microstructure (16K displacement map). (d) Photograph under flash illumination. See Fig. 3, Subject 2 for the skin patches used in microstructure synthesis.
In this work, we develop a real-time geometry capture approach to digital face replacement for a ... more In this work, we develop a real-time geometry capture approach to digital face replacement for a dynamic performance. Digital face replacement has major applications in visual effects for motion pictures as well as interactive applications such as video games and simulation and training environments. Our approach looks into extending the 3D face scanning technology developed at the ICT Graphics Lab [Ma 2008] to support seamless face replacement along with separated diffuse and specular albedo textures and surface ...
We demonstrate a novel light probe which can estimate the full dynamic range of a scene with mult... more We demonstrate a novel light probe which can estimate the full dynamic range of a scene with multiple bright light sources. It places diffuse strips between mirrored spherical quadrants, effectively co-locating diffuse and mirrored probes to record the full dynamic range of illumination in a single exposure. From this image, we estimate the intensity of multiple saturated light sources by solving a linear system.
ACM Transactions on Graphics, 2010
Abstract We present a novel technique for acquiring detailed facial geometry of a dynamic perform... more Abstract We present a novel technique for acquiring detailed facial geometry of a dynamic performance using extended spherical gradient illumination. Key to our method is a new algorithm for jointly aligning two photographs, under a gradient illumination condition and its complement, to a full-on tracking frame, providing dense temporal correspondences under changing lighting conditions. We employ a two-step algorithm to reconstruct detailed geometry for every captured frame. In the first step, we coalesce information from the ...
ACM Transactions on Graphics, 2009
Abstract We present a set of algorithms and an associated display system capable of producing cor... more Abstract We present a set of algorithms and an associated display system capable of producing correctly rendered eye contact between a three-dimensionally transmitted remote participant and a group of observers in a 3D teleconferencing system. The participant's face is scanned in 3D at 30Hz and transmitted in real time to an autostereoscopic horizontal-parallax 3D display, displaying him or her over more than a 180 field of view observable to multiple observers. To render the geometry with correct perspective, we create a fast ...
ACM Transactions on Graphics, 2011
Abstract We present a novel process for acquiring detailed facial geometry with high resolution d... more Abstract We present a novel process for acquiring detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints using polarized spherical gradient illumination. Key to our method is a new pair of linearly polarized lighting patterns which enables multiview diffuse-specular separation under a given spherical illumination condition from just two photographs. The patterns--one following lines of latitude and one following lines of longitude--allow the use of fixed linear ...
Blendshape interpolation is one of the most successful techniques for creating emotive digital ch... more Blendshape interpolation is one of the most successful techniques for creating emotive digital characters for entertainment and simulation purposes. The quest for ever more realistic digital characters has pushed for higher resolution blendshapes. Many existing correspondence techniques, however, have difficulty establishing correspondences between very detailed blendshapes. To further aggravate the problem, in certain cases (such as pronounced wrinkles) exact 1:1