Sang-won Leigh | Massachusetts Institute of Technology (MIT) (original) (raw)

Papers by Sang-won Leigh

Learning advanced skills on a musical instrument takes a range of physical and cognitive efforts.... more Learning advanced skills on a musical instrument takes a range of physical and cognitive efforts. For instance, practicing polyrhythm is a complex task that requires the development of both musical and physical skills. This paper explores the use of automation in the context of learning advanced skills on the guitar. Our robotic guitar is capable of physically plucking on the strings along with a musician, providing both haptic and audio guidance to the musician. We hypothesize that a multimodal and firstperson experience of "being able to play" could increase learning efficacy. We discuss the novel learning application and a user study, through which we illustrate the implication and potential issues in systems that provide temporary skills and in-situ multimodal guidance for learning.

Thesis: Ph. D., Massachusetts Institute of Technology, School of Architecture and Planning, Progr... more Thesis: Ph. D., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2018.Cataloged from PDF version of thesis. Page 170 blank.Includes bibliographical references (pages 157-169).Throughout history we have augmented our physical abilities with machines. Concepts for flying machines and the ideas behind today's exoskeletons were recorded as early as the 13th century. Today, as technology permeates every aspect of our lives, it is easy to imagine a much closer integration of machines into the tasks we carry out. This thesis explores a vision of humans and machines symbiotically working together on a task through co-action and coagency. This vision opens up many opportunities in between the extremes of autonomous robots and master-slave systems, through more complex systems in which human and machine collaborate to perform actions and manipulate robotic extensions. This dissertation also reports on three extensive experimen...

Augmented Humans 2022

Figure 1: We argue that eliciting a sense of self allows for user experiences such as "I did that... more Figure 1: We argue that eliciting a sense of self allows for user experiences such as "I did that (with my body)", going beyond "I made the machine do that (for me)" and "The machine did that (for me)", which is often the current approach when it comes to physical integration of computer systems.

Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction, 2022

This paper explores a novel tool and materials for brush-like expression in 3D forms. We envision... more This paper explores a novel tool and materials for brush-like expression in 3D forms. We envision a 2.5D painting device that aims at closing the gap between painterly expression, digital fabrication, and the incorporation of expressive elements present in traditional painting with structural capability. The paper describes the development of the new pigment material, a custom handheld extrusion device, and discusses a user study investigating emerging relationships with the developed medium. We find the reactivity and variability of the medium enabling strong expressive agency and new interaction opportunities. With these, we discuss what are aspects in making that could be embraced to enhance users' creative agency and ownership in making, postulating a space between artistic expression and utilitarian making-e.g. digital fabrication.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14, 2015

Remnance of Form is an interactive installation that explores the dynamic tension between an obje... more Remnance of Form is an interactive installation that explores the dynamic tension between an object and its shadow. By fusing light, projection, and motion technologies, the shadow can now detach itself from its former role. This creates a new narrative that challenges our perception of reality, what's real and what's not.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, 2015

The huge influx of mobile display devices is transforming computing into multi-device interaction... more The huge influx of mobile display devices is transforming computing into multi-device interaction, demanding a fluid mechanism for using multiple devices in synergy. In this paper, we present a novel interaction system that allows a collocated large display and a small handheld device to work together. The smartphone acts as a physical interface for near-surface interactions on a computer screen. Our system enables accurate position tracking of a smartphone placed on or over any screen by displaying a 2D color pattern that is captured using the smartphone's back-facing camera. As a result, the smartphone can directly interact with data displayed on the host computer, with precisely aligned visual feedback from both devices. The possible interactions are described and classified in a framework, which we exemplify on the basis of several implemented applications. Finally, we present a technical evaluation and describe how our system is unique compared to other existing near-surface interaction systems. The proposed technique can be implemented on existing devices without the need for additional hardware, promising immediate integration into existing systems.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, 2015

clayodor (\klei-o-dor\) is a clay-like malleable material that changes smell based on user manipu... more clayodor (\klei-o-dor\) is a clay-like malleable material that changes smell based on user manipulation of its shape. This work explores the tangibility of shape changing materials to capture smell, an ephemeral and intangible sensory input. We present the design of a proof-of-concept prototype, and discussions on the challenges of navigating smell though form.

We present GTGraffiti, a graffiti painting system from Georgia Tech that tackles challenges in ar... more We present GTGraffiti, a graffiti painting system from Georgia Tech that tackles challenges in art, hardware, and human-robot collaboration. The problem of painting graffiti in a human style is particularly challenging and requires a system-level approach because the robotics and art must be designed around each other. The robot must be highly dynamic over a large workspace while the artist must work within the robot’s limitations. Our approach consists of three stages: artwork capture, robot hardware, and planning & control. We use motion capture to capture collaborator painting motions which are then composed and processed into a time-varying linear feedback controller for a cable-driven parallel robot (CDPR) to execute. In this work, we will describe the capturing process, the design and construction of a purpose-built CDPR, and the software for turning an artist’s vision into control commands. Our work represents an important step towards faithfully recreating human graffiti art...

Cord UIs are sensorial augmented cords that allow for simple metaphor-rich interactions to interf... more Cord UIs are sensorial augmented cords that allow for simple metaphor-rich interactions to interface with their connected devices. Cords offer a large underexplored space for interactions as well as unique properties and a diverse set of metaphors that make them potentially interesting tangible interfaces. We use cords as input devices and explore different interactions like tying knots, stretching, pinching and kinking to control the flow of data and/or power. We also look at ways to use objects in combination with augmented cords to manipulate data or properties of a device. For instance, placing a clamp on a cable can obstruct the audio signal to the headphones. Using special materials such as piezo copolymer cables and stretchable cords we built five working prototypes to showcase the interactions described in this paper.

Interactions, 2015

The Arts Track at TEI 2015 explored sensory connections among people and the world around them. A... more The Arts Track at TEI 2015 explored sensory connections among people and the world around them. An international jury of artists, designers, and educators selected evocative, thought-provoking, and experientially rich pieces ranging from full-room installations to wearables and performance-based art. The pieces selected for the exhibit demonstrate not only technological novelty but also formal sophistication, conceptual creativity, and engagement with issues of active concern to the fields of interactive art. Here, we present a small sample demonstrating the breadth of the exhibition. Please see http://www.tei-conf.org/15/artists/ for the full list. Elizabeth Goodman and Younghui Kim, TEI'15 Arts Track Chairs

Companion Publication of the 2020 ACM Designing Interactive Systems Conference, 2020

Latent Sheep Dreaming is a physical-digital installation exploring cohesion and conflict between ... more Latent Sheep Dreaming is a physical-digital installation exploring cohesion and conflict between human and machine agents in generative processes. With a stylus-based console, users sketch sheep captured by the machine and reimagined through rule- and neural network-based generative algorithms. The machine's imagination process is projected on a display wall and physicalized via a pen plotter, documenting imagined shapes that emerge from user and machine collaboration.

This paper introduces studies conducted with musicians that aim to understand modes of human-robo... more This paper introduces studies conducted with musicians that aim to understand modes of human-robot interaction, situated between automation and human augmentation. Our robotic guitar system used for the study consists of various sound generating mechanisms, either driven by software or by a musician directly. The control mechanism allows the musician to have a varying degree of agency over the overall musical direction. We present interviews and discussions on open-ended experiments conducted with music students and musicians. The outcome of this research includes new modes of playing the guitar given the robotic capabilities, and an understanding of how automation can be integrated into instrument-playing processes. The results present insights into how a human-machine hybrid system can increase the efficacy of training or exploration, without compromising human engagement with a task.

In this paper we present Cord UI, a novel user interface that utilizes cords that are connected t... more In this paper we present Cord UI, a novel user interface that utilizes cords that are connected to our everyday electrical devices. Cords offer unique properties and a diverse set of metaphors that make them potentially interesting tangible interfaces. Cord UI explores different interactions like tying knots or pinching/sliding to control the flow of data and/or electricity. We also look at ways to manipulate data or certain properties of a device by using a combination of the cord and some object. For instance, placing a clamp on a cable can obstruct the audio signal to the headphones. To test and evaluate our ideas, we built prototypes by augmenting standard cables with different sensors. We think that these augmented cords can make certain controls more seamless and intuitive.

Tools and user interfaces are usually designed for users with normal physical condition. Even tho... more Tools and user interfaces are usually designed for users with normal physical condition. Even though more products are built with accessibility considerations, there is still severe lack of opportunity for people with disabilities to use products to their full potential. In this paper, we present the idea of using human muscles as output interface, using functional electrical stimulation (FES). We envision that this would allow for filling the gap between products and a user’s capability, thereby promising every product’s universal accessibility. To match our behavioral and perceptual capability with the product on use, we can reprogram the way we control our body or give direct muscle motion-feedback. We believe that this idea would create new area and provoke discussion in the field of user interactions and their applications.

Playing musical instruments involves producing gradually more challenging body movements and tran... more Playing musical instruments involves producing gradually more challenging body movements and transitions, where the kinematic constraints of the body play a crucial role in structuring the resulting music. We seek to make a bridge between currently accessible motor patterns, and musical possibilities beyond those – afforded through the use of a robotic augmentation. Guitar Machine is a robotic device that presses on guitar strings and assists a musician by fretting alongside her on the same guitar. This paper discusses the design of the system, strategies for using the system to create novel musical patterns, and a user study that looks at the effects of the temporary acquisition of enhanced physical ability. Our results indicate that the proposed human-robot interaction would equip users to explore new musical avenues on the guitar, as well as provide an enhanced understanding of the task at hand on the basis of the robotically acquired ability.

Through several vignettes - Disappearing, Showing Fear, Many Faces, and Dream of Flying, a transf... more Through several vignettes - Disappearing, Showing Fear, Many Faces, and Dream of Flying, a transforming shadow deconstructs the notion of materialistic view of the physical world as well as offers to become a surrogate for channelling the images and imaginations about one's own self.

ArXiv, 2021

We present a system that paints graffiti art with applications in art preservation, human-robot c... more We present a system that paints graffiti art with applications in art preservation, human-robot collaboration, and other tasks which require large-scale dynamic motion. The problem of painting graffiti in a human style is particularly challenging and requires a systems approach because the art and robotics must be designed around each other. Our approach consists of three stages: artwork capture, robot hardware, and robot planning and control. We use motion capture to capture collaborator painting motions which are then composed and processed into a time-varying linear feedback controller for a cable-driven parallel robot (CDPR) to execute. In this work, we will describe the capturing process, the design and construction of a purpose-built CDPR, and the software for turning an artist’s vision into control commands. While we acknowledge that further work is needed for a system which can perfectly recreate human artwork, our results represent a contribution by demonstrating that we ca...

Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

A close integration of human and machine has been envisioned by researchers and artists for gener... more A close integration of human and machine has been envisioned by researchers and artists for generations, however, there has been little effort in investigating the possibility and plausibility of the idea until recent years. We seek to open a discussion on how the notion of self is plastic, and how innately we are capable of empowering and extending ourselves through technologies. Neuroscience studies show using a tool not only offers new capabilities, but also reconstructs our cognitive architecture to include the tool as a part of ourselves. This adaptive nature of the body image poses an opportunity for designing interfaces that become natural extensions to us. In this paper we introduce previous studies drawn from various fields of study, and discuss the role of the body in the self-world relationship, body image plasticity, and how designing the body may affect neural developments. We also offer a categorization of related technologies, along with our current explorations. Finally, potential issues and challenges in realizing the presented form of interfaces are addressed.

Learning advanced skills on a musical instrument takes a range of physical and cognitive efforts.... more Learning advanced skills on a musical instrument takes a range of physical and cognitive efforts. For instance, practicing polyrhythm is a complex task that requires the development of both musical and physical skills. This paper explores the use of automation in the context of learning advanced skills on the guitar. Our robotic guitar is capable of physically plucking on the strings along with a musician, providing both haptic and audio guidance to the musician. We hypothesize that a multimodal and firstperson experience of "being able to play" could increase learning efficacy. We discuss the novel learning application and a user study, through which we illustrate the implication and potential issues in systems that provide temporary skills and in-situ multimodal guidance for learning.

Thesis: Ph. D., Massachusetts Institute of Technology, School of Architecture and Planning, Progr... more Thesis: Ph. D., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2018.Cataloged from PDF version of thesis. Page 170 blank.Includes bibliographical references (pages 157-169).Throughout history we have augmented our physical abilities with machines. Concepts for flying machines and the ideas behind today's exoskeletons were recorded as early as the 13th century. Today, as technology permeates every aspect of our lives, it is easy to imagine a much closer integration of machines into the tasks we carry out. This thesis explores a vision of humans and machines symbiotically working together on a task through co-action and coagency. This vision opens up many opportunities in between the extremes of autonomous robots and master-slave systems, through more complex systems in which human and machine collaborate to perform actions and manipulate robotic extensions. This dissertation also reports on three extensive experimen...

Augmented Humans 2022

Figure 1: We argue that eliciting a sense of self allows for user experiences such as "I did that... more Figure 1: We argue that eliciting a sense of self allows for user experiences such as "I did that (with my body)", going beyond "I made the machine do that (for me)" and "The machine did that (for me)", which is often the current approach when it comes to physical integration of computer systems.

Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction, 2022

This paper explores a novel tool and materials for brush-like expression in 3D forms. We envision... more This paper explores a novel tool and materials for brush-like expression in 3D forms. We envision a 2.5D painting device that aims at closing the gap between painterly expression, digital fabrication, and the incorporation of expressive elements present in traditional painting with structural capability. The paper describes the development of the new pigment material, a custom handheld extrusion device, and discusses a user study investigating emerging relationships with the developed medium. We find the reactivity and variability of the medium enabling strong expressive agency and new interaction opportunities. With these, we discuss what are aspects in making that could be embraced to enhance users' creative agency and ownership in making, postulating a space between artistic expression and utilitarian making-e.g. digital fabrication.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14, 2015

Remnance of Form is an interactive installation that explores the dynamic tension between an obje... more Remnance of Form is an interactive installation that explores the dynamic tension between an object and its shadow. By fusing light, projection, and motion technologies, the shadow can now detach itself from its former role. This creates a new narrative that challenges our perception of reality, what's real and what's not.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, 2015

The huge influx of mobile display devices is transforming computing into multi-device interaction... more The huge influx of mobile display devices is transforming computing into multi-device interaction, demanding a fluid mechanism for using multiple devices in synergy. In this paper, we present a novel interaction system that allows a collocated large display and a small handheld device to work together. The smartphone acts as a physical interface for near-surface interactions on a computer screen. Our system enables accurate position tracking of a smartphone placed on or over any screen by displaying a 2D color pattern that is captured using the smartphone's back-facing camera. As a result, the smartphone can directly interact with data displayed on the host computer, with precisely aligned visual feedback from both devices. The possible interactions are described and classified in a framework, which we exemplify on the basis of several implemented applications. Finally, we present a technical evaluation and describe how our system is unique compared to other existing near-surface interaction systems. The proposed technique can be implemented on existing devices without the need for additional hardware, promising immediate integration into existing systems.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, 2015

clayodor (\klei-o-dor\) is a clay-like malleable material that changes smell based on user manipu... more clayodor (\klei-o-dor\) is a clay-like malleable material that changes smell based on user manipulation of its shape. This work explores the tangibility of shape changing materials to capture smell, an ephemeral and intangible sensory input. We present the design of a proof-of-concept prototype, and discussions on the challenges of navigating smell though form.

We present GTGraffiti, a graffiti painting system from Georgia Tech that tackles challenges in ar... more We present GTGraffiti, a graffiti painting system from Georgia Tech that tackles challenges in art, hardware, and human-robot collaboration. The problem of painting graffiti in a human style is particularly challenging and requires a system-level approach because the robotics and art must be designed around each other. The robot must be highly dynamic over a large workspace while the artist must work within the robot’s limitations. Our approach consists of three stages: artwork capture, robot hardware, and planning & control. We use motion capture to capture collaborator painting motions which are then composed and processed into a time-varying linear feedback controller for a cable-driven parallel robot (CDPR) to execute. In this work, we will describe the capturing process, the design and construction of a purpose-built CDPR, and the software for turning an artist’s vision into control commands. Our work represents an important step towards faithfully recreating human graffiti art...

Cord UIs are sensorial augmented cords that allow for simple metaphor-rich interactions to interf... more Cord UIs are sensorial augmented cords that allow for simple metaphor-rich interactions to interface with their connected devices. Cords offer a large underexplored space for interactions as well as unique properties and a diverse set of metaphors that make them potentially interesting tangible interfaces. We use cords as input devices and explore different interactions like tying knots, stretching, pinching and kinking to control the flow of data and/or power. We also look at ways to use objects in combination with augmented cords to manipulate data or properties of a device. For instance, placing a clamp on a cable can obstruct the audio signal to the headphones. Using special materials such as piezo copolymer cables and stretchable cords we built five working prototypes to showcase the interactions described in this paper.

Interactions, 2015

The Arts Track at TEI 2015 explored sensory connections among people and the world around them. A... more The Arts Track at TEI 2015 explored sensory connections among people and the world around them. An international jury of artists, designers, and educators selected evocative, thought-provoking, and experientially rich pieces ranging from full-room installations to wearables and performance-based art. The pieces selected for the exhibit demonstrate not only technological novelty but also formal sophistication, conceptual creativity, and engagement with issues of active concern to the fields of interactive art. Here, we present a small sample demonstrating the breadth of the exhibition. Please see http://www.tei-conf.org/15/artists/ for the full list. Elizabeth Goodman and Younghui Kim, TEI'15 Arts Track Chairs

Companion Publication of the 2020 ACM Designing Interactive Systems Conference, 2020

Latent Sheep Dreaming is a physical-digital installation exploring cohesion and conflict between ... more Latent Sheep Dreaming is a physical-digital installation exploring cohesion and conflict between human and machine agents in generative processes. With a stylus-based console, users sketch sheep captured by the machine and reimagined through rule- and neural network-based generative algorithms. The machine's imagination process is projected on a display wall and physicalized via a pen plotter, documenting imagined shapes that emerge from user and machine collaboration.

This paper introduces studies conducted with musicians that aim to understand modes of human-robo... more This paper introduces studies conducted with musicians that aim to understand modes of human-robot interaction, situated between automation and human augmentation. Our robotic guitar system used for the study consists of various sound generating mechanisms, either driven by software or by a musician directly. The control mechanism allows the musician to have a varying degree of agency over the overall musical direction. We present interviews and discussions on open-ended experiments conducted with music students and musicians. The outcome of this research includes new modes of playing the guitar given the robotic capabilities, and an understanding of how automation can be integrated into instrument-playing processes. The results present insights into how a human-machine hybrid system can increase the efficacy of training or exploration, without compromising human engagement with a task.

In this paper we present Cord UI, a novel user interface that utilizes cords that are connected t... more In this paper we present Cord UI, a novel user interface that utilizes cords that are connected to our everyday electrical devices. Cords offer unique properties and a diverse set of metaphors that make them potentially interesting tangible interfaces. Cord UI explores different interactions like tying knots or pinching/sliding to control the flow of data and/or electricity. We also look at ways to manipulate data or certain properties of a device by using a combination of the cord and some object. For instance, placing a clamp on a cable can obstruct the audio signal to the headphones. To test and evaluate our ideas, we built prototypes by augmenting standard cables with different sensors. We think that these augmented cords can make certain controls more seamless and intuitive.

Tools and user interfaces are usually designed for users with normal physical condition. Even tho... more Tools and user interfaces are usually designed for users with normal physical condition. Even though more products are built with accessibility considerations, there is still severe lack of opportunity for people with disabilities to use products to their full potential. In this paper, we present the idea of using human muscles as output interface, using functional electrical stimulation (FES). We envision that this would allow for filling the gap between products and a user’s capability, thereby promising every product’s universal accessibility. To match our behavioral and perceptual capability with the product on use, we can reprogram the way we control our body or give direct muscle motion-feedback. We believe that this idea would create new area and provoke discussion in the field of user interactions and their applications.

Playing musical instruments involves producing gradually more challenging body movements and tran... more Playing musical instruments involves producing gradually more challenging body movements and transitions, where the kinematic constraints of the body play a crucial role in structuring the resulting music. We seek to make a bridge between currently accessible motor patterns, and musical possibilities beyond those – afforded through the use of a robotic augmentation. Guitar Machine is a robotic device that presses on guitar strings and assists a musician by fretting alongside her on the same guitar. This paper discusses the design of the system, strategies for using the system to create novel musical patterns, and a user study that looks at the effects of the temporary acquisition of enhanced physical ability. Our results indicate that the proposed human-robot interaction would equip users to explore new musical avenues on the guitar, as well as provide an enhanced understanding of the task at hand on the basis of the robotically acquired ability.

Through several vignettes - Disappearing, Showing Fear, Many Faces, and Dream of Flying, a transf... more Through several vignettes - Disappearing, Showing Fear, Many Faces, and Dream of Flying, a transforming shadow deconstructs the notion of materialistic view of the physical world as well as offers to become a surrogate for channelling the images and imaginations about one's own self.

ArXiv, 2021

We present a system that paints graffiti art with applications in art preservation, human-robot c... more We present a system that paints graffiti art with applications in art preservation, human-robot collaboration, and other tasks which require large-scale dynamic motion. The problem of painting graffiti in a human style is particularly challenging and requires a systems approach because the art and robotics must be designed around each other. Our approach consists of three stages: artwork capture, robot hardware, and robot planning and control. We use motion capture to capture collaborator painting motions which are then composed and processed into a time-varying linear feedback controller for a cable-driven parallel robot (CDPR) to execute. In this work, we will describe the capturing process, the design and construction of a purpose-built CDPR, and the software for turning an artist’s vision into control commands. While we acknowledge that further work is needed for a system which can perfectly recreate human artwork, our results represent a contribution by demonstrating that we ca...

Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

A close integration of human and machine has been envisioned by researchers and artists for gener... more A close integration of human and machine has been envisioned by researchers and artists for generations, however, there has been little effort in investigating the possibility and plausibility of the idea until recent years. We seek to open a discussion on how the notion of self is plastic, and how innately we are capable of empowering and extending ourselves through technologies. Neuroscience studies show using a tool not only offers new capabilities, but also reconstructs our cognitive architecture to include the tool as a part of ourselves. This adaptive nature of the body image poses an opportunity for designing interfaces that become natural extensions to us. In this paper we introduce previous studies drawn from various fields of study, and discuss the role of the body in the self-world relationship, body image plasticity, and how designing the body may affect neural developments. We also offer a categorization of related technologies, along with our current explorations. Finally, potential issues and challenges in realizing the presented form of interfaces are addressed.