David Novick - Academia.edu (original) (raw)
Papers by David Novick
For embodied conversational agents (ECAs) the relationship between gesture and rapport is an open... more For embodied conversational agents (ECAs) the relationship between gesture and rapport is an open question. To enable us to learn whether adding breathing behaviors to an agent similar to SimSensei would lead users interacting to perceive the agent as more natural, we built an application, called Paola Chat, in which the ECA could display naturalistic breathing animations. Our study had two phases. In the first phase, we determined the most natural amplitude for the agent’s breathing. In the second phase, we assessed the effect of breathing on the users’ perceptions of rapport and naturalness. The study had a within-subjects design, with breathing/not-breathing as the independent variable. Despite our expectation that increased naturalness from breathing would lead users to report greater rapport in the breathing condition than in the not-breathing condition, the study’s results suggest that the animation of breathing appears to neither increase nor decrease these perceptions.
The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality ... more The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality applications with embodied conversational agents. VAIF is intended for use by both expert and non-expert users, in contrast with more sophisticated and complex development tools such as the Virtual Human Toolkit. To determine if VAIF is actually usable by a range of users, we conducted a two-phase summative usability test, with a total of 43 participants. We also tried porting to VAIF a scene from an earlier VR application. The results of the usability study suggest that people with little or even no experience in creating embodied conversational agents can install VAIF and build interaction scenes from scratch, with relatively low rates of encountering problem episodes. However, the usability testing disclosed aspects of VAIF and its user’s guide that could be improved to reduce the number of problem episodes that users encounter.
Learning and Collaboration Technologies. Ubiquitous and Virtual Environments for Learning and Collaboration, 2019
This research studies pedagogical agents as a learning companion (PALs) and aims to find which ap... more This research studies pedagogical agents as a learning companion (PALs) and aims to find which approach is more effective for learners in terms of learning, engagement and rapport: a PAL that is the same gender as the student, a PAL that is female, or a PAL that is male. We compared results in terms of learning and rapport with respect to the gender of a PAL. To this end, we implemented a multimodal interactive virtual-reality application to teach students about the 1770 Boston Massacre. Our results suggest that (1) both male and female participants will learn well with a female PAL, and (2) differences in rapport do not seem to affect learning.
Lecture Notes in Computer Science, 2015
Our research seeks to provide embodied conversational agents (ECAs) with behaviors that enable th... more Our research seeks to provide embodied conversational agents (ECAs) with behaviors that enable them to build and maintain rapport with human users. To conduct this research, we need to build agents and systems that can maintain high levels of engagement with humans over multiple interaction sessions. These sessions can potentially extend to longer periods of time to examine long-term effects of the virtual agent’s behaviors. Our current ECA interacts with humans in a game called “Survival on Jungle Island.” Throughout this game, users interact with our agent across several scenes. Each scene is composed of a collection of speech input, speech output, gesture input, gesture output, scenery, triggers, and decision points. Our prior system was developed with procedural code, which did not lend itself to rapid extension to new game scenes. So to enable effective authoring of the scenes for the “Jungle” game, we adopted a declarative approach. We developed ECA middleware that parses, interprets, and executes XML files that define the scenes. This paper presents the XML coding scheme and its implementation and describes the functional back-end enabled by the scene scripts.
Lecture Notes in Computer Science, 2015
In animating embodied conversational agents (ECAs), run-time blending of animations can provide a... more In animating embodied conversational agents (ECAs), run-time blending of animations can provide a large library of movements that increases the appearance of naturalness while decreasing the number of animations to be developed. This approach avoids the need to develop a costly full library of possible animations in advance of use. Our principal scientific contribution is the development of a model for gesture constraints that enables blended animations to represent naturalistic movement. Rather than creating over-detailed, fine-grained procedural animations or hundreds of motion-captured animation files, animators can include sets of their own animations for agents, blend them, and easily reuse animations, while constraining the ECA to use motions that would occur and transition naturally.
Much of our current research explores differences between extraverts and introverts in their perc... more Much of our current research explores differences between extraverts and introverts in their perception and production of gestures in multimodal interaction with embodied conversational agent (ECA). While several excellent corpora of conversational gestures have been collected, these corpora do not distinguish conversants by personality dimensions. To enable study of the differences in distribution of conversational gestures between extraverts and introverts, we tracked and automatically annotated gestures of 59 subjects interacting with an ECA in the “Survival on Jungle Island” immersive multimodal adventure. Our work in developing provides an initial corpus for analysis of gesture differences, based on Jung’s four personality-type dimensions, of humans interacting with an ECA and suggests that it may be feasible to annotate gestures automatically in real time, based on a gesture lexicon. Preliminary analysis of the automated annotations suggests that introverts more frequently per...
For developers of meaningful games, higher levels of engagement can keep players interacting with... more For developers of meaningful games, higher levels of engagement can keep players interacting with the game, thus achieving the game’s intended learning outcomes. And for games based on the technology of embodied conversational agents (ECAs), higher levels of engagement could lead to increased rapport between the player and the agent, again increasing play. Rapport is a complex and extensive behavioral state of affinity, synchronicity, coordination and mutual understanding that is difficult to model, measure and interpret. In this paper we present how fullbody physical engagement helps build rapport and engagement by establishing a stronger sense of emotional connection between players and virtual characters.
Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animat... more Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animation skills, advanced programming knowledge, and in some cases artificial intelligence or the integration of complex interaction models. Features like lip-syncing to an audio file, recognizing the users' speech, or having the character move at certain times in certain ways, are inaccessible to researchers that want to build and use these agents for education, research, or industrial uses. VAIF, the Virtual Agent Interaction Framework, is an extensively documented system that attempts to bridge that gap and provide inexperienced researchers the tools and means to develop their own agents in a centralized, lightweight platform that provides all these features through a simple interface within the Unity game engine.
Two related goals of IVA research involve increasing the believability and perceived trustworthin... more Two related goals of IVA research involve increasing the believability and perceived trustworthiness of agents and increasing the user's sense of engagement. In our research, we use human-human interaction as a model for agent behaviors that build rapport between humans and agents. Our paralinguistic model of rapport (Novick & Gris, 2014) comprises a sense of emotional connection, a sense of mutual understanding, and a sense of physical connection. Our research, which focuses primarily on the sense of physical connection, seeks to increase the naturalness of non-verbal interaction to correspondingly increase human-IVA rapport (Tickle-Degnan & Rosenthal, 1987; Huang et al., 2011). For these studies, we have developed two agents, with variable non-verbal behaviors, in immersive games designed to maintain users’ engagement.
2018 ASEE Annual Conference & Exposition Proceedings
Distinguished Chair in Engineering and Professor of Engineering Education and Leadership, earned ... more Distinguished Chair in Engineering and Professor of Engineering Education and Leadership, earned his J.D.at Harvard University in 1977 and his Ph.D. in Computer and Information Science at the University of Oregon in 1988. Before coming to UTEP he was on the faculty of the Department of Computer Science and Engineering at the Oregon Graduate Institute and then Director of Research at the European Institute of Cognitive Sciences and Engineering. At UTEP he has served in a number of positions including as Chair of the Department of Computer Science, Associate Provost, Associate Dean of Engineering for Graduate Studies and Research, and co-director of the Mike Loya Center for Innovation and Commerce. His research focuses on interactive systems, especially human interaction with intelligent virtual agents, and on interaction in support of innovation. He served as General Co
ACM SIGCHI Bulletin, 1999
The ACM SIGCHI (Association for Computing Machinery Special lnterest Group in Computer Human Inte... more The ACM SIGCHI (Association for Computing Machinery Special lnterest Group in Computer Human Interaction) community conducted a deliberative process involving a high-visibility committee, a day-long workshop at CH199 (Pittsburgh, PA, May 15, 1999) and a collaborative authoring process. This interim report is offered to produce further discussion and input leading to endorsement by the SIGCHI Executive Committee and then other professional societies. The scope of this research agenda included advanced information and communications technology research that could yield benefits over the next two to five years. The participants included representatives from several countries, but this draft needs refinement to satisfy the unique situations of each nation and each situation.
This paper presents a dataset collected from natural dialogs which enables to test the ability of... more This paper presents a dataset collected from natural dialogs which enables to test the ability of dialog systems to learn new facts from user utterances throughout the dialog. This interactive learning will help with one of the most prevailing problems of open domain dialog system, which is the sparsity of facts a dialog system can reason about. The proposed dataset, consisting of 1900 collected dialogs, allows simulation of an interactive gaining of denotations and questions explanations from users which can be used for the interactive learning.
Human-Computer Interaction. User Interface Design, Development and Multimodality
The literature suggests that familiarity and rapport are enhanced by larger, more extraverted ges... more The literature suggests that familiarity and rapport are enhanced by larger, more extraverted gestures. However, the sizes of the increases in amplitude have not been reported. We sought to determine whether this relationship holds true for interaction between humans and embodied conversational agents. To this end, we conducted an experiment in which we increased gesture amplitude, with quantification of the gesture sizes. We hypothesized that rapport would be increased in the larger-gesture condition. However, our results were exactly the opposite: Rapport fell significantly in the larger-gesture condition. This means that larger may not always be better for building human-agent rapport. Our unexpected results may be because our agent's gestures were simply too big, odd, awkward, or strange, or because of a statistical anomaly.
Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96
This paper explores the role of gaze in coordinating turn-taking in mixed-initiative conversation... more This paper explores the role of gaze in coordinating turn-taking in mixed-initiative conversation and specifically how gaze indicators might be usefully modeled in computational dialogue systems. We analyzed about 20 minutes of videotape of eight dialogues by four pairs of subjects performing a simple face-to-face cooperative laboratory task. We extend previous studies by explicating gaze patterns in face-to-face conversations, formalizing the most frequent pattern as a computational model of turn-taking, and testing the model through an agent-based simulation. Prior conversation simulations of conversational control acts relied on abstract speech-act representations of control. This study advances the computational account of dialogue through simulation of direct physical expression of gaze to coordinate conversational turns.
Interspeech 2005
As a priority-setting exercise, we compared interactions between users and a simple spoken dialog... more As a priority-setting exercise, we compared interactions between users and a simple spoken dialog system to interactions between users and a human operator. We observed usability events, places in which system behavior differed from human behavior, and for each we noted the impact, root causes, and prospects for improvement. We suggest some priority issues for research, involving not only such core areas as speech recognition and synthesis and language understanding and generation, but also less-studied topics such as adaptive or flexible timeouts, turn-taking and speaking rate.
Proceedings of the Twenty-Fourth Annual Hawaii International Conference on System Sciences
Interface construction kits have become firmly established and well-accepted in recent years. The... more Interface construction kits have become firmly established and well-accepted in recent years. They offer solutions to the programming complexity created by introducing user interface operations into traditional programming languages. The authors argue that an understanding of the productivity gain afforded by construction kits must be shaped by separating programming language from programming environment. The QUICK user interface construction kit is described and used as an example to explore what the authors call the middle ground in language abstraction: the prototype model of object-oriented languages
This panel presents the Computing Alliance of Hispanic-Serving Institutions. The Alliance, develo... more This panel presents the Computing Alliance of Hispanic-Serving Institutions. The Alliance, developed by eight HSIs with funding from the NSF's Broadening Participation Program, seeks to increase the numbers of Hispanics in all areas of computing, including increasing the number of Hispanic students who enter the professoriate, retaining and advancing Hispanic faculty, and developing and sustaining competitive research and education programs
Proceedings of the 23rd annual international conference on Design of communication documenting & designing for pervasive information - SIGDOC '05, 2005
To reduce potential discrepancies between textual and graphical content in documentation, it is p... more To reduce potential discrepancies between textual and graphical content in documentation, it is possible to produce both text and graphics from a single common source. One approach to cogeneration of text and graphics uses a single logical specification; a second approach starts with CAD-based representation and produces a corresponding textual account. This paper explores these two different approaches, reports the results of using prototypes embodying the approaches to represent simple figures, and discusses issues that were identified through use of the prototypes. While it appears feasible to co-generate text and graphics automatically, the process raises deep issues of design of communications, including the intent of the producer of the documentation.
For embodied conversational agents (ECAs) the relationship between gesture and rapport is an open... more For embodied conversational agents (ECAs) the relationship between gesture and rapport is an open question. To enable us to learn whether adding breathing behaviors to an agent similar to SimSensei would lead users interacting to perceive the agent as more natural, we built an application, called Paola Chat, in which the ECA could display naturalistic breathing animations. Our study had two phases. In the first phase, we determined the most natural amplitude for the agent’s breathing. In the second phase, we assessed the effect of breathing on the users’ perceptions of rapport and naturalness. The study had a within-subjects design, with breathing/not-breathing as the independent variable. Despite our expectation that increased naturalness from breathing would lead users to report greater rapport in the breathing condition than in the not-breathing condition, the study’s results suggest that the animation of breathing appears to neither increase nor decrease these perceptions.
The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality ... more The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality applications with embodied conversational agents. VAIF is intended for use by both expert and non-expert users, in contrast with more sophisticated and complex development tools such as the Virtual Human Toolkit. To determine if VAIF is actually usable by a range of users, we conducted a two-phase summative usability test, with a total of 43 participants. We also tried porting to VAIF a scene from an earlier VR application. The results of the usability study suggest that people with little or even no experience in creating embodied conversational agents can install VAIF and build interaction scenes from scratch, with relatively low rates of encountering problem episodes. However, the usability testing disclosed aspects of VAIF and its user’s guide that could be improved to reduce the number of problem episodes that users encounter.
Learning and Collaboration Technologies. Ubiquitous and Virtual Environments for Learning and Collaboration, 2019
This research studies pedagogical agents as a learning companion (PALs) and aims to find which ap... more This research studies pedagogical agents as a learning companion (PALs) and aims to find which approach is more effective for learners in terms of learning, engagement and rapport: a PAL that is the same gender as the student, a PAL that is female, or a PAL that is male. We compared results in terms of learning and rapport with respect to the gender of a PAL. To this end, we implemented a multimodal interactive virtual-reality application to teach students about the 1770 Boston Massacre. Our results suggest that (1) both male and female participants will learn well with a female PAL, and (2) differences in rapport do not seem to affect learning.
Lecture Notes in Computer Science, 2015
Our research seeks to provide embodied conversational agents (ECAs) with behaviors that enable th... more Our research seeks to provide embodied conversational agents (ECAs) with behaviors that enable them to build and maintain rapport with human users. To conduct this research, we need to build agents and systems that can maintain high levels of engagement with humans over multiple interaction sessions. These sessions can potentially extend to longer periods of time to examine long-term effects of the virtual agent’s behaviors. Our current ECA interacts with humans in a game called “Survival on Jungle Island.” Throughout this game, users interact with our agent across several scenes. Each scene is composed of a collection of speech input, speech output, gesture input, gesture output, scenery, triggers, and decision points. Our prior system was developed with procedural code, which did not lend itself to rapid extension to new game scenes. So to enable effective authoring of the scenes for the “Jungle” game, we adopted a declarative approach. We developed ECA middleware that parses, interprets, and executes XML files that define the scenes. This paper presents the XML coding scheme and its implementation and describes the functional back-end enabled by the scene scripts.
Lecture Notes in Computer Science, 2015
In animating embodied conversational agents (ECAs), run-time blending of animations can provide a... more In animating embodied conversational agents (ECAs), run-time blending of animations can provide a large library of movements that increases the appearance of naturalness while decreasing the number of animations to be developed. This approach avoids the need to develop a costly full library of possible animations in advance of use. Our principal scientific contribution is the development of a model for gesture constraints that enables blended animations to represent naturalistic movement. Rather than creating over-detailed, fine-grained procedural animations or hundreds of motion-captured animation files, animators can include sets of their own animations for agents, blend them, and easily reuse animations, while constraining the ECA to use motions that would occur and transition naturally.
Much of our current research explores differences between extraverts and introverts in their perc... more Much of our current research explores differences between extraverts and introverts in their perception and production of gestures in multimodal interaction with embodied conversational agent (ECA). While several excellent corpora of conversational gestures have been collected, these corpora do not distinguish conversants by personality dimensions. To enable study of the differences in distribution of conversational gestures between extraverts and introverts, we tracked and automatically annotated gestures of 59 subjects interacting with an ECA in the “Survival on Jungle Island” immersive multimodal adventure. Our work in developing provides an initial corpus for analysis of gesture differences, based on Jung’s four personality-type dimensions, of humans interacting with an ECA and suggests that it may be feasible to annotate gestures automatically in real time, based on a gesture lexicon. Preliminary analysis of the automated annotations suggests that introverts more frequently per...
For developers of meaningful games, higher levels of engagement can keep players interacting with... more For developers of meaningful games, higher levels of engagement can keep players interacting with the game, thus achieving the game’s intended learning outcomes. And for games based on the technology of embodied conversational agents (ECAs), higher levels of engagement could lead to increased rapport between the player and the agent, again increasing play. Rapport is a complex and extensive behavioral state of affinity, synchronicity, coordination and mutual understanding that is difficult to model, measure and interpret. In this paper we present how fullbody physical engagement helps build rapport and engagement by establishing a stronger sense of emotional connection between players and virtual characters.
Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animat... more Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animation skills, advanced programming knowledge, and in some cases artificial intelligence or the integration of complex interaction models. Features like lip-syncing to an audio file, recognizing the users' speech, or having the character move at certain times in certain ways, are inaccessible to researchers that want to build and use these agents for education, research, or industrial uses. VAIF, the Virtual Agent Interaction Framework, is an extensively documented system that attempts to bridge that gap and provide inexperienced researchers the tools and means to develop their own agents in a centralized, lightweight platform that provides all these features through a simple interface within the Unity game engine.
Two related goals of IVA research involve increasing the believability and perceived trustworthin... more Two related goals of IVA research involve increasing the believability and perceived trustworthiness of agents and increasing the user's sense of engagement. In our research, we use human-human interaction as a model for agent behaviors that build rapport between humans and agents. Our paralinguistic model of rapport (Novick & Gris, 2014) comprises a sense of emotional connection, a sense of mutual understanding, and a sense of physical connection. Our research, which focuses primarily on the sense of physical connection, seeks to increase the naturalness of non-verbal interaction to correspondingly increase human-IVA rapport (Tickle-Degnan & Rosenthal, 1987; Huang et al., 2011). For these studies, we have developed two agents, with variable non-verbal behaviors, in immersive games designed to maintain users’ engagement.
2018 ASEE Annual Conference & Exposition Proceedings
Distinguished Chair in Engineering and Professor of Engineering Education and Leadership, earned ... more Distinguished Chair in Engineering and Professor of Engineering Education and Leadership, earned his J.D.at Harvard University in 1977 and his Ph.D. in Computer and Information Science at the University of Oregon in 1988. Before coming to UTEP he was on the faculty of the Department of Computer Science and Engineering at the Oregon Graduate Institute and then Director of Research at the European Institute of Cognitive Sciences and Engineering. At UTEP he has served in a number of positions including as Chair of the Department of Computer Science, Associate Provost, Associate Dean of Engineering for Graduate Studies and Research, and co-director of the Mike Loya Center for Innovation and Commerce. His research focuses on interactive systems, especially human interaction with intelligent virtual agents, and on interaction in support of innovation. He served as General Co
ACM SIGCHI Bulletin, 1999
The ACM SIGCHI (Association for Computing Machinery Special lnterest Group in Computer Human Inte... more The ACM SIGCHI (Association for Computing Machinery Special lnterest Group in Computer Human Interaction) community conducted a deliberative process involving a high-visibility committee, a day-long workshop at CH199 (Pittsburgh, PA, May 15, 1999) and a collaborative authoring process. This interim report is offered to produce further discussion and input leading to endorsement by the SIGCHI Executive Committee and then other professional societies. The scope of this research agenda included advanced information and communications technology research that could yield benefits over the next two to five years. The participants included representatives from several countries, but this draft needs refinement to satisfy the unique situations of each nation and each situation.
This paper presents a dataset collected from natural dialogs which enables to test the ability of... more This paper presents a dataset collected from natural dialogs which enables to test the ability of dialog systems to learn new facts from user utterances throughout the dialog. This interactive learning will help with one of the most prevailing problems of open domain dialog system, which is the sparsity of facts a dialog system can reason about. The proposed dataset, consisting of 1900 collected dialogs, allows simulation of an interactive gaining of denotations and questions explanations from users which can be used for the interactive learning.
Human-Computer Interaction. User Interface Design, Development and Multimodality
The literature suggests that familiarity and rapport are enhanced by larger, more extraverted ges... more The literature suggests that familiarity and rapport are enhanced by larger, more extraverted gestures. However, the sizes of the increases in amplitude have not been reported. We sought to determine whether this relationship holds true for interaction between humans and embodied conversational agents. To this end, we conducted an experiment in which we increased gesture amplitude, with quantification of the gesture sizes. We hypothesized that rapport would be increased in the larger-gesture condition. However, our results were exactly the opposite: Rapport fell significantly in the larger-gesture condition. This means that larger may not always be better for building human-agent rapport. Our unexpected results may be because our agent's gestures were simply too big, odd, awkward, or strange, or because of a statistical anomaly.
Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96
This paper explores the role of gaze in coordinating turn-taking in mixed-initiative conversation... more This paper explores the role of gaze in coordinating turn-taking in mixed-initiative conversation and specifically how gaze indicators might be usefully modeled in computational dialogue systems. We analyzed about 20 minutes of videotape of eight dialogues by four pairs of subjects performing a simple face-to-face cooperative laboratory task. We extend previous studies by explicating gaze patterns in face-to-face conversations, formalizing the most frequent pattern as a computational model of turn-taking, and testing the model through an agent-based simulation. Prior conversation simulations of conversational control acts relied on abstract speech-act representations of control. This study advances the computational account of dialogue through simulation of direct physical expression of gaze to coordinate conversational turns.
Interspeech 2005
As a priority-setting exercise, we compared interactions between users and a simple spoken dialog... more As a priority-setting exercise, we compared interactions between users and a simple spoken dialog system to interactions between users and a human operator. We observed usability events, places in which system behavior differed from human behavior, and for each we noted the impact, root causes, and prospects for improvement. We suggest some priority issues for research, involving not only such core areas as speech recognition and synthesis and language understanding and generation, but also less-studied topics such as adaptive or flexible timeouts, turn-taking and speaking rate.
Proceedings of the Twenty-Fourth Annual Hawaii International Conference on System Sciences
Interface construction kits have become firmly established and well-accepted in recent years. The... more Interface construction kits have become firmly established and well-accepted in recent years. They offer solutions to the programming complexity created by introducing user interface operations into traditional programming languages. The authors argue that an understanding of the productivity gain afforded by construction kits must be shaped by separating programming language from programming environment. The QUICK user interface construction kit is described and used as an example to explore what the authors call the middle ground in language abstraction: the prototype model of object-oriented languages
This panel presents the Computing Alliance of Hispanic-Serving Institutions. The Alliance, develo... more This panel presents the Computing Alliance of Hispanic-Serving Institutions. The Alliance, developed by eight HSIs with funding from the NSF's Broadening Participation Program, seeks to increase the numbers of Hispanics in all areas of computing, including increasing the number of Hispanic students who enter the professoriate, retaining and advancing Hispanic faculty, and developing and sustaining competitive research and education programs
Proceedings of the 23rd annual international conference on Design of communication documenting & designing for pervasive information - SIGDOC '05, 2005
To reduce potential discrepancies between textual and graphical content in documentation, it is p... more To reduce potential discrepancies between textual and graphical content in documentation, it is possible to produce both text and graphics from a single common source. One approach to cogeneration of text and graphics uses a single logical specification; a second approach starts with CAD-based representation and produces a corresponding textual account. This paper explores these two different approaches, reports the results of using prototypes embodying the approaches to represent simple figures, and discusses issues that were identified through use of the prototypes. While it appears feasible to co-generate text and graphics automatically, the process raises deep issues of design of communications, including the intent of the producer of the documentation.