Javier Gonzalez-Sanchez | Arizona State University (original) (raw)

Papers by Javier Gonzalez-Sanchez

ABSTRACT One important way for systems to adapt to their individual users is related to their abi... more ABSTRACT One important way for systems to adapt to their individual users is related to their ability to show empathy. Being empathetic implies that the computer is able to recognize a user's affective states and understand the implication of those states. Detection of affective states is a step forward to provide machines with the necessary intelligence to appropriately interact with humans. This course provides a description and demonstration of tools and methodologies for automatically detecting affective states with a multimodal approach.

Elsevier eBooks, 2017

Affect is inextricably related to human cognitive processes and expresses a great deal about huma... more Affect is inextricably related to human cognitive processes and expresses a great deal about human necessities (Picard, 1997); affect signals what matters to us and what we care about. Furthermore, affect impacts our rational decision-making and action selection (Picard, 2010). Providing computers with the capability to recognize, understand, and respond to human affective states would narrow the communication gap between the highly emotional human and the emotionally detached computer, enhancing their interactions. Computer applications in learning, health care, and entertainment stand to benefit from such capabilities. Affect is a conceptual quantity with fuzzy boundaries and with substantial individual difference variations in expression and experience (Picard, 1997). This makes measuring affect a challenging task. This chapter does not intend to present a comprehensive survey of the methods used to measure affect but instead provides a roadmap through a selection of the principal approaches and technologies. Section "Affect, Emotions, and Measurement" introduces the concepts and background behind affect measurement. Section "Gathering Data: Approaches and Technologies"

Springer eBooks, 2014

The Affective Meta-Tutoring system is comprised of (1) a tutor that teaches system dynamics model... more The Affective Meta-Tutoring system is comprised of (1) a tutor that teaches system dynamics modeling, (2) a meta-tutor that teaches good strategies for learning how to model from the tutor, and (3) an affective learning companion that encourages students to use the learning strategy that the metatutor teaches. The affective learning companion's messages are selected by using physiological sensors and log data to determine the student's affective state. Evaluations compared the learning gains of three conditions: the tutor alone, the tutor plus meta-tutor and the tutor, meta-tutor and affective learning companion.

2013 35th International Conference on Software Engineering (ICSE), May 1, 2013

ABSTRACT One expected characteristic in modern systems is self-adaptation, the capability of moni... more ABSTRACT One expected characteristic in modern systems is self-adaptation, the capability of monitoring and reacting to changes into the environment. A particular case of self- adaptation is affective-driven self-adaptation. Affective-driven self-adaptation is about having consciousness of user’s affects (emotions) and drive self-adaptation reacting to changes in those affects. Most of the previous work around self-adaptive systems deals with performance, resources, and error recovery as variables that trigger a system reaction. Moreover, most effort around affect recognition has been put towards offline analysis of affect, and to date only few applications exist that are able to infer user’s affect in real-time and trigger self-adaptation mechanisms. In response to this deficit, this work proposes a software product line approach to jump-start the development of affect-driven self-adaptive systems by offering the definition of a domain-specific architecture, a set of components (organized as a framework), and guidelines to tailor those components. Study cases with systems for learning and gaming will confirm the capability of the software product line to provide desired functionalities and qualities.

Feedback represents a vital component of the learning process and is especially important for Com... more Feedback represents a vital component of the learning process and is especially important for Computer Science students. With class sizes that are often large, it can be challenging to provide individualized feedback to students. Consistent, constructive, supportive feedback through a tutoring companion can scaffold the learning process for students. This work contributes to the construction of a tutoring companion designed to provide this feedback to students. It aims to bridge the gap between the messages the compiler delivers, and the support required for a novice student to understand the problem and fix their code. Particularly, it provides support for students learning about recursion in a beginning university Java programming course. Besides also providing affective support, a tutoring companion could be more effective when it is embedded into the environment that the student is already using, instead of an additional tool for the student to learn. The proposed Tutoring Companion is embedded into the Eclipse Integrated Development Environment (IDE). This thesis focuses on the reasoning model for the Tutoring Companion and is developed using the techniques of a neural network. While a student uses the IDE, the Tutoring Companion collects 16 data points, including the presence of certain key words, cyclomatic complexity, and error messages from the compiler, every time it detects an event, such as a run attempt, debug attempt, or a request for help, in the IDE. This data is used as inputs to the neural network. The neural network produces a correlating single output code for the feedback to be provided to the student, which is displayed in the IDE.

Springer eBooks, 2013

Research in affective computing and educational technology has shown the potential of affective i... more Research in affective computing and educational technology has shown the potential of affective interventions to increase student's self-concept and motivation while learning. Our project aims to investigate whether the use of affective interventions in a meta-cognitive tutor can help students achieve deeper modeling of dynamic systems by being persistent in their use of metacognitive strategies during and after tutoring. This article is an experience report on how we designed and implemented the affective intervention. (The meta-tutor is described in a separate paper.) We briefly describe the theories of affect underlying the design and how the agent's affective behavior is defined and implemented. Finally, the evaluation of a detector-driven categorization of student behavior, that guides the agent's affective interventions, against a categorization performed by human coders, is presented.

ABSTRACT There is a growing interest in how to leverage information about user's emotions... more ABSTRACT There is a growing interest in how to leverage information about user's emotions as a mean of personalizing the response of computer systems. This is particularly useful for computer-aided learning, health, and entertainment systems. However, there are few architectures, frameworks, libraries, or software tools that allow developers to easily integrate emotion recognition into their software projects. The work reported in this paper offers a way to address this shortcoming in models by proposing the use of software design patterns for modeling a multimodal emotion recognition framework. The framework is designed to: (1) integrate existing sensing devices and SDK platforms, (2) include diverse inference algorithms, and (3) correlate measurements from diverse sources. We describe our experience using this model and its impact on facets, such as creating a common language among stakeholders, supporting an incremental development, and adjusting to a highly shifting development team, as well as the qualities achieved and trade-offs made.

Lecture Notes in Computer Science, 2014

Intelligent Tutoring Systems (ITSs) constitute an alternative to expert human tutors, providing d... more Intelligent Tutoring Systems (ITSs) constitute an alternative to expert human tutors, providing direct customized instruction and feedback to students. ITSs could positively impact education if adopted on a large scale, but doing that requires tools to enable their mass production. This circumstance is the key motivation for this work. We present a component-based approach for a system architecture for ITSs equipped with meta-tutoring and affective capabilities. We elicited the requirements that those systems might address and created a system architecture that models their structure and behavior to drive development efforts. Our experience applying the architecture in the incremental implementation of a four-year project is discussed.

Lecture Notes in Computer Science, 2013

User modeling in AIED has been extended in the past decades to include affective and motivational... more User modeling in AIED has been extended in the past decades to include affective and motivational aspects of learner's interaction in intelligent tutoring systems. An issue in such systems is researchers' ability to understand and detect students' cognitive and meta-cognitive processes while they learn. In order to study those factors, various detectors have been created that classify episodes in log data as gaming, high/low effort on task, robust learning, etc. When simulating students' learning processes in an ITS, a question remains as to how to create those detectors, and how reliable their simulation of the user's learning processes can be. In this article, we present our method for creating a detector of shallow modeling practices within a meta-tutor instructional system. The detector was defined using HCI (human-computer interaction) task modeling as well as a coding scheme defined by human coders from past users' screen recordings of software use. The detector produced classifications of student behavior that were highly similar to classifications produced by human coders with a kappa of .925.

International Journal of Environmental Research and Public Health

Over seven million people suffer from an impairment in Mexico; 64.1% are gait-related, and 36.2% ... more Over seven million people suffer from an impairment in Mexico; 64.1% are gait-related, and 36.2% are children aged 0 to 14 years. Furthermore, many suffer from neurological disorders, which limits their verbal skills to provide accurate feedback. Robot-assisted gait therapy has shown significant benefits, but the users must make an active effort to accomplish muscular memory, which usually is only around 30% of the time. Moreover, during therapy, the patients’ affective state is mostly unsatisfied, wide-awake, and powerless. This paper proposes a method for increasing the efficiency by combining affective data from an Emotiv Insight, an Oculus Go headset displaying an immersive interaction, and a feedback system. Our preliminary study had eight patients during therapy and eight students analyzing the footage using the self-assessment Manikin. It showed that it is possible to use an EEG headset and identify the affective state with a weighted average precision of 97.5%, recall of 87....

2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), 2019

With Computer Science (CS) class sizes that are often large, it is challenging to provide effecti... more With Computer Science (CS) class sizes that are often large, it is challenging to provide effective personalized feedback to students. Intelligent Tutoring Companions can provide such feedback and improve CS students' experience. This work describes the construction of a Tutoring Companion, Annete, designed to support students in a university Java programming course by providing them with intelligent feedback generated by a neural network. Annete is embedded into the Eclipse Integrated Development Environment (IDE), which is an environment that is already familiar to students in programming courses. Embedding Annete into Eclipse improves her effectiveness, as the students do not need to learn how to use an additional tool. While the student works in Eclipse, Annete collects 21 pieces of data from the student's code, including whether certain key words are used, error messages from the compiler, and cyclomatic complexity. When a run attempt, debug attempt, or a request for help occurs in Eclipse, Annete uses the data available to infer a feedback message to show to the student. Our approach is evaluated among 28 CS students completing a programming assignment while Annete assists them. Results suggest that students feel supported while working with Annete and show potential for using neural network modeling with embedded tutoring companions in the future. Challenges are discussed, as well as opportunities for future work.

Building affect-driven adaptive environments is a task geared toward creating environments able t... more Building affect-driven adaptive environments is a task geared toward creating environments able to change based on the affective state of a target user. In our project, the environment is the well-known game, Pac-Man. To provide affect-driven adaptive capabilities, diverse sensors were utilized to gather a user's physiological data and an emotion recognition framework was used to fuse the sensed data and infer affective states. The game changes driven by those affective states aim to improve the user experience by keeping or increasing player's engagement.

Lecture Notes in Computer Science, 2013

While modeling dynamic systems in an efficient manner is an important skill to acquire for a scie... more While modeling dynamic systems in an efficient manner is an important skill to acquire for a scientist, it is a difficult skill to acquire. A simple step-based tutoring system, called AMT, was designed to help students learn how to construct models of dynamic systems using deep modeling practices. In order to increase the frequency of deep modeling and reduce the amount of guessing/gaming, a meta-tutor coaching students to follow a deep modeling strategy was added to the original modeling tool. This paper presents the results of two experiments investigating the effectiveness of the meta-tutor when compared to the original software. The results indicate that students who studied with the meta-tutor did indeed engage more in deep modeling practices.

Lecture Notes in Computer Science, 2013

International Journal of Artificial Intelligence in Education, 2015

Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA '14, 2014

Proceedings of the 18th Conference on Pattern Languages of Programs - PLoP '11, 2011

Intelligent Tutoring Systems are capable of becoming an alternative to expert human tutors, able ... more Intelligent Tutoring Systems are capable of becoming an alternative to expert human tutors, able to provide direct customized instruction and feedback to students. Although Intelligent Tutoring Systems could differ widely in their attached knowledge bases and user interfaces (including interaction mechanisms), their behaviors are quite similar; thus, it must be possible to establish a common software model for them. A common software model is a step forward to move these systems from proof-of-concepts and academic research tools to widely available tools in schools and homes. The work reported here addresses: (1) the use of Design Patterns to create an object-oriented software model for Intelligent Tutoring Systems; (2) the application of the model into a two-year development project; (3) the qualities achieved and trade-offs made. Besides that, this paper describes our experience using patterns, and the impact in facts such as creating a common language among stakeholders, supporting an incremental development and adjustment to a highly shifting developing team.

2013 IEEE 13th International Conference on Advanced Learning Technologies, 2013

The ability of a learning system to infer a student's affects has become highly relevant to be ab... more The ability of a learning system to infer a student's affects has become highly relevant to be able to adjust its pedagogical strategies. Several methods have been used to infer affects. One of the most recognized for its reliability is facebased affect recognition. Another emerging one involves the use of brain-computer interfaces. In this paper we compare those strategies and explore if, to a great extent, it is possible to infer the values of one source from the other source.

Companion to the 22nd ACM SIGPLAN conference on Object oriented programming systems and applications companion - OOPSLA '07, 2007

Introductory programming course have two very specific difficulties for novice students. First is... more Introductory programming course have two very specific difficulties for novice students. First is the lack of real world examples in the sessions. It is very difficult to find areas of application where all the students are familiar enough and that offers challenging and engaging examples. Second it is the lack of palpable results of the job done. Introductory courses in other fields generate products that the students can show to others, and feel proud about it. In CS1, for example, explaining loops by printing a series of numbers on the screen doesn't yield the same sense of accomplishment as drawing a basic perspective in an architecture course. We propose using Project Hoshimi [1], a Microsoft Platform, as a base for introducing computer programming to CS1 students. Through the paper we discuss how it could be implemented in the classroom, the main advantages and disadvantages in our experience of using Project Hoshimi, comparing its use against other more traditional approaches, as well as against other graphic programming methods such as Alice or videogame based learning.

Elsevier eBooks, 2017

Springer eBooks, 2014

2013 35th International Conference on Software Engineering (ICSE), May 1, 2013

Springer eBooks, 2013

Lecture Notes in Computer Science, 2014

Lecture Notes in Computer Science, 2013

International Journal of Environmental Research and Public Health

2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), 2019

Lecture Notes in Computer Science, 2013

International Journal of Artificial Intelligence in Education, 2015

Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA '14, 2014

Proceedings of the 18th Conference on Pattern Languages of Programs - PLoP '11, 2011

2013 IEEE 13th International Conference on Advanced Learning Technologies, 2013

Companion to the 22nd ACM SIGPLAN conference on Object oriented programming systems and applications companion - OOPSLA '07, 2007

One novel part of the design of interactions between people and computers, related with the facet... more One novel part of the design of interactions between people and computers, related with the facet of securing user satisfaction, is the capability of systems to adapt to their individual users showing empathy. Being empathetic implies that the computer is able to recognize user’s affective states and understand the implication of those states. Automatic detection of affective states requires the computer: to sense information; to process and understand information integrating several sources that could range from brain-waves signals and biofeedback readings, passing from gestures recognition, to posture and pressure sensing; and to apply algorithms and data processing tools to understand user’s affective states.

Through this course, attendees will:

a) Learn about sensing approaches used to detect affective states: brain-computer interfaces, face-based emotion recognition systems, eye-tracking system, and physiological sensors –including skin conductivity, posture, and pressure sensors—.

b) Understand the pros and cons of the diverse sensing approaches used to detect affective states.

c) Learn about the data that is gathered from each device and understand its characteristics.

d) Learn about approaches and tools to pre-process, synchronize, and analyze data.

This course is open to researchers, practitioners, and educators interested in incorporating affective computing as part of their adaptive and personalized technology toolbox.

(in Spanish) - Conforme el tamaño y complejidad del software se incrementa, diseñarlo va más allá... more (in Spanish) - Conforme el tamaño y complejidad del software se incrementa, diseñarlo va más allá de la aplicación sistemática y cuantificable de una metodología para su construcción, operación y mantenimiento. La arquitectura de software como disciplina, se centra en la idea de reducir la complejidad del software especificando, modelando y administrando dependencias, comportamientos, y cualidades. Este taller presenta una guía para profesionales de la industria de software, bajo un enfoque técnico, práctico y ágil, respecto a:

a) ¿por qué extender el diseño de software a un nivel arquitectónico?

b) ¿cuándo es valioso abordar el diseño de un proyecto desde un enfoque arquitectónico?

c) ¿qué enfoque utilizar (model-driven, reuse-driven o risk-driven)?

d) ¿cómo iniciar la adopción de un enfoque arquitectónico?

Concluyendo con una discusión cualitativa y cuantitativa del uso de estilos, modelos, patrones y tácticas arquitectónicas.

The computer’s ability to recognize human emotional states given physiological signals is gaining... more The computer’s ability to recognize human emotional states given physiological signals is gaining in popularity to create empathetic systems such as learning environments, health care systems and videogames. Despite that, there are few frameworks, libraries, architectures, or software tools, which allow systems developers to easily integrate emotion recognition into their software projects. The work reported here offers a first step to fill this gap in the lack of frameworks and models, addressing: (a) the modeling of an agent-driven component-based architecture for multimodal emotion recognition, called ABE, and (b) the use of ABE to implement a multimodal emotion recognition framework to support third-party systems becoming empathetic systems.

"Apple’s mobile handheld devices have been a huge success; beginning with the launching of the iP... more "Apple’s mobile handheld devices have been a huge success; beginning with the launching of the iPhone in 2007 and that has been maintained until present days with the launching of the iPad in 2010. There are now more than 85,000 apps available for the more than 50 million iPhone and iPod touch customers worldwide and over 125,000 developers in Apple’s iPhone Developer Program. That is why iPhone is a new and widely extended platform to develop object-oriented applications.iPhone platform involves several and amazing technologies that makes programming it a cool activity for experienced programmers willing to know the goodness of a mobile device, and a great option for students or novices to learn programming. Both of them will find the power of the iPhone libraries and development tools amazing to quickly start building powerful iPhone apps. We will be using Xcode under Mac OS X as our IDE, and Objective-C and iPhone API as our programming tools to create iPhone applications (from basic to medium level), which finally can be loaded into the iPhone device. We’ll start with the classical “Hello World” and continue to develop applications using graphical user interfaces, handling multi-touch and motion detection, communication interfaces, and different media.
"

This tutorial will discuss key software architecture concepts allowing researchers and practition... more This tutorial will discuss key software architecture concepts allowing researchers and practitioners to move from software development as a one-ofa-kind endeavor to software development as a system of modules that can be widely used in highly adaptable ways. Participants will understand how the use of software architecture models avoids difficulty by providing robust and flexible systems, applies well-established methodologies, improves user experience and, warrants that the final product satisfies all the requirements (functional and non functional). The attendees will better understand how to apply software patterns as part of their software development methodology. They will engage in exercises that apply software patterns at architectural and design levels to create an Intelligent Tutor System (ITS) with an associated widespread use of components. And, understand how software architecture can improve ITS development to a point where one could be sure that the product would work, as planned, in uncontrolled and non-standardized environments and be able to be incrementally improved to be adapted with new technologies. The attendee will become familiar with the standard vocabularies of patterns in order to apply it as a way of shared communication inside the team and outside with the experts and consultants. This will improve attendees’ abilities to express and transfer knowledge that can enrich the development and improvement of systems.

"LEGO ® Mindstorms ® robots can do more than you ever expected. How to get in there?. The answer ... more "LEGO ® Mindstorms ® robots can do more than you ever expected. How to get in there?. The answer is combining the LEGO Mindstorms cool and fun hardware with the power of the Java platform–the world’s hottest programming technology.
This tutorial will show you how to get in, step by step. We’ll start with the basics and then we’ll teach you techniques to control LEGO ® Mindstorms ® robots. This tutorial will teach you how to build robots combining intelligence and power. We will cover:

1. Installing LeJOS, the Java Virtual Machine designed for the LEGO Mindstorms programmable controller
2. Setting up your Java platform development environment for LEGO Mindstorms
3. Programming rotation and custom sensors.
4. Using behavior control programming
5. Gathering map data and transmitting it back to your computer"

Formal lexical, syntactic and semantic descriptions, compilation and implementation issues, and t... more Formal lexical, syntactic and semantic descriptions, compilation and implementation issues, and theoretical foundations for programming paradigms. There are four programming assignments in this course: a Lexical Analyzer, a Parser, a Semantics Analyzer and an Intermediate Code Generator. The four assignments bundled together result in a compiler for a custom programming language.