Alice Oh | KAIST - Academia.edu (original) (raw)

Papers by Alice Oh

Multiplayer online battle arena games provide an excellent opportunity to study team performance.... more Multiplayer online battle arena games provide an excellent opportunity to study team performance. When designing a team, players must negotiate a proficiency-congruency dilemma between selecting roles that best match their experience and roles that best complement the existing roles on the team. We adopt a mixed-methods approach to explore how users negotiate this dilemma. Using data from League of Legends, we define a similarity space to operationalize team design constructs about role proficiency, generality, and congruency. We collect publicly available data from 3.36 million users to test the influence of these constructs on team performance. We also conduct focus groups with novice and elite players to understand how players' team design practices vary with expertise. We find that player proficiency increases team performance more than team congruency. These findings have implications for players, designers, and theorists about how to recommend team designs that jointly prioritize individuals' expertise and teams' compatibility.

arXiv (Cornell University), Oct 25, 2022

Research on Korean grammatical error correction (GEC) is limited, compared to other major languag... more Research on Korean grammatical error correction (GEC) is limited, compared to other major languages such as English. We attribute this problematic circumstance to the lack of a carefully designed evaluation benchmark for Korean GEC. In this work, we collect three datasets from different sources (Kor-Lang8, Kor-Native, and Kor-Learner) that covers a wide range of Korean grammatical errors. Considering the nature of Korean grammar, We then define 14 error types for Korean and provide KAGAS (Korean Automatic Grammatical error Annotation System), which can automatically annotate error types from parallel corpora. We use KAGAS on our datasets to make an evaluation benchmark for Korean, and present baseline models trained from our datasets. We show that the model trained with our datasets significantly outperforms the currently used statistical Korean GEC system (Hanspell) on a wider range of error types, demonstrating the diversity and usefulness of the datasets. The implementations and datasets are open-sourced. 1

Proceedings of the International AAAI Conference on Web and Social Media

We present a computational framework for understanding the social aspects of emotions in Twitter ... more We present a computational framework for understanding the social aspects of emotions in Twitter conversations. Using unannotated data and semisupervised machine learning, we look at emotional transitions, emotional influences among the conversation partners, and patterns in the overall emotional exchanges. We find that conversational partners usually express the same emotion, which we name Emotion accommodation, but when they do not, one of the conversational partners tends to respond with a positive emotion. We also show that tweets containing sympathy, apology, and complaint are significant emotion influencers. We verify the emotion classification part of our framework by a human-annotated corpus.

Proceedings of the International AAAI Conference on Web and Social Media

For many people who speak more than one language,their language proficiency for each of the langu... more For many people who speak more than one language,their language proficiency for each of the languagesvaries. We can conjecture that people who use onelanguage (primary language) more than another wouldshow higher language proficiency in that primary language.It is, however, difficult to observe and quantifythat problem because natural language use is difficultto collect in large amounts. We identify Wikipedia asa great resource for studying multilingualism, and weconduct a quantitative analysis of the language complexityof primary and non-primary users of English,German, and Spanish. Our preliminary results indicatethat there are indeed consistent differences of languagecomplexity in the Wikipedia articles chosen by primaryand non-primary users, as well as differences in the editsby the two groups of users.

6th European Conference on Speech Communication and Technology (Eurospeech 1999)

The Carnegie Mellon Communicator system helps users create complex travel itineraries through a c... more The Carnegie Mellon Communicator system helps users create complex travel itineraries through a conversational interface. Itineraries consist of (multi-leg) flights, hotel and car reservations and are built from actual travel information for North America, obtained from the Web. The system manages dialog using a schema-based approach. Schemas correspond to major units of task information (such as a flight leg) and define conversational topics, or foci of interaction, meaningful to the user.

6th International Conference on Spoken Language Processing (ICSLP 2000)

The Carnegie Mellon Communicator is a telephone-based dialog system that supports planning in a t... more The Carnegie Mellon Communicator is a telephone-based dialog system that supports planning in a travel domain. The implementation of such a system requires two complimentary components, an architecture capable of managing interaction and the task, as well as a knowledge base that captures the speech, language and task characteristics specific to the domain. Given a suitable architecture, the principal effort in development in taken up in the acquisition and processing of a domain knowledge base. This paper describes a variety of techniques we have applied to modeling in acoustic, language, task, generation and synthesis components of the system.

Proceedings of the Fifth Annual ACM Conference on Learning at Scale

In programming education, instructors often supplement lectures with active learning experiences ... more In programming education, instructors often supplement lectures with active learning experiences by offering programming lab sessions where learners themselves practice writing code. However, widely accessed instructional programming screencasts are not equipped with assessment format that encourages such hands-on programming activities. We introduce Elicast, a screencast tool for recording and viewing programming lectures with embedded programming exercises, to provide hands-on programming experiences in the screen-cast. In Elicast, instructors embed multiple programming exercises while creating a screencast, and learners engage in the exercises by writing code within the screencast, receiving auto-graded results immediately. We conducted an exploratory study of Elicast with five experienced instructors and 63 undergraduate students. We found that instructors structured the lectures into small learning units using embedded exercises as checkpoints. Also, learners more actively engaged in the screencast lectures, checked their understanding of the content through the embedded exercises, and more frequently modified and executed the code during the lectures.

arXiv (Cornell University), May 20, 2021

Multiplayer online battle arena games provide an excellent opportunity to study team performance.... more Multiplayer online battle arena games provide an excellent opportunity to study team performance. When designing a team, players must negotiate a proficiency-congruency dilemma between selecting roles that best match their experience and roles that best complement the existing roles on the team. We adopt a mixed-methods approach to explore how users negotiate this dilemma. Using data from League of Legends, we define a similarity space to operationalize team design constructs about role proficiency, generality, and congruency. We collect publicly available data from 3.36 million users to test the influence of these constructs on team performance. We also conduct focus groups with novice and elite players to understand how players' team design practices vary with expertise. We find that player proficiency increases team performance more than team congruency. These findings have implications for players, designers, and theorists about how to recommend team designs that jointly pri...

Individuals who successfully make their livelihood by talking with others, for example travel age... more Individuals who successfully make their livelihood by talking with others, for example travel agents, can be presumed to have optimized their language for the task at hand in terms of conciseness and intelligibility. It makes sense to exploit this effort for the purpose of building better generation components for a spoken dialog system. The Stochastic Generation technique, introduced by<br>Oh and Rudnicky (2002), is one such approach. In this approach, utterances in a corpus of domain expert utterances are classified as to speech act and individual concepts tagged. Statistical n-gram models are built for each speech-act class then used generatively to create novel utterances. These have been shown to be comparable in quality to human productions. The class and tag scheme is concrete and closely tied to the domain at hand; we believe this produces a distinct advantage in speed of implementation and quality of<br>results. The current paper describes the classification and...

Findings of the Association for Computational Linguistics: EMNLP 2021, 2021

Finding counterevidence to statements is key to many tasks, including counterargument generation.... more Finding counterevidence to statements is key to many tasks, including counterargument generation. We build a system that, given a statement, retrieves counterevidence from diverse sources on the Web. At the core of this system is a natural language inference (NLI) model that determines whether a candidate sentence is valid counterevidence or not. Most NLI models to date, however, lack proper reasoning abilities necessary to find counterevidence that involves complex inference. Thus, we present a knowledge-enhanced NLI model that aims to handle causality-and example-based inference by incorporating knowledge graphs. Our NLI model outperforms baselines for NLI tasks, especially for instances that require the targeted inference. In addition, this NLI model further improves the counterevidence retrieval system, notably finding complex counterevidence better. 1 Type Description Examples Mainstream News Mainstream news about daily issues and general topics. www.cnn.com, www.bbc.com Research Journal Peer-reviewed papers or dissertations. link.springer.com, www.nature.com Report Surveys, statistics, and reports. Should be a source of substantial data rather than a summary of reports.

Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019

An overwhelming number of true and false news stories are posted and shared in social networks, a... more An overwhelming number of true and false news stories are posted and shared in social networks, and users diffuse the stories based on multiple factors. Diffusion of news stories from one user to another depends not only on the stories' content and the genuineness but also on the alignment of the topical interests between the users. In this paper, we propose a novel Bayesian nonparametric model that incorporates homogeneity of news stories as the key component that regulates the topical similarity between the posting and sharing users' topical interests. Our model extends hierarchical Dirichlet process to model the topics of the news stories and incorporates Bayesian Gaussian process latent variable model to discover the homogeneity values. We train our model on a real-world social network dataset and find homogeneity values of news stories that strongly relate to their labels of genuineness and their contents. Finally, we show that the supervised version of our model predicts the labels of news stories better than the state-of-the-art neural network and Bayesian models.

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020

Automatic evaluation of open-domain dialogue response generation is very challenging because ther... more Automatic evaluation of open-domain dialogue response generation is very challenging because there are many appropriate responses for a given context. Existing evaluation models merely compare the generated response with the ground truth response and rate many of the appropriate responses as inappropriate if they deviate from the ground truth. One approach to resolve this problem is to consider the similarity of the generated response with the conversational context. In this paper, we propose an automatic evaluation model based on that idea and learn the model parameters from an unlabeled conversation corpus. Our approach considers the speakers in defining the different levels of similar context. We use a Twitter conversation corpus that contains many speakers and conversations to test our evaluation model. Experiments show that our model outperforms the other existing evaluation metrics in terms of high correlation with human annotation scores. We also show that our model trained on Twitter can be applied to movie dialogues without any additional training. We provide our code and the learned parameters so that they can be used for automatic evaluation of dialogue response generation models.

Machine Learning, 2017

We present the hierarchical Dirichlet scaling process (HDSP), a Bayesian nonparametric mixed memb... more We present the hierarchical Dirichlet scaling process (HDSP), a Bayesian nonparametric mixed membership model. The HDSP generalizes the hierarchical Dirichlet process to model the correlation structure between metadata in the corpus and mixture components. We construct the HDSP based on the normalized gamma representation of the Dirichlet process, and this construction allows incorporating a scaling function that controls the membership probabilities of the mixture components. We develop two scaling methods to demonstrate that different modeling assumptions can be expressed in the HDSP. We also derive the corresponding approximate posterior inference algorithms using variational Bayes. Through experiments on datasets of newswire, medical journal articles, conference proceedings, and product reviews, we show that the HDSP results in a better predictive performance than labeled LDA, partially labeled LDA, and author topic model and a better negative review classification performance than the supervised topic model and SVM.

Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016

Headlines are particularly important for online news outlets where there are many similar news st... more Headlines are particularly important for online news outlets where there are many similar news stories competing for users' attention. Traditionally, journalists have followed rules-of-thumb and experience to master the art of crafting catchy headlines, but with the valuable resource of largescale click-through data of online news articles, we can apply quantitative analysis and text mining techniques to acquire an in-depth understanding of headlines. In this paper, we conduct a large-scale analysis and modeling of 150K news articles published over a period of four months on the Yahoo home page. We define a simple method to measure clickvalue of individual words, and analyze how temporal trends and linguistic attributes affect click-through rate (CTR). We then propose a novel generative model, headline click-based topic model (HCTM), that extends latent Dirichlet allocation (LDA) to reveal the effect of topical context on the click-value of words in headlines. HCTM leverages clicks in aggregate on previously published headlines to identify words for headlines that will generate more clicks in the future. We show that by jointly taking topics and clicks into account we can detect changes in user interests within topics. We evaluate HCTM in two different experimental settings and compare its performance with ALDA (adapted LDA), LDA, and TextRank. The first task, full headline, is to retrieve full headline used for a news article given the body of news article. The second task, good headline, is to specifically identify words in the headline that have high click values for current news audience. For full headline task, our model performs on par with ALDA, a state-of-the art web-page summarization method that utilizes click-through information. For good headline task, which is of more practical importance to both individual journalists and online news outlets, our model significantly outperforms all other comparative methods.

PLOS ONE, 2016

Multilingualism is common offline, but we have a more limited understanding of the ways multiling... more Multilingualism is common offline, but we have a more limited understanding of the ways multilingualism is displayed online and the roles that multilinguals play in the spread of content between speakers of different languages. We take a computational approach to studying multilingualism using one of the largest usergenerated content platforms, Wikipedia. We study multilingualism by collecting and analyzing a large dataset of the content written by multilingual editors of the English, German, and Spanish editions of Wikipedia. This dataset contains over two million paragraphs edited by over 15,000 multilingual users from July 8 to August 9, 2013. We analyze these multilingual editors in terms of their engagement, interests, and language proficiency in their primary and non-primary (secondary) languages and find that the English edition of Wikipedia displays different dynamics from the Spanish and German editions. Users primarily editing the Spanish and German editions make more complex edits than users who edit these editions as a second language. In contrast, users editing the English edition as a second language make edits that are just as complex as the edits by users who primarily edit the English edition. In this way, English serves a special role bringing together content written by multilinguals from many language editions. Nonetheless, language remains a formidable hurdle to the spread of content: we find evidence for a complexity barrier whereby editors are less likely to edit complex content in a second language. In addition, we find that multilinguals are less engaged and show lower levels of language proficiency in their second languages. We also examine the topical interests of multilingual editors and find that there is no significant difference between primary and non-primary editors in each language.

Proceedings of the Joint Workshop on Social Dynamics and Personal Attributes in Social Media, 2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014

Self-disclosure, the act of revealing oneself to others, is an important social behavior that str... more Self-disclosure, the act of revealing oneself to others, is an important social behavior that strengthens interpersonal relationships and increases social support. Although there are many social science studies of self-disclosure, they are based on manual coding of small datasets and questionnaires. We conduct a computational analysis of self-disclosure with a large dataset of naturally-occurring conversations, a semi-supervised machine learning algorithm, and a computational analysis of the effects of self-disclosure on subsequent conversations. We use a longitudinal dataset of 17 million tweets, all of which occurred in conversations that consist of five or more tweets directly replying to the previous tweet, and from dyads with twenty of more conversations each. We develop self-disclosure topic model (SDTM), a variant of latent Dirichlet allocation (LDA) for automatically classifying the level of self-disclosure for each tweet. We take the results of SDTM and analyze the effects of self-disclosure on subsequent conversations. Our model significantly outperforms several comparable methods on classifying the level of selfdisclosure, and the analysis of the longitudinal data using SDTM uncovers significant and positive correlation between selfdisclosure and conversation frequency and length.

Proceedings of the ANLP-NAACL 2000 Workshop on Conversational Systems - ConversationalSys '00, 2000

The two current approaches to language generation, Template-based and rule-based (linguistic) NLG... more The two current approaches to language generation, Template-based and rule-based (linguistic) NLG, have limitations when applied to spoken dialogue systems, in part because they were developed for text generation. In this paper, we propose a new corpus-based approach to natural language generation, specifically designed for spoken dialogue systems.

2010 IEEE Second International Conference on Social Computing, 2010

Weblogs (blogs) serve as a gateway to a large blog reader population, so blog authors can potenti... more Weblogs (blogs) serve as a gateway to a large blog reader population, so blog authors can potentially influence a large reader population by expressing their thoughts and expertise in their blog posts. An important and complex problem, then, is figuring out why and how influence propagates through the blogosphere. While a number of previous research has looked at the network characteristics of blogs to analyze influence propagation through the blogspace, we hypothesize that a blog's influence depends on its contents as well as its network positions. Thus, in this paper, we explore two different influence propagation metrics showing different influence characteristics: Digg score and comment counts. Then, we present the results of our experiments to predict the level of influence propagation of a blog by applying machine learning algorithms to its contents and network positions. We observed over 70,000 blog posts, pruned from over 20,000,000 posts, and we found that the prediction accuracy using the content and the network features simultaneously shows the best F-score in various measures. We expect that this research result will contribute to understanding the problem of influence propagation through the blogosphere, and to developing applications for recommending influential blogs to social web users.

Multiplayer online battle arena games provide an excellent opportunity to study team performance.... more Multiplayer online battle arena games provide an excellent opportunity to study team performance. When designing a team, players must negotiate a proficiency-congruency dilemma between selecting roles that best match their experience and roles that best complement the existing roles on the team. We adopt a mixed-methods approach to explore how users negotiate this dilemma. Using data from League of Legends, we define a similarity space to operationalize team design constructs about role proficiency, generality, and congruency. We collect publicly available data from 3.36 million users to test the influence of these constructs on team performance. We also conduct focus groups with novice and elite players to understand how players' team design practices vary with expertise. We find that player proficiency increases team performance more than team congruency. These findings have implications for players, designers, and theorists about how to recommend team designs that jointly prioritize individuals' expertise and teams' compatibility.

arXiv (Cornell University), Oct 25, 2022

Research on Korean grammatical error correction (GEC) is limited, compared to other major languag... more Research on Korean grammatical error correction (GEC) is limited, compared to other major languages such as English. We attribute this problematic circumstance to the lack of a carefully designed evaluation benchmark for Korean GEC. In this work, we collect three datasets from different sources (Kor-Lang8, Kor-Native, and Kor-Learner) that covers a wide range of Korean grammatical errors. Considering the nature of Korean grammar, We then define 14 error types for Korean and provide KAGAS (Korean Automatic Grammatical error Annotation System), which can automatically annotate error types from parallel corpora. We use KAGAS on our datasets to make an evaluation benchmark for Korean, and present baseline models trained from our datasets. We show that the model trained with our datasets significantly outperforms the currently used statistical Korean GEC system (Hanspell) on a wider range of error types, demonstrating the diversity and usefulness of the datasets. The implementations and datasets are open-sourced. 1

Proceedings of the International AAAI Conference on Web and Social Media

We present a computational framework for understanding the social aspects of emotions in Twitter ... more We present a computational framework for understanding the social aspects of emotions in Twitter conversations. Using unannotated data and semisupervised machine learning, we look at emotional transitions, emotional influences among the conversation partners, and patterns in the overall emotional exchanges. We find that conversational partners usually express the same emotion, which we name Emotion accommodation, but when they do not, one of the conversational partners tends to respond with a positive emotion. We also show that tweets containing sympathy, apology, and complaint are significant emotion influencers. We verify the emotion classification part of our framework by a human-annotated corpus.

Proceedings of the International AAAI Conference on Web and Social Media

For many people who speak more than one language,their language proficiency for each of the langu... more For many people who speak more than one language,their language proficiency for each of the languagesvaries. We can conjecture that people who use onelanguage (primary language) more than another wouldshow higher language proficiency in that primary language.It is, however, difficult to observe and quantifythat problem because natural language use is difficultto collect in large amounts. We identify Wikipedia asa great resource for studying multilingualism, and weconduct a quantitative analysis of the language complexityof primary and non-primary users of English,German, and Spanish. Our preliminary results indicatethat there are indeed consistent differences of languagecomplexity in the Wikipedia articles chosen by primaryand non-primary users, as well as differences in the editsby the two groups of users.

6th European Conference on Speech Communication and Technology (Eurospeech 1999)

The Carnegie Mellon Communicator system helps users create complex travel itineraries through a c... more The Carnegie Mellon Communicator system helps users create complex travel itineraries through a conversational interface. Itineraries consist of (multi-leg) flights, hotel and car reservations and are built from actual travel information for North America, obtained from the Web. The system manages dialog using a schema-based approach. Schemas correspond to major units of task information (such as a flight leg) and define conversational topics, or foci of interaction, meaningful to the user.

6th International Conference on Spoken Language Processing (ICSLP 2000)

The Carnegie Mellon Communicator is a telephone-based dialog system that supports planning in a t... more The Carnegie Mellon Communicator is a telephone-based dialog system that supports planning in a travel domain. The implementation of such a system requires two complimentary components, an architecture capable of managing interaction and the task, as well as a knowledge base that captures the speech, language and task characteristics specific to the domain. Given a suitable architecture, the principal effort in development in taken up in the acquisition and processing of a domain knowledge base. This paper describes a variety of techniques we have applied to modeling in acoustic, language, task, generation and synthesis components of the system.

Proceedings of the Fifth Annual ACM Conference on Learning at Scale

In programming education, instructors often supplement lectures with active learning experiences ... more In programming education, instructors often supplement lectures with active learning experiences by offering programming lab sessions where learners themselves practice writing code. However, widely accessed instructional programming screencasts are not equipped with assessment format that encourages such hands-on programming activities. We introduce Elicast, a screencast tool for recording and viewing programming lectures with embedded programming exercises, to provide hands-on programming experiences in the screen-cast. In Elicast, instructors embed multiple programming exercises while creating a screencast, and learners engage in the exercises by writing code within the screencast, receiving auto-graded results immediately. We conducted an exploratory study of Elicast with five experienced instructors and 63 undergraduate students. We found that instructors structured the lectures into small learning units using embedded exercises as checkpoints. Also, learners more actively engaged in the screencast lectures, checked their understanding of the content through the embedded exercises, and more frequently modified and executed the code during the lectures.

arXiv (Cornell University), May 20, 2021

Multiplayer online battle arena games provide an excellent opportunity to study team performance.... more Multiplayer online battle arena games provide an excellent opportunity to study team performance. When designing a team, players must negotiate a proficiency-congruency dilemma between selecting roles that best match their experience and roles that best complement the existing roles on the team. We adopt a mixed-methods approach to explore how users negotiate this dilemma. Using data from League of Legends, we define a similarity space to operationalize team design constructs about role proficiency, generality, and congruency. We collect publicly available data from 3.36 million users to test the influence of these constructs on team performance. We also conduct focus groups with novice and elite players to understand how players' team design practices vary with expertise. We find that player proficiency increases team performance more than team congruency. These findings have implications for players, designers, and theorists about how to recommend team designs that jointly pri...

Individuals who successfully make their livelihood by talking with others, for example travel age... more Individuals who successfully make their livelihood by talking with others, for example travel agents, can be presumed to have optimized their language for the task at hand in terms of conciseness and intelligibility. It makes sense to exploit this effort for the purpose of building better generation components for a spoken dialog system. The Stochastic Generation technique, introduced by<br>Oh and Rudnicky (2002), is one such approach. In this approach, utterances in a corpus of domain expert utterances are classified as to speech act and individual concepts tagged. Statistical n-gram models are built for each speech-act class then used generatively to create novel utterances. These have been shown to be comparable in quality to human productions. The class and tag scheme is concrete and closely tied to the domain at hand; we believe this produces a distinct advantage in speed of implementation and quality of<br>results. The current paper describes the classification and...

Findings of the Association for Computational Linguistics: EMNLP 2021, 2021

Finding counterevidence to statements is key to many tasks, including counterargument generation.... more Finding counterevidence to statements is key to many tasks, including counterargument generation. We build a system that, given a statement, retrieves counterevidence from diverse sources on the Web. At the core of this system is a natural language inference (NLI) model that determines whether a candidate sentence is valid counterevidence or not. Most NLI models to date, however, lack proper reasoning abilities necessary to find counterevidence that involves complex inference. Thus, we present a knowledge-enhanced NLI model that aims to handle causality-and example-based inference by incorporating knowledge graphs. Our NLI model outperforms baselines for NLI tasks, especially for instances that require the targeted inference. In addition, this NLI model further improves the counterevidence retrieval system, notably finding complex counterevidence better. 1 Type Description Examples Mainstream News Mainstream news about daily issues and general topics. www.cnn.com, www.bbc.com Research Journal Peer-reviewed papers or dissertations. link.springer.com, www.nature.com Report Surveys, statistics, and reports. Should be a source of substantial data rather than a summary of reports.

Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019

An overwhelming number of true and false news stories are posted and shared in social networks, a... more An overwhelming number of true and false news stories are posted and shared in social networks, and users diffuse the stories based on multiple factors. Diffusion of news stories from one user to another depends not only on the stories' content and the genuineness but also on the alignment of the topical interests between the users. In this paper, we propose a novel Bayesian nonparametric model that incorporates homogeneity of news stories as the key component that regulates the topical similarity between the posting and sharing users' topical interests. Our model extends hierarchical Dirichlet process to model the topics of the news stories and incorporates Bayesian Gaussian process latent variable model to discover the homogeneity values. We train our model on a real-world social network dataset and find homogeneity values of news stories that strongly relate to their labels of genuineness and their contents. Finally, we show that the supervised version of our model predicts the labels of news stories better than the state-of-the-art neural network and Bayesian models.

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020

Automatic evaluation of open-domain dialogue response generation is very challenging because ther... more Automatic evaluation of open-domain dialogue response generation is very challenging because there are many appropriate responses for a given context. Existing evaluation models merely compare the generated response with the ground truth response and rate many of the appropriate responses as inappropriate if they deviate from the ground truth. One approach to resolve this problem is to consider the similarity of the generated response with the conversational context. In this paper, we propose an automatic evaluation model based on that idea and learn the model parameters from an unlabeled conversation corpus. Our approach considers the speakers in defining the different levels of similar context. We use a Twitter conversation corpus that contains many speakers and conversations to test our evaluation model. Experiments show that our model outperforms the other existing evaluation metrics in terms of high correlation with human annotation scores. We also show that our model trained on Twitter can be applied to movie dialogues without any additional training. We provide our code and the learned parameters so that they can be used for automatic evaluation of dialogue response generation models.

Machine Learning, 2017

We present the hierarchical Dirichlet scaling process (HDSP), a Bayesian nonparametric mixed memb... more We present the hierarchical Dirichlet scaling process (HDSP), a Bayesian nonparametric mixed membership model. The HDSP generalizes the hierarchical Dirichlet process to model the correlation structure between metadata in the corpus and mixture components. We construct the HDSP based on the normalized gamma representation of the Dirichlet process, and this construction allows incorporating a scaling function that controls the membership probabilities of the mixture components. We develop two scaling methods to demonstrate that different modeling assumptions can be expressed in the HDSP. We also derive the corresponding approximate posterior inference algorithms using variational Bayes. Through experiments on datasets of newswire, medical journal articles, conference proceedings, and product reviews, we show that the HDSP results in a better predictive performance than labeled LDA, partially labeled LDA, and author topic model and a better negative review classification performance than the supervised topic model and SVM.

Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016

Headlines are particularly important for online news outlets where there are many similar news st... more Headlines are particularly important for online news outlets where there are many similar news stories competing for users' attention. Traditionally, journalists have followed rules-of-thumb and experience to master the art of crafting catchy headlines, but with the valuable resource of largescale click-through data of online news articles, we can apply quantitative analysis and text mining techniques to acquire an in-depth understanding of headlines. In this paper, we conduct a large-scale analysis and modeling of 150K news articles published over a period of four months on the Yahoo home page. We define a simple method to measure clickvalue of individual words, and analyze how temporal trends and linguistic attributes affect click-through rate (CTR). We then propose a novel generative model, headline click-based topic model (HCTM), that extends latent Dirichlet allocation (LDA) to reveal the effect of topical context on the click-value of words in headlines. HCTM leverages clicks in aggregate on previously published headlines to identify words for headlines that will generate more clicks in the future. We show that by jointly taking topics and clicks into account we can detect changes in user interests within topics. We evaluate HCTM in two different experimental settings and compare its performance with ALDA (adapted LDA), LDA, and TextRank. The first task, full headline, is to retrieve full headline used for a news article given the body of news article. The second task, good headline, is to specifically identify words in the headline that have high click values for current news audience. For full headline task, our model performs on par with ALDA, a state-of-the art web-page summarization method that utilizes click-through information. For good headline task, which is of more practical importance to both individual journalists and online news outlets, our model significantly outperforms all other comparative methods.

PLOS ONE, 2016

Multilingualism is common offline, but we have a more limited understanding of the ways multiling... more Multilingualism is common offline, but we have a more limited understanding of the ways multilingualism is displayed online and the roles that multilinguals play in the spread of content between speakers of different languages. We take a computational approach to studying multilingualism using one of the largest usergenerated content platforms, Wikipedia. We study multilingualism by collecting and analyzing a large dataset of the content written by multilingual editors of the English, German, and Spanish editions of Wikipedia. This dataset contains over two million paragraphs edited by over 15,000 multilingual users from July 8 to August 9, 2013. We analyze these multilingual editors in terms of their engagement, interests, and language proficiency in their primary and non-primary (secondary) languages and find that the English edition of Wikipedia displays different dynamics from the Spanish and German editions. Users primarily editing the Spanish and German editions make more complex edits than users who edit these editions as a second language. In contrast, users editing the English edition as a second language make edits that are just as complex as the edits by users who primarily edit the English edition. In this way, English serves a special role bringing together content written by multilinguals from many language editions. Nonetheless, language remains a formidable hurdle to the spread of content: we find evidence for a complexity barrier whereby editors are less likely to edit complex content in a second language. In addition, we find that multilinguals are less engaged and show lower levels of language proficiency in their second languages. We also examine the topical interests of multilingual editors and find that there is no significant difference between primary and non-primary editors in each language.

Proceedings of the Joint Workshop on Social Dynamics and Personal Attributes in Social Media, 2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014

Self-disclosure, the act of revealing oneself to others, is an important social behavior that str... more Self-disclosure, the act of revealing oneself to others, is an important social behavior that strengthens interpersonal relationships and increases social support. Although there are many social science studies of self-disclosure, they are based on manual coding of small datasets and questionnaires. We conduct a computational analysis of self-disclosure with a large dataset of naturally-occurring conversations, a semi-supervised machine learning algorithm, and a computational analysis of the effects of self-disclosure on subsequent conversations. We use a longitudinal dataset of 17 million tweets, all of which occurred in conversations that consist of five or more tweets directly replying to the previous tweet, and from dyads with twenty of more conversations each. We develop self-disclosure topic model (SDTM), a variant of latent Dirichlet allocation (LDA) for automatically classifying the level of self-disclosure for each tweet. We take the results of SDTM and analyze the effects of self-disclosure on subsequent conversations. Our model significantly outperforms several comparable methods on classifying the level of selfdisclosure, and the analysis of the longitudinal data using SDTM uncovers significant and positive correlation between selfdisclosure and conversation frequency and length.

Proceedings of the ANLP-NAACL 2000 Workshop on Conversational Systems - ConversationalSys '00, 2000

The two current approaches to language generation, Template-based and rule-based (linguistic) NLG... more The two current approaches to language generation, Template-based and rule-based (linguistic) NLG, have limitations when applied to spoken dialogue systems, in part because they were developed for text generation. In this paper, we propose a new corpus-based approach to natural language generation, specifically designed for spoken dialogue systems.

2010 IEEE Second International Conference on Social Computing, 2010

Weblogs (blogs) serve as a gateway to a large blog reader population, so blog authors can potenti... more Weblogs (blogs) serve as a gateway to a large blog reader population, so blog authors can potentially influence a large reader population by expressing their thoughts and expertise in their blog posts. An important and complex problem, then, is figuring out why and how influence propagates through the blogosphere. While a number of previous research has looked at the network characteristics of blogs to analyze influence propagation through the blogspace, we hypothesize that a blog's influence depends on its contents as well as its network positions. Thus, in this paper, we explore two different influence propagation metrics showing different influence characteristics: Digg score and comment counts. Then, we present the results of our experiments to predict the level of influence propagation of a blog by applying machine learning algorithms to its contents and network positions. We observed over 70,000 blog posts, pruned from over 20,000,000 posts, and we found that the prediction accuracy using the content and the network features simultaneously shows the best F-score in various measures. We expect that this research result will contribute to understanding the problem of influence propagation through the blogosphere, and to developing applications for recommending influential blogs to social web users.