Tell Me About Yourself (original) (raw)

Abstract

The rise of increasingly more powerful chatbots offers a new way to collect information through conversational surveys, where a chatbot asks open-ended questions, interprets a user’s free-text responses, and probes answers whenever needed. To investigate the effectiveness and limitations of such a chatbot in conducting surveys, we conducted a field study involving about 600 participants. In this study with mostly open-ended questions, half of the participants took a typical online survey on Qualtrics and the other half interacted with an AI-powered chatbot to complete a conversational survey. Our detailed analysis of over 5,200 free-text responses revealed that the chatbot drove a significantly higher level of participant engagement and elicited significantly better quality responses measured by Gricean Maxims in terms of their informativeness, relevance, specificity, and clarity. Based on our results, we discuss design implications for creating AI-powered chatbots to conduct effective surveys and beyond.

References

[1]

David A. Aaker. 1976. Design and analysis: A researcher’s handbook. Journal of Marketing Research 13, 000003 (1976), 318.

[2]

James F. Allen, Donna K. Byron, Myroslava Dzikovska, George Ferguson, Lucian Galescu, and Amanda Stent. 2001. Toward conversational human-computer interaction. AI Magazine 22, 4 (2001), 27.

[3]

Irwin Altman and Dalmas A. Taylor. 1973. Social Penetration: The Development of Interpersonal Relationships. Holt, Rinehart 8 Winston.

[4]

Christopher Antoun, Mick P. Couper, and Frederick G. Conrad. 2017. Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly 81, S1 (2017), 280--306.

[5]

Christoph Bartneck, Dana Kulić, Elizabeth Croft, and Susana Zoghbi. 2009. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics 1, 1 (2009), 71--81.

[6]

C. Bauer, K. Figl, and R. Motschnig-Pitrik. 2010. Introducing ‘active listening’ to instant messaging and e-mail: Benefits and limitations. IADIS International Journal on WWW/Internet 7, 2 (2010), 1--17.

[7]

D. Behr, L. Kaczmirek, W. Bandilla, and M. Braun. 2012. Asking probing questions in web surveys. Journal of Social Science Computer Review 30, 4 (2012), 487--498.

[8]

Pazit Ben-Nun. 2008. Respondent fatigue. Encyclopedia of Survey Research Methods 2 (2008), 742--743.

[9]

A. Bianchi and J. G. Phillips. 2005. Psychological predictors of problem mobile phone use. CyberPsychology 8 Behavior 8, 1 (2005), 39--51.

[10]

Timothy Bickmore and Justine Cassell. 2001. Relational agents: A model and implementation of building user trust. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 396--403.

[11]

Timothy Bickmore, Laura Pfeifer, and Daniel Schulman. 2011. Relational agents improve engagement and learning in science museum visitors. In Proceedings of the International Workshop on Intelligent Virtual Agents. Springer, 55--67.

[12]

Timothy Bickmore, Daniel Schulman, and Langxuan Yin. 2010. Maintaining engagement in long-term interventions with relational agents. Applied Artificial Intelligence 24, 6 (2010), 648--666.

[13]

Dan Bohus and Alexander I. Rudnicky. 2009. The ravenclaw dialog management framework: Architecture and systems. Computer Speech 8 Language 23, 3 (2009), 332--361.

[14]

Susan E. Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. (1990).

[15]

A. Carstens and J. Beck. 2004. Get ready for the gamer generation. TechTrends 49, 3 (2004), 22--25.

[16]

Justine Cassell and Timothy Bickmore. 2000. External manifestations of trustworthiness in the interface. Communications of the ACM 43, 12 (2000), 50--56.

[17]

Yukina Chen. 2017. The Effects of Question Customization on the Quality of an Open-Ended Question. Nebraska Department of Education, Data, Research, and Evaluation.

[18]

Frederick G. Conrad, Mick P. Couper, Roger Tourangeau, Mirta Galesic, and T. Yan. 2005. Interactive feedback can improve the quality of responses in web surveys. In Proceedings of the Section on Survey Research Methods. 3835--3840.

[19]

Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andrés Monroy-Hernández. 2017. Calendar. help: Designing a workflow-based scheduling agent with humans in the loop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2382--2393.

[20]

Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. Superagent: A customer service chatbot for e-commerce websites. In Proceedings of the ACL 2017, System Demonstrations. 97--102.

[21]

E. de Leeuw, J. Hox, and A. Boev. 2015. Handling do-not-know answers: Exploring new approaches in online and mixed-mode surveys. Journal of Social Science Computer Review 34, 1 (2015), 116--132.

[22]

David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. SimSensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068.

[23]

Jeff Dunn. 2016. We put Siri, Alexa, Google Assistant, and Cortana through a marathon of tests to see who’s winning the virtual assistant race - here’s what we found. Business Insider.

[24]

Laila Dybkjaer, Niels Ole Bernsen, and Hans Dybkjaer. 1996. GRICE INCORPORATED: Cooperativity in spoken dialogue. In Proceedings of the 16th International Conference on Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/C96-1056.

[25]

Laila Dybkjaer, Niels Ole Bernsen, and Wolfgang Minker. 2004. Evaluation and usability of multimodal spoken language dialogue systems. Speech Communication 43, 1–2 (2004), 33--54.

[26]

S. Egelman, E. Chi, and S. Dow. 2014. Crowdsourcing in HCI research. In Ways of Knowing in HCI, J. Olson and W. Kellogg (Eds.). Springer, 267--289.

[27]

Joel R. Evans and Anil Mathur. 2005. The value of online surveys. Internet Research 15, 2 (2005), 195--219.

[28]

Asbjørn Følstad and Petter Bae Brandtzæg. 2017. Chatbots and the new world of HCI. Interactions 24, 4 (2017), 38--42.

[29]

Eric N. Forsythand and Craig H. Martell. 2007. Lexical and discourse analysis of online chat dialog. In Proceedings of the International Conference on Semantic Computing. IEEE, 19--26.

[30]

Ronald D. Fricker and Matthias Schonlau. 2002. Advantages and disadvantages of internet research surveys: Evidence from the literature. Field Methods 14, 4 (2002), 347--367.

[31]

Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1631--1640.

[32]

Arthur C. Graesser, Patrick Chipman, Brian C. Haynes, and Andrew Olney. 2005. AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education 48, 4 (2005), 612--618.

[33]

Herbert P. Grice. 1975. Logic and conversation. In Speech Acts. Brill, 41–58.

[34]

Jonathan Grudin and Richard Jacques. 2019. Chatbots, humbots, and the quest for artificial general intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 209.

[35]

Elisabeth Hayes and Maryellen Ohrnberger. 2013. The gamer generation teaches school: The gaming practices and attitudes towards technology of pre-service teachers. Journal of Technology and Teacher Education 21, 2 (2013), 154--177.

[36]

Dirk Heerwegh and Geert Loosveldt. 2006. Personalizing e-mail contacts: Its influence on web survey response rate and social desirability response bias. International Journal of Public Opinion Research 19, 2 (2006), 258--268.

[37]

Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Proceedings of the workshop on Speech and Natural Language.

[38]

J. B. Hirsh, S. K. Kang, and G. V. Bodenhausen. 2012. Personalized persuasion: Tailoring persuasive appeals to recipientsâ personality traits. Psychological Science 23, 6 (2012), 578--581.

[39]

Knut Hofland and Stig Johansson. 1982. Word Frequencies in British and American English. Norwegian Computing Centre for the Humanities.

[40]

Kate S. Hone and Robert Graham. 2000. Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering 6, 3–4 (2000), 287--303.

[41]

Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. 2015. Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 506--516.

[42]

Douglas Samuel Jones. 1979. Elementary Information Theory. Clarendon Press.

[43]

Jie Kang, Kyle Condiff, Shuo Chang, Joseph A. Konstan, Loren Terveen, and F. Maxwell Harper. 2017. Understanding how people use natural language to ask for recommendations. In Proceedings of the 11th ACM Conference on Recommender Systems. ACM, 229--237.

[44]

Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing data from chatbot and web surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19). ACM, New York, NY, 12.

[45]

Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability.

[46]

Jon A. Krosnick. 1999. Survey research. Annual Review of Psychology 50, 1 (1999), 537--567.

[47]

Paul J. Lavrakas. 2008. Encyclopedia of Survey Research Methods. Sage Publications.

[48]

Stephanie J. Law, Joshua Bourdage, and Thomas A O’Neill. 2016. To fake or not to fake: Antecedents to interview faking, warning instructions, and its impact on applicant reactions. Frontiers in Psychology 7 (2016), 1771.

[49]

Geoffrey Neil Leech. 1992. 100 million words of English: the british national corpus (BNC).

[50]

Jingyi Li, Michelle X. Zhou, Huahai Yang, and Gloria Mark. 2017. Confiding in and listening to virtual agents: The effect of personality. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. ACM, 275--286.

[51]

Vera Q. Liao, Muhammed Masud Hussain, Praveen Chandar, Matthew Davis, Marco Crasso, Dakuo Wang, Michael Muller, Sadat N. Shami, and Werner Geyer. 2018. All work and no play? conversations with a question-and-answer chatbot in the wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). ACM, New York, NY, Vol. 13.

[52]

Joseph Carl Robnett Licklider. 1960. Man-computer symbiosis. IRE Transactions on Human Factors in Electronics1 (1960), 4--11.

[53]

Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2122--2132.

[54]

S. Louw, R. W. Todd, and P. Jimakorn. 2010. Active listening in qualitative research interviews. In Proceedings of the Doing Research in Applied Linguistics. 71--82.

[55]

Gale M. Lucas, Jonathan Gratch, Aisha King, and Louis-Philippe Morency. 2014. It’s only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior 37 (2014), 94--100.

[56]

Ewa Luger and Abigail Sellen. 2016. Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5286--5297.

[57]

Aigul Mavletova. 2013. Data quality in PC and mobile web surveys. Social Science Computer Review 31, 6 (2013), 725--743.

[58]

Scott McGlashan, Norman Fraser, Nigel Gilbert, Eric Bilange, Paul Heisterkamp, and Nick Youd. 1992. Dialogue management for telephone information systems. In Proceedings of the 3rd Conference on Applied Natural Language Processing. Association for Computational Linguistics, 245--246.

[59]

Michael McTear, Zoraida Callejas, and David Griol. 2016. The Conversational Interface: Talking to Smart Devices. Springer.

[60]

Youngme Moon. 1998. Intimate Self-disclosure Exhanges: Using Computers to Build Reciprocal Relationships with Consumers. Division of Research, Harvard Business School.

[61]

Michael G. Morris and Viswanath Venkatesh. 2000. Age differences in technology adoption decisions: Implications for a changing work force. Personnel Psychology 53, 2 (2000), 375--403.

[62]

Hendrik Muller, Aaron Sedley, and Elizabeth Ferrall-Nunge. 2014. Survey research in HCI. In Ways of Knowing in HCI. Springer, 229--266.

[63]

David Novick and Iván Gris. 2014. Building rapport between human and ECA: A pilot study. In Proceedings of the International Conference on Human-Computer Interaction. Springer, 472--480.

[64]

Marije Oudejans and Leah Melani Christian. 2010. Using interactive features to motivate and probe responses to open-ended questions. In Social and Behavioral Research and the Internet. 215--244.

[65]

W. K. Park. 2005. Mobile phone addiction. In Mobile Communications. 253–272.

[66]

Tero Parviainen. 2010. teropa/nlp. Retrieved from https://github.com/teropa/nlp/tree/master/resources/corpora/webtext.

[67]

Richard Paul and Linda Elder. 2006. The Thinker’s Guide to the Art of Socratic Questioning. Foundation for Critical Thinking Dillon Beach, CA.

[68]

R. E. Petty, J. T. Cacioppo, and R. Goldman. 1981. Personal involvement as a determinant of argument-based persuasion. Journal of Personality and Social Psychology 41, 5 (1981), 847.

[69]

Stephen R. Porter. 2004. Overcoming survey research problems.

[70]

Stephen R. Porter. 2004. Raising response rates: What works? New Directions for Institutional Research 2004, 121 (2004), 5--21.

[71]

Stephen R. Porter, Michael E. Whitcomb, and William H. Weitzer. 2004. Multiple surveys of students and survey fatigue. New Directions for Institutional Research 2004, 121 (2004), 63--73.

[72]

Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval. ACM, 117--126.

[73]

Ursa Reja, Katja Lozar Manfreda, Valentina Hlebec, and Vasja Vehovar. 2003. Open-ended vs. close-ended questions in web questionnaires. Developments in Applied Statistics 19, 1 (2003), 159--177.

[74]

Francesco Ricci, Lior Rokach, and Bracha Shapira. 2015. Recommender systems: Introduction and challenges. In Recommender Systems Handbook. Springer, 1--34.

[75]

P. H. Rossi, J. D. Wright, and A. B. Anderson (Eds.). 2013. Handbook of Survey Research. Academic Press.

[76]

Shruti Sannon, Brett Stoll, Dominic DiFranzo, Malte Jung, and Natalya N. Bazarova. 2018. How personification and interactivity influence stress-related disclosures to conversational agents. In Proceedings of the Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 285--288.

[77]

Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Arturo Toledo, Ann Paradiso, Mary Czerwinski, Gloria Mark, and Marsha M. Linehan. 2018. Pocket skills: A conversational mobile web app to support dialectical behavioral therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 398.

[78]

Daniel Schulman and Timothy Bickmore. 2009. Persuading users through counseling dialogue with a conversational agent. In Proceedings of the 4th International Conference on Persuasive Technology. ACM, 25.

[79]

Ameneh Shamekhi, Q. Vera Liao, Dakuo Wang, Rachel K. E. Bellamy, and Thomas Erickson. 2018. Face value? Exploring the effects of embodiment for a group facilitation agent. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 391.

[80]

Eleanor Singer and Mick P. Couper. 2017. Some methodological uses of responses to open questions and other verbatim comments in quantitative surveys. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology 11, 2 (2017), 115--134.

[81]

S. Shyam Sundar and Jinyoung Kim. 2019. Machine heuristic: When we trust computers more than humans with our personal information. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 538.

[82]

Ella Tallyn, Hector Fried, Rory Gianni, Amy Isard, and Chris Speed. 2018. The ethnobot: Gathering ethnographies in the age of iot. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 604.

[83]

Paul Thomas, Mary Czerwinski, Daniel McDuff, Nick Craswell, and Gloria Mark. 2018. Style and alignment in information-seeking conversation. In Proceedings of the 2018 Conference on Human Information Interaction 8 Retrieval. ACM, 42--51.

[84]

Andrea L. Thomaz and Cynthia Breazeal. 2008. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence 172, 6–7 (2008), 716--737.

[85]

David Traum. 2017. Computational approaches to dialogue. In The Routledge Handbook of Language and Dialogue, Edda Weigand (Ed.). Routledge, 143–161.

[86]

Johanne R. Trippas, Damiano Spina, Lawrence Cavedon, Hideo Joho, and Mark Sanderson. 2018. Informing the design of spoken conversational search: Perspective paper. In Proceedings of the 2018 Conference on Human Information Interaction8Retrieval. ACM, 32--41.

[87]

Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin Hamilton, and Karrie Karahalios. 2018. The illusion of control: Placebo effects of control settings. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 16.

[88]

Viswanath Venkatesh, Michael G. Morris, and Phillip L. Ackerman. 2000. A longitudinal field investigation of gender differences in individual technology adoption decision-making processes. Organizational Behavior and Human Decision Processes 83, 1 (2000), 33--60.

[89]

Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of the 8th Conference of the European Chapter of the Association for Computational Linguistics.

[90]

Sproull L. and Subramani R. Walker, J.1996. When the interface is a face. Human-Computer Interaction 11, 2 (1996), 97--124.

[91]

Yang Wang, Liang Gou, Anbang Xu, Michelle X. Zhou, Huahai Yang, and Hernan Badenes. 2015. Veilme: An interactive visualization tool for privacy configuration of using personality traits. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 817--826.

[92]

Tom Wells, Justin T. Bailey, and Michael W. Link. 2014. Comparison of smartphone and online computer survey administration. Social Science Computer Review 32, 2 (2014), 238--255.

[93]

Alex C. Williams, Harmanpreet Kaur, Gloria Mark, Anne Loomis Thompson, Shamsi T. Iqbal, and Jaime Teevan. 2018. Supporting workplace detachment and reattachment with conversational intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 88.

[94]

Ziang Xiao, Michelle X. Zhou, and Wai-Tat Fu. 2019. Who should be my teammates: Using a conversational agent to understand individuals and help teaming. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI’19). ACM, New York, NY, 437--447.

[95]

Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3506--3510.

[96]

Yunfeng Zhang, Q. Vera Liao, and Biplav Srivastava. 2018. Towards an optimal dialog strategy for information retrieval using both open-and close-ended questions. In Proceedings of the 23rd International Conference on Intelligent User Interfaces. ACM, 365--369.

[97]

Michelle X. Zhou, Gloria Mark, Jingyi Li, and Huahai Yang. 2019. Trusting virtual agents: The effect of personality. ACM Transactions on Interactive Intelligent Systems 9, 2–3 (2019), 1--36.

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 27, Issue 3

June 2020

262 pages

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2020

Online AM: 07 May 2020

Accepted: 01 January 2020

Revised: 01 November 2019

Received: 01 May 2019

Published in TOCHI Volume 27, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Conversational agent
  2. chatbot
  3. open-ended questions
  4. survey

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

Reflects downloads up to 14 Oct 2024

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

Full Access

Media

Figures

Other

Tables

Affiliations

Ziang Xiao

University of Illinois at Urbana-Champaign

Michelle X. Zhou

Q. Vera Liao

Gloria Mark

Changyan Chi

Wenxi Chen

Huahai Yang