Marco Tulio Ribeiro – (original) (raw)

I'm a researcher at Google DeepMind. I am also an Affiliate Assistant Professor at the University of Washington, where I was previously a Ph.D student advised by Carlos Guestrin andSameer Singh.

My research is mostly on helping humans interact with machine learning models meaningfully. That involves interpretability, trust, debugging, feedback, etc.

Despite various attempts, I haven't made much progress on the (much harder) problem of getting a particular group of humans to all look at a camera at the same time →→→

Blog

I wrote these posts on how to pick a project for an intern, but I figured others might be interested too:

• Part 1: Coming up with research ideas
• Part 2: Organizing and evaluating research ideas

I also wrote this one on writing:

• Writing, part 1 — the process

Publications

• Sparks of artificial general intelligence: Early experiments with GPT-4
  Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Yi Zhang
• ScatterShot: Interactive In-context Example Curation for Text Transformation
  Tongshuang Wu, Hua Shen, Daniel Weld, Jeffrey Heer, Marco Tulio Ribeiro
  In: International Conference on Intelligent User Interfaces (IUI), 2023
  Best Paper Honorable mention
• ART: Automatic multi-step reasoning and tool-use for large language models
  Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi,
  Luke Zettlemoyer, Marco Tulio Ribeiro
  In submission
• Editing Models with Task Arithmetic
  Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi
  In: International Conference on Learning Representations (ICLR), 2023
  [code]
• Adaptive Testing and Debugging of NLP Models
  Marco Tulio Ribeiro*, Scott Lundberg* (Equal contribution)
  In: Association for Computational Linguistics (ACL), 2022
  [code] [bibtex]
• Fixing Model Bugs with Natural Language Patches
  Shikhar Murty, Christopher D. Manning, Scott Lundberg, Marco Tulio Ribeiro
  In: Empirical Methods in Natural Language Processing (EMNLP), 2022
• Finding and Fixing Spurious Patterns with Explanations
  Gregory Plumb, Marco Tulio Ribeiro, Ameet Talwalkar
  In: Transactions on Machine Learning Research (TMLR), 2022
• ExSum: From Local Explanations to Model Understanding
  Yilun Zhou, Marco Tulio Ribeiro, Julie Shah
  In: Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), 2022
  [code]
• What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
  Ángel Alexander Cabrera, Marco Tulio Ribeiro, Bongshin Lee, Rob DeLine, Adam Perer, Steven M. Drucker
  In: ACM Transactions on Computer-Human Interaction (TOCHI), 2022
  [bibtex]
• Do Feature Attribution Methods Correctly Attribute Features?
  Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, Julie Shah
  In: AAAI Conference on Artificial Intelligence (AAAI), 2022
  [code]
• Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
  Gagan Bansal*, Tongshuang Wu*, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld
  In: CHI 2021: the 2021 Conference on Human Factors in Computing Systems
• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
  Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel Weld
  In: Association for Computational Linguistics (ACL), 2021
  [code] [bibtex]
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList
  Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh.
  In: Association for Computational Linguistics (ACL), 2020
  Best Paper Award
  [code] [talk] [slides] [longer slides] [bibtex]
• SQuINTing at VQA Models: Interrogating VQA Models with Sub-Questions
  Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz,
  Marco Tulio Ribeiro, Besmira Nushi, Ece Kamar.
  In: Conference on Computer Vision and Pattern Recognition (CVPR), 2020
  [bibtex]
• Errudite: Scalable, Reproducible, and Testable Error Analysis
  Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel Weld.
  In: Association for Computational Linguistics (ACL), 2019
  [code] [bibtex] [blog]
• Are Red Roses Red? Evaluating Consistency of Question-Answering Models
  Marco Tulio Ribeiro, Carlos Guestrin, Sameer Singh.
  In: Association for Computational Linguistics (ACL), 2019
  [code] [bibtex]
• Semantically Equivalent Adversarial Rules for Debugging NLP Models
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: Association for Computational Linguistics (ACL), 2018
  Honorable mention for best paper award
  [code] [talk] [slides] [bibtex]
• Anchors: High-Precision Model-Agnostic Explanations
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: AAAI Conference on Artificial Intelligence (AAAI), 2018
  [code] [slides] [bibtex]
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016
  Audience appreciation award [video]
  [code] [talk] [slides] [bibtex] [blog]
• Model-Agnostic Interpretability of Machine Learning
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: ICML Workshop on Human Interpretability in Machine Learning (WHI), 2016
  Best paper award
  [bibtex]