I'm a researcher at Google DeepMind. I am also an Affiliate Assistant Professor at the University of Washington, where I was previously a Ph.D student advised by Carlos Guestrin and Sameer Singh.

My research is mostly on helping humans interact with machine learning models meaningfully. That involves interpretability, trust, debugging, feedback, etc.

Despite various attempts, I haven't made much progress on the (much harder) problem of getting a particular group of humans to all look at a camera at the same time →→→

Blog

I wrote these posts on how to pick a project for an intern, but I figured others might be interested too:

• Part 1: Coming up with research ideas
• Part 2: Organizing and evaluating research ideas

I also wrote this one on writing:

• Writing, part 1 — the process

Publications

• Sparks of artificial general intelligence: Early experiments with GPT-4
  Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Yi Zhang

• ScatterShot: Interactive In-context Example Curation for Text Transformation
  Tongshuang Wu, Hua Shen, Daniel Weld, Jeffrey Heer, Marco Tulio Ribeiro
  In: International Conference on Intelligent User Interfaces (IUI), 2023
  Best Paper Honorable mention

• ART: Automatic multi-step reasoning and tool-use for large language models
  Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi,
  Luke Zettlemoyer, Marco Tulio Ribeiro
  In submission

• Editing Models with Task Arithmetic
  Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi
  In: International Conference on Learning Representations (ICLR), 2023
  [code]

• Adaptive Testing and Debugging of NLP Models
  Marco Tulio Ribeiro*, Scott Lundberg* (Equal contribution)
  In: Association for Computational Linguistics (ACL), 2022
  [code] [bibtex]

• Fixing Model Bugs with Natural Language Patches
  Shikhar Murty, Christopher D. Manning, Scott Lundberg, Marco Tulio Ribeiro
  In: Empirical Methods in Natural Language Processing (EMNLP), 2022

• Finding and Fixing Spurious Patterns with Explanations
  Gregory Plumb, Marco Tulio Ribeiro, Ameet Talwalkar
  In: Transactions on Machine Learning Research (TMLR), 2022

• ExSum: From Local Explanations to Model Understanding
  Yilun Zhou, Marco Tulio Ribeiro, Julie Shah
  In: Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), 2022
  [code]

• What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
  Ángel Alexander Cabrera, Marco Tulio Ribeiro, Bongshin Lee, Rob DeLine, Adam Perer, Steven M. Drucker
  In: ACM Transactions on Computer-Human Interaction (TOCHI), 2022
  [bibtex]

• Do Feature Attribution Methods Correctly Attribute Features?
  Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, Julie Shah
  In: AAAI Conference on Artificial Intelligence (AAAI), 2022
  [code]

• Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
  Gagan Bansal*, Tongshuang Wu*, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld
  In: CHI 2021: the 2021 Conference on Human Factors in Computing Systems

• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
  Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel Weld
  In: Association for Computational Linguistics (ACL), 2021
  [code] [bibtex]

• Beyond Accuracy: Behavioral Testing of NLP models with CheckList
  Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh.
  In: Association for Computational Linguistics (ACL), 2020
  Best Paper Award
  [code] [talk] [slides] [longer slides] [bibtex]

• SQuINTing at VQA Models: Interrogating VQA Models with Sub-Questions
  Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz,
  Marco Tulio Ribeiro, Besmira Nushi, Ece Kamar.
  In: Conference on Computer Vision and Pattern Recognition (CVPR), 2020
  [bibtex]

• Errudite: Scalable, Reproducible, and Testable Error Analysis
  Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel Weld.
  In: Association for Computational Linguistics (ACL), 2019
  [code] [bibtex] [blog]

• Are Red Roses Red? Evaluating Consistency of Question-Answering Models
  Marco Tulio Ribeiro, Carlos Guestrin, Sameer Singh.
  In: Association for Computational Linguistics (ACL), 2019
  [code] [bibtex]

• Semantically Equivalent Adversarial Rules for Debugging NLP Models
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: Association for Computational Linguistics (ACL), 2018
  Honorable mention for best paper award
  [code] [talk] [slides] [bibtex]

• Anchors: High-Precision Model-Agnostic Explanations
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: AAAI Conference on Artificial Intelligence (AAAI), 2018
  [code] [slides] [bibtex]

• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016
  Audience appreciation award [video]
  [code] [talk] [slides] [bibtex] [blog]

• Model-Agnostic Interpretability of Machine Learning
  Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.
  In: ICML Workshop on Human Interpretability in Machine Learning (WHI), 2016
  Best paper award
  [bibtex]