Ameya Godbole (original) (raw)

I am a fourth year PhD student in Computer Science at the University of Southern California advised by Prof. Robin Jia. I work on hallucination-free and faithful generation with retrieval augmentation. I worked on leveraging the planning capabilities of LLMs to guide retrieval. I also study the behavior of factuality detection systems and their reliability in model selection. In the past, I have worked on retrieval for open-domain question answering (ODQA), knowledge base completion (KBC), and semantic parsing for knowledge base QA (KBQA).

Previously, I was a Research Fellow with the wonderful Information Extraction and Synthesis Lab (IESL) at the University of Massachusetts, Amherst. In addition to research, I worked on developing and testing features for the OpenReview platform.

1. Analysis of Plan-based Retrieval for Grounded Text Generation
   Ameya Godbole, Nicholas Monath, Seungyeon Kim, Ankit Singh Rawat, Andrew McCallum, and Manzil Zaheer
   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024
   In text generation, hallucinations refer to the generation of seemingly coherent text that contradicts established knowledge. One compelling hypothesis is that hallucinations occur when a language model is given a generation task outside its parametric knowledge (due to rarity, recency, domain, etc.). A common strategy to address this limitation is to infuse the language models with retrieval mechanisms, providing the model with relevant knowledge for the task. In this paper, we leverage the planning capabilities of instruction-tuned LLMs and analyze how planning can be used to guide retrieval to further reduce the frequency of hallucinations. We empirically evaluate several variations of our proposed approach on long-form text generation tasks. By improving the coverage of relevant facts, plan-guided retrieval and generation can produce more informative responses while providing a higher rate of attribution to source documents.
2. Benchmarking Long-tail Generalization with Likelihood Splits
   In Findings of the Association for Computational Linguistics: EACL 2023, May 2023
   In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances. We propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. We create ‘Likelihood Splits’ where examples that are assigned lower likelihood by a pre-trained language model (LM) are placed in the test set, and more likely examples are in the training set. This simple approach can be customized to construct meaningful train-test splits for a wide range of tasks. Likelihood Splits surface more challenges than random splits: relative error rates of state-of-the-art models increase by 59% for semantic parsing on Spider, 93% for natural language inference on SNLI, and 33% for yes/no question answering on BoolQ, on our splits compared with the corresponding random splits. Moreover, Likelihood Splits create fairer benchmarks than adversarial filtering; when the LM used to create the splits is also employed as the task model, our splits do not unfairly penalize the LM.
3. Knowledge Base Question Answering by Case-based Reasoning over Subgraphs
   Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew Mccallum
   In Proceedings of the 39th International Conference on Machine Learning, 17–23 jul 2022
   Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed. However, we hypothesize in a large KB, reasoning patterns required to answer a query type reoccur for various entities in their respective subgraph neighborhoods. Leveraging this structural similarity between local neighborhoods of different subgraphs, we introduce a semiparametric model (CBR-SUBG) with (i) a nonparametric component that for each query, dynamically retrieves other similar k-nearest neighbor (KNN) training queries along with query-specific subgraphs and (ii) a parametric component that is trained to identify the (latent) reasoning patterns from the subgraphs of KNN queries and then apply them to the subgraph of the target query. We also propose an adaptive subgraph collection strategy to select a query-specific compact subgraph, allowing us to scale to full Freebase KB containing billions of facts. We show that CBR-SUBG can answer queries requiring subgraph reasoning patterns and performs competitively with the best models on several KBQA benchmarks. Our subgraph collection strategy also produces more compact subgraphs (e.g. 55% reduction in size for WebQSP while increasing answer recall by 4.85%)\footnoteCode, model, and subgraphs are available at \htmlhttps://github.com/rajarshd/CBR-SUBG.
4. Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering
   Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, Zhiyu Gong, Abhishek Singhal, Mo Yu, Xiaoxiao Guo, Tian Gao, Hamed Zamani, Manzil Zaheer, and 1 more author
   In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Nov 2019
   Multi-hop question answering (QA) requires an information retrieval (IR) system that can find \textitmultiple supporting evidence needed to answer the question, making the retrieval process very challenging. This paper introduces an IR technique that uses information of entities present in the initially retrieved evidence to learn to ‘\textithop’ to other relevant evidence. In a setting, with more than \textbf5 million Wikipedia paragraphs, our approach leads to significant boost in retrieval performance. The retrieved evidence also increased the performance of an existing QA model (without any training) on the benchmark by \textbf10.59 F1.