Publications

2024

1. LLM Agents
   @ ICLR
   
   COLM
   under review

   OpenAgents: An Open Platform for Language Agents in the Wild

   Tianbao Xie^*, Fan Zhou^*, Zhoujun Cheng^*, Peng Shi^*, Luoxuan Weng^*, Yitao Liu^*, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu,

   In ICLR 2024 Workshop on Large Language Model (LLM) Agents

   Abs arXiv PDF

   Language agents show potential in being capable of utilizing natural language for varied and intricate tasks in diverse environments, particularly when built upon large language models (LLMs). Current language agent frameworks aim to fa- cilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-level de- signs. We present OpenAgents, an open platform for using and hosting language agents in the wild of everyday life. OpenAgents includes three agents: (1) Data Agent for data analysis with Python/SQL and data tools; (2) Plugins Agent with 200+ daily API tools; (3) Web Agent for autonomous web browsing. OpenAgents enables general users to interact with agent functionalities through a web user in- terface optimized for swift responses and common failures while offering develop- ers and researchers a seamless deployment experience on local setups, providing a foundation for crafting innovative language agents and facilitating real-world evaluations. We elucidate the challenges and opportunities, aspiring to set a foun- dation for future research and development of real-world language agents.

2. arxiv

   CodeUpdateArena: Benchmarking Knowledge Editing on API Updates

   Liu, Zeyu Leo, Pandit, Shrey, Ye, Xi, Choi, Eunsol, and Durrett, Greg

   In Submission to ICLR 2025

   Abs arXiv PDF

   Large language models (LLMs) are increasingly being used to synthesize and reason about source code. The libraries and API functions they invoke are continuously evolving, with functionality being added or changing. Yet, no prior work has studied how an LLM’s knowledge about code API functions can be updated. To fill this gap, we present ‘CodeUpdateArena‘, a benchmark for knowledge editing in the code domain. An instance in our benchmark consists of a synthetic API function update paired with a program synthesis example that uses the updated functionality; our goal is to update an LLM to be able to solve this program synthesis example without providing documentation of the update at inference time. Compared to knowledge editing for facts, success here is more challenging: a code LLM must reason about the semantics of the modified function rather than just reproduce its syntax. Our dataset is constructed by first prompting GPT-4 to generate atomic and executable function updates. Then, for each update, we generate program synthesis examples whose code solutions are prone to use the update. Our benchmark covers updates of various types to 54 functions from seven diverse Python packages, with a total of 670 program synthesis examples. Our experiments show that fine-tuning open-source code LLMs (i.e., DeepSeek, CodeLlama) on documentation of a new update does not allow them to incorporate changes for problem-solving. However, prepending the same information does help, establishing that the information is present, and careful fine-tuning on examples demonstrating the update shows improvement, paving the way for better knowledge editing techniques for code.

2023

1. EMNLP

   Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model

   Leo Z. Liu, Tim Dettmers, Xi Victoria Lin, Veselin Stoyanov, and Xian Li,

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

   Abs arXiv PDF

   Large and sparse feed-forward layers (S-FFN) such as Mixture-of-Experts (MoE) have proven effective in scaling up Transformers model size for pretraining large language models. By only activating part of the FFN parameters conditioning on input, S-FFN improves generalization performance while keeping training and inference costs (in FLOPs) fixed. In this work, we analyzed two major design choices of S-FFN: the memory block (a.k.a. expert) size and the memory block selection method under a general conceptual framework of sparse neural memory. Using this unified framework, we compare several S-FFN architectures for language modeling and provide insights into their relative efficacy and efficiency. We found a simpler selection method — Avg-K that selects blocks through their mean aggregated hidden states, achieving lower perplexity in language model pretraining compared to existing MoE architectures including Switch Transformer (Fedus et al., 2021) and HashLayer (Roller et al., 2021).

2. MRL
   @ EMNLP

   Learning to translate by learning to communicate

   C.M. Downey, Xuhui Zhou, Leo Z. Liu, and Shane Steinert-Threlkeld,

   In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL)

   Abs arXiv PDF

   We formulate and test a technique to use Emergent Communication (EC) with a pre-trained multilingual model to improve on modern Unsupervised NMT systems, especially for low-resource languages. It has been argued that the current dominant paradigm in NLP of pre-training on text-only corpora will not yield robust natural language understanding systems, and the need for grounded, goal-oriented, and interactive language learning has been high lighted. In our approach, we embed a multilingual model (mBART, Liu et al., 2020) into an EC image-reference game, in which the model is incentivized to use multilingual generations to accomplish a vision-grounded task. The hypothesis is that this will align multiple languages to a shared task space. We present two variants of EC Fine-Tuning (Steinert-Threlkeld et al., 2022), one of which outperforms a backtranslation-only baseline in all four languages investigated, including the low-resource language Nepali.

2022

1. EmeCom
   @ ICLR

   Emergent Communication Fine-tuning (EC-FT) for Pretrained Language Models

   Shane Steinert-Threlkeld, Xuhui Zhou, Leo Z. Liu, and C. M. Downey,

   In Proc. of ICLR EmeCom Workshop (Runner-up Best Paper)

   Abs PDF

   It has recently been argued that the currently dominant paradigm in NLP of pretraining on text-only corpora will not yield robust natural language understanding systems. One strain of this argumentation highlights the need for grounded, goal-oriented, and interactive language learning. In this position paper, we articulate how Emergent Communication (EC) can be used in conjunction with large pretrained language models as a ‘Fine-Tuning’ (FT) step (hence, EC-FT) in order to provide them with supervision from such learning scenarios. We discuss methodological issues and difficulties with making this work, and then illustrate the overall idea with a case study in unsupervised machine translation, before concluding with a discussion on the relation to multimodal pretraining.

2021

1. Findings
   @ EMNLP

   Probing Across Time: What Does RoBERTa Know and When?

   Leo Z. Liu^*, Yizhong Wang^*, Jungo Kasai, Hannaneh Hajishirzi, Noah A. Smith,

   In Findings of the Association for Computational Linguistics: EMNLP 2021

   Abs arXiv PDF

   Models of language trained on very large corpora have been demonstrated useful for natural language processing. As fixed artifacts, they have become the object of intense study, with many researchers “probing” the extent to which they acquire and readily demonstrate linguistic abstractions, factual and commonsense knowledge, and reasoning abilities. Recent work applied several probes to intermediate training stages to observe the developmental process of a large-scale model (Chiang et al., 2020). Following this effort, we systematically answer a question: for various types of knowledge a language model learns, when during (pre)training are they acquired? Using RoBERTa as a case study, we find: linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. Reasoning abilities are, in general, not stably acquired. As new datasets, pretraining protocols, and probes emerge, we believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.

2020

1. BlackboxNLP
   @ EMNLP

   Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets

   Chuanrong Li^α, Lin Shengshuo^α, Leo Z. Liu^α, Xinyi Wu^α, Xuhui Zhou^α, and Shane Steinert-Threlkeld,

   In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP 2020

   Abs arXiv PDF

   Although large-scale pretrained language models, such as BERT and RoBERTa, have achieved superhuman performance on in-distribution test sets, their performance suffers on out-of-distribution test sets (e.g., on contrast sets). Building contrast sets often requires human-expert annotation, which is expensive and hard to create on a large scale. In this work, we propose a Linguistically-Informed Transformation (LIT) method to automatically generate contrast sets, which enables practitioners to explore linguistic phenomena of interests as well as compose different phenomena. Experimenting with our method on SNLI and MNLI shows that current pretrained language models, although being claimed to contain sufficient linguistic knowledge, struggle on our automatically generated contrast sets. Furthermore, we improve models’ performance on the contrast sets by applying LIT to augment the training data, without affecting performance on the original data.