AI Breakdown

In this episode, we discuss Explorative Inbetweening of Time and Space by Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya, Michael J. Black, Xuaner Zhang. The paper presents a method for generating video sequences from just a starting and ending frame, called bounded generation, by utilizing a new sampling strategy named Time Reversal Fusion. This strategy merges the forward and backward denoising processes guided by the start and end frames to create videos that naturally transition between the two given frames, enable smooth inbetweening of motion, and create looping videos when the frames are the same. Time Reversal Fusion is shown to outperform previous methods in terms of generating complex movements and 3D-consistent visuals without additional training on a model.

In this episode, we discuss Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking by Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, Noah D. Goodman. The paper presents Quiet-STaR, an advancement of Self-Taught Reasoner (STaR), which teaches Language Models to generate internal rationales to enhance text predictions. By introducing a tokenwise parallel sampling algorithm, learnable tokens for marking thoughts, and extending teacher-forcing, the approach addresses practical challenges in model implementation. Results demonstrate that the approach enables the model to better predict challenging tokens, answer complex questions, and improve performance on benchmarks without task-specific fine-tuning, signifying progress towards more generative and scalable reasoning in LMs.

In this episode, we discuss Evaluating Large Language Models at Evaluating Instruction Following by Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, Danqi Chen. This paper examines the effectiveness of using large language models (LLMs) to evaluate the performance of other models in following instructions, and introduces a new meta-evaluation benchmark called LLM-BAR. The benchmark consists of 419 pairs of texts, with one text in each pair following a given instruction and the other not, designed to challenge the evaluative capabilities of LLMs. The findings show that LLM evaluators vary in their ability to judge instruction adherence and suggest that even the best evaluators need improvement, with the paper proposing new prompting strategies to enhance LLM evaluator performance.

In this episode, we discuss Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation by Se-eun Yoon, Zhankui He, Jessica Maria Echterhoff, Julian McAuley. The paper presents a new protocol with five tasks to assess the performance of synthetic users, generated by large language models, aiming to mimic human behavior in conversational recommender systems. The tasks evaluate essential features such as discussing items, stating preferences, asking for recommendations, and providing feedback. Initial evaluations show that these tasks can identify how language models differ from actual human behavior and suggest how model tuning and prompting can improve the synthetic users' resemblance to real users.

In this episode, we discuss Branch-Solve-Merge Improves Large Language Model Evaluation and Generation by Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, Xian Li. The paper introduces the BRANCH-SOLVE-MERGE (BSM) method for improving Large Language Models (LLMs). This method enhances task planning and coherence in LLMs by breaking tasks into sub-tasks, solving them separately, and then combining the solutions. BSM has shown significant improvements in response evaluation and constrained text generation, including better alignment with human judgment, reduced biases, and higher constraint satisfaction.

In this episode, we discuss MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training by Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang. This study investigates how different architectural components and data types impact the performance of Multimodal Large Language Models (MLLMs). The authors discovered that using a combination of different data types is crucial for high performance, and that the design of the image encoder is more influential than the vision-language connector. They applied these insights to create MM1, a series of state-of-the-art multimodal models with up to 30 billion parameters, which excel at few-shot learning and complex reasoning tasks.

In this episode, we discuss Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking by Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, Noah D. Goodman. The paper presents Quiet-STaR, an improved self-reasoning language model that internally generates rationales to enhance text prediction abilities. This approach mitigates challenges associated with computational costs and limitations in token prediction by using a new tokenwise parallel sampling algorithm and an extended teacher-forcing method. The enhanced model demonstrates improved zero-shot performance on reasoning benchmarks and a reduction in perplexity without task-specific fine-tuning, indicating a more scalable and general reasoning capability in language models.

In this episode, we discuss WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? by Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, Alexandre Lacoste. The paper introduces WorkArena, a benchmark created to evaluate large language model-based agents that interact with web-based enterprise software like ServiceNow, along with BrowserGym, a tool for creating and testing these agents. The study assesses the agents' abilities to complete typical knowledge worker tasks, finding that while agents have potential in this area, there is still a substantial gap before achieving complete task automation. The results also reveal differences in the performances of open versus closed-source language models, pointing to a key direction for continued research and improvement.

In this episode, we discuss Synth 2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings by Sahand Sharifzadeh, Christos Kaplanis, Shreya Pathak, Dharshan Kumaran, Anastasija Ilic, Jovana Mitrovic, Charles Blundell, Andrea Banino. The paper introduces a method that combines Large Language Models (LLMs) and image generation models to synthetically create image-text pairs for training Visual-Language Models (VLMs), thus circumventing the need for extensive human-labeled data. Synthetic image embeddings, generated from LLM-produced captions, are used to effectively train VLMs, achieving a 17% performance improvement over baselines while using less data. Additionally, this synthetic data creation in the image embedding space is shown to be 25% faster than working in the pixel space, offering a scalable and efficient solution for enhancing VLM training.

In this episode, we discuss Is Cosine-Similarity of Embeddings Really About Similarity? by Harald Steck, Chaitanya Ekanadham, Nathan Kallus. The paper investigates the use of cosine-similarity in quantifying semantic similarity between embedded vectors in high-dimensional space, and reveals potential issues when applied to embeddings from regularized linear models. Analytical study of these models shows that cosine-similarity can produce meaningless or non-unique similarity measures, with the effects of regularization often implicitly influencing the results. The authors warn against the uncritical use of cosine-similarity in deep learning models due to these findings and suggest considering alternative methods to ensure the validity and clarity of semantic similarity assessments.