AI Breakdown

In this episode, we discuss Improving Factuality with Explicit Working Memory by Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Sun, Luke Zettlemoyer, Gargi Gosh, Wen-tau Yih. The paper presents Ewe, a novel method that incorporates explicit working memory into large language models to improve factuality in long-form text generation by updating memory in real-time based on feedback from external resources. Ewe demonstrates superior performance over existing approaches across four datasets, boosting the VeriScore metric without compromising response helpfulness. The study highlights the significance of memory update rules, configuration, and retrieval datastore quality in enhancing the model's accuracy.

In this episode, we discuss Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control by Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu. The paper introduces "Diffusion as Shader" (DaS), a novel approach that supports various video control tasks within a unified framework by utilizing 3D control signals, overcoming the limitations of existing methods which are typically restricted to 2D signals. DaS achieves precise video manipulation, such as camera control and content editing, by employing 3D tracking videos, resulting in enhanced temporal consistency. The approach was fine-tuned within three days using 8 H800 GPUs and demonstrates strong performance in tasks like mesh-to-video generation and motion transfer, with further resources available online.

In this episode, we discuss FaceLift: Single Image to 3D Head with View Generation and GS-LRM by Weijie Lyu, Yi Zhou, Ming-Hsuan Yang, Zhixin Shu. FaceLift is a feed-forward approach for rapid and high-quality 360-degree head reconstruction using a single image, utilizing a multi-view latent diffusion model followed by a GS-LRM reconstructor to create 3D representations from generated views. It is trained primarily on synthetic datasets, showing strong real-world generalization, and outperforms existing 3D head reconstruction methods. Additionally, FaceLift enables 4D novel view synthesis for video inputs and can be integrated with 2D reanimation techniques for 3D facial animation.

In this episode, we discuss GenHMR: Generative Human Mesh Recovery by Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Pu Wang, Hongfei Xue, Srijan Das, Chen Chen. The paper introduces GenHMR, a novel generative framework for human mesh recovery (HMR) that addresses uncertainties in converting 2D images to 3D mesh. It employs a pose tokenizer and an image-conditional masked transformer to learn distributions of pose tokens, improving upon deterministic and probabilistic approaches. The model also includes a 2D pose-guided refinement technique and demonstrates superior performance compared to current methods.

In this episode, we discuss Video Creation by Demonstration by Yihong Sun, Hao Zhou, Liangzhe Yuan, Jennifer J. Sun, Yandong Li, Xuhui Jia, Hartwig Adam, Bharath Hariharan, Long Zhao, Ting Liu. The paper introduces Video Creation by Demonstration, utilizing a method called 𝛿-Diffusion to generate videos that smoothly continue from a given context image, integrating actions from a demonstration video. This approach relies on self-supervised learning for future frame prediction in unlabeled videos, using implicit latent control for flexible video generation. The proposed method surpasses current baselines in both human and machine evaluations, showcasing potential for interactive world simulations.

In this episode, we discuss Byte Latent Transformer: Patches Scale Better Than Tokens by Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srinivasan Iyer. The Byte Latent Transformer (BLT) presents a novel approach to large language models by processing data at the byte level, eliminating the need for traditional tokenization. It maintains performance comparable to tokenization-based models while offering improvements in efficiency, robustness, and scaling capability. BLT's dynamic encoding of bytes into variable-sized patches allows more efficient utilization of computational resources and successful scaling to larger model sizes, showcasing its potential in handling raw byte data without a fixed vocabulary.

In this episode, we discuss Align3R: Aligned Monocular Depth Estimation for Dynamic Videos by Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sai-Kit Yeung, Wenping Wang, Yuan Liu. Align3R is introduced as a method for achieving temporally consistent depth maps in videos using monocular inputs, addressing the challenge of maintaining consistency across frames. It leverages the DUSt3R model, enhanced with fine-tuning and optimization of depth maps and camera poses, particularly for dynamic scenes. The effectiveness of Align3R is supported by extensive experiments demonstrating its superiority over baseline methods in delivering consistent video depth and camera pose estimations.

In this episode, we discuss FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion by Haonan Qiu, Shiwei Zhang, Yujie Wei, Ruihang Chu, Hangjie Yuan, Xiang Wang, Yingya Zhang, Ziwei Liu. The paper introduces FreeScale, a tuning-free inference method that enhances visual diffusion models' ability to generate high-resolution images by combining data from different receptive scales. FreeScale effectively extracts necessary frequency components to improve visual output quality, overcoming issues like repetitive patterns in high-frequency details. Experiments demonstrate that FreeScale significantly enhances high-resolution image and video generation, supporting the creation of 8k-resolution content without further tuning.

In this episode, we discuss ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis by Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, Yonghong Tian. ViewCrafter introduces a new method for synthesizing high-fidelity novel views from single or sparse images, using video diffusion models enhanced with sparse 3D information. It incorporates an iterative synthesis and camera trajectory planning approach to expand 3D clues and novel view areas for applications such as immersive experiences and text-to-3D scene generation. The method shows superior performance in generating consistent views from limited data, and related resources are available online.

In this episode, we discuss o1-Coder: an o1 Replication for Coding by Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, Jitao Sang. The paper discusses "O1-CODER," which aims to replicate OpenAI's o1 model focusing on coding tasks, utilizing reinforcement learning and Monte Carlo Tree Search to boost System-2 thinking. The framework involves a Test Case Generator for code testing, MCTS for code data generation, and iterative model refinement to transition from pseudocode to full code generation. It highlights challenges in deploying o1-like models, suggests a shift towards System-2 paradigms, and plans to update resources and findings on their GitHub repository.