#165 - Sora challenger, Astribot's S1, Med-Gemini, Refusal in LLMs
May 5, 2024
auto_awesome
AI news highlights include China's Sora challenger, ChatGPT's memory feature, and Amazon Q assistant. Exciting developments in robot technology with a super-fast humanoid robot video. Tesla faces scrutiny over Autopilot as OpenAI partners with Financial Times. Huawei's move to support HBM memory manufacturing in China. Research covers Gemini Models in Medicine and challenges in AI safety regulations.
MedGemini excels in medical tasks with self-training and uncertainty-guided search.
Filler tokens improve language model reasoning and performance on complex tasks.
SenseNova 5.0 surpasses GPT-4 with transformer and recurrent structures.
Octopus v4 enhances collaboration of language models for more efficient workflow.
Deep dives
Training of MedGemini for Medical Applications
MedGemini, a version of the Gemini model, is fine-tuned for medical applications, showcasing high performance on various medical benchmarks. It utilizes self-training and uncertainty-guided search to improve outputs for tasks like gene extraction and question answering.
Utilizing Fillers in Computational Reasoning
Researchers explore the efficacy of filler tokens like 'dot, dot, dot' in language models for reasoning about Code, exhibiting that additional computation enabled by fillers can enhance performance on complex tasks.
Advancement of SenseNova 5.0
SenseNova 5.0, a language model from SenseTime, claims to surpass GPT-4 on certain benchmarks. Trained with 10 billion tokens and possessing a 200,000 context window, it integrates transformer and recurrent structures for enhanced capabilities.
Octopus v4's Graph of Language Models
Octopus v4 introduces a cloud-on-device collaboration model utilizing a graph structure for directing queries to sub-workers. This approach enhances the coordination of multiple language models and tools, creating a more efficient workflow.
Multitoken Prediction in Large Language Models
Researchers propose the use of multi-token prediction in language models to predict multiple tokens simultaneously, leading to improved downstream benchmarks and potentially faster inference times. This strategy alters transformer architectures to yield more efficient predictions.
Potential Implications of Modifying Activations in LLM Networks
Researchers found that by manipulating the activations of a network, it was possible to make large language models (LLMs) perform actions as desired, essentially 'jailbreaking' them. This implies that altering network activations can lead to significant control over LLMs, granting the ability to override their responses and actions. This discovery sheds light on the inner workings of LLMs and showcases the potential for fine-tuning capabilities while bypassing certain safety mechanisms, offering a glimpse into the functional operation of transformers.
Challenges and Developments in AI Regulation and Safety Standards
The podcast discussed the evolving landscape of AI regulation, highlighting the UK government's efforts to enforce safety measures in AI development. Despite governmental initiatives, tech companies have shown resistance to stringent regulations, particularly in adopting pre-release testing for AI models. The discussion underscored the complexity of aligning regulatory frameworks with industry practices, touching on the tensions between government mandates and corporate compliance. The narrative hinted at the intricate interplay between technological advancements, ethical considerations, and governmental policies in shaping the future trajectory of AI governance.