Exploring Scaling Laws and Limitations of Language Models

Our 126th episode with a summary and discussion of last week's big AI news!

NOTE: sorry about the poor audio quality :( will be better next week!

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/

Email us your questions and feedback at contact@lastweekin.ai

Timestamps:

• (00:00) Intro / Banter
• (02:35) Response to listener comments / corrections
• (03:20) News Preview
• Tools & Apps
  • (04:24) Zoom can now give you AI summaries of the meetings you’ve missed
  • (06:02) A majority of Americans have heard of ChatGPT, but few have tried it themselves
  • Lighting round
    • (12:04) Google says Gmail on your phone just got a lot faster thanks to A.I.
    • (13:00) Artifact news app now uses AI to rewrite headline of a clickbait article
    • (15:06) Instacart launches new in-app AI search tool powered by ChatGPT
    • (17:17) Microsoft Has Launched “Jugalbandi”—A New Generative AI App for India
    • (18:34) Microsoft Teams on Windows 11 gets Discord-like communities and an AI art tool
• Applications & Business
  • (19:55) Baidu’s $145M AI fund signals China’s push for AI self-reliance
  • (22:25) AI Market Set To Break The Trillion-Dollar Barrier: Surging Toward $1.06 Trillion By 2028 Whether You Trust It Or Not
  • (26:07) The A.I. job culling has already begun and 4,000 people lost work last month to the technology, according to a new report
  • (30:15) AI chatbots lose money every time you use them. That is a problem.
  • Lighting round
    • (34:00) Tech stocks surge as wave of interest in AI drives $4tn rally
    • (35:16) ANYBotics raises $50 million to help deploy its robot dog
    • (37:14) Microsoft signs deal for A.I. computing power with Nvidia-backed CoreWeave that could be worth billions
    • (39:40) Apple is looking for engineers to work with Generative AI in a mixed reality environment
    • (41:15) Lightmatter’s photonic AI hardware is ready to shine with $154M in new funding
    • (43:05) Character.AI, the a16z-backed chatbot startup, tops 1.7M installs in first week
• Projects & Open Source
  • (45:21) UAE’s Falcon 40B AI model is now royalty free for commercial, research use
  • (47:45) Evaluating and uncovering open LLMs
• Research & Advancements
  • (53:10) QLoRA: Efficient Finetuning of Quantized LLMs
  • (58:05) OpenAI is pursuing a new way to fight AI 'hallucinations'
  • (01:02:20) Do You Really Need Reinforcement Learning (RL) in RLHF? A New Stanford Research Proposes DPO (Direct Preference Optimization)
  • (01:06:50) Reward Collapse in Aligning Large Language Models
  • Lighting round
    • (01:11:15) How Should We Maximize the Planning Ability of LLMs While Reducing the Computation Cost? Meet SwiftSage : A Novel Generative Agent for Complex Interactive Reasoning Tasks, Inspired by the Dual-Process Theory of Human Cognition
    • (01:15:15) How to Keep Scaling Large Language Models when Data Runs Out? A New AI Research Trains 400 Models with up to 9B Parameters and 900B Tokens to Create an Extension of Chinchilla Scaling Laws for Repeated Data
    • (01:17:58) This AI Research Dives Into The Limitations and Capabilities of Transformer Large Language Models (LLMs), Empirically and Theoretically, on Compositional Tasks"
• Policy & Safety
  • (01:20:55) ChatGPT Plugins Open Security Holes From PDFs, Websites and More
  • (01:25:02) US, Europe Working on Voluntary AI Code of Conduct as Calls Grow for Regulation
  • Lighting round
    • (01:27:58) With new grant program, OpenAI aims to crowdsource AI regulation
    • (01:30:34) Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”
    • (01:33:13) Japan Goes All In: Copyright Doesn’t Apply To AI Training
    • (01:36:48) China warns of ‘complicated and challenging circumstances’ posed by AI risk
• Synthetic Media & Art
  • (01:39:39) Welcome to the new surreal. How AI-generated video is changing film.
  • (01:41:02) ‘Those who hate AI are insecure’: inside Hollywood’s battle over artificial intelligence"
• (01:43:20) Outro