Exploring Image and Text Vectorization in Machine Learning

Scott and Wes are joined by special guest Xenova to explore local AI models in JavaScript. From Hugging Face to Transformers.js and practical applications like real-time speech recognition and object detection, this episode dives deep into the world of machine learning.

• 00:00 Welcome to Syntax!
• 00:41 Brought to you by Sentry.io
• 01:05 Who is Xenova?
• 02:08 What is Hugging Face?
• 03:29 What is Transformers.js?
• 06:16 How was the library developed?
• SponsorBlock
• 09:04 How is it able to run?
• 10:09 Do they have to run in Python and how does Onnx work?
• Onnx.ai
• Hugging Face Optimum
• 14:19 What are some things you can do with this tech?
• 16:15 Vision tools.
• 17:38 This is actually running locally.
• 18:35 Doodle Dash
• 21:09 They currently run on CPU, what is required to make it run on GPU?
• 24:44 Can you run in JavaScript?
• 28:32 How it works with image vectors.
• 34:23 Why would people want to run it in another language?
• 35:55 Resizing images in the browser instead of on the server.
• 38:55 Applications distributed on the web vs running locally.
• 43:54 Electron has Node and Chrome, where would you run Transformers.js?
• 44:32 The API of Transformers.js
• 46:30 Object Detection.
• Semantic Image Search Client
• Real-Time Object Detection
• Background Removal Tool
• 48:33 What is the easiest way to get started?
• 51:26 Real-time speech recognition on the horizon?
• 52:08 Will we ever be able to run Stable Diffusion via JavaScript?
• 56:10 The Web LLM.
• 57:22 Practical applications for YouTube.
• 59:39 What we want to build for Syntax.fm.
• 01:06:43 Mean pooling, why it’s necessary.
• 01:09:30 Stopping YouTube spam comments.
• 01:10:34 K-Means Clustering.
• Text Clustering
• 01:13:49 Quantization.
• 01:17:35 Sick Picks + Shameless Plugs.

• Xeonva: WebGPU

• Xenova: Xenova on X

Syntax: X Instagram Tiktok LinkedIn Threads

Wes: X Instagram Tiktok LinkedIn Threads

Scott:X Instagram Tiktok LinkedIn Threads

Randy: X Instagram YouTube Threads