Oxide and Friends Adversarial Machine Learning
Mar 27, 2024
Nicholas Carlini discusses adversarial machine learning, revealing how sequences of tokens can trick language models into ignoring restrictions. The hosts explore the peculiarities of C programming and delve into the surprising effectiveness of adversarial attacks on machine learning models, emphasizing the need for security-conscious approaches in ML development.
AI Snips
Chapters
Transcript
Episode notes
Making Models Misbehave with Images
- Nicholas Carlini and team made image models say nasty things by slightly altering inputs using gradient descent techniques.
- They optimized images to mislead multi-modal models, showing vulnerability was trivial to exploit.
Adversarial Noise Reveals Model Mystery
- Adversarial examples look like noise, not meaningful feature changes, revealing models use features different from human perception.
- This says we don't truly understand what features models rely on for classifications.
Adversarial Transferability Across Models
- Adversarial examples transfer across different models because models share similar data features.
- This universality explains why an attack on one model can fool others, despite differences in architecture and size.