AI Engineering Podcast Protecting AI Systems: Understanding Vulnerabilities and Attack Surfaces
May 3, 2025
51:49
Summary
In this episode of the AI Engineering Podcast Kasimir Schulz, Director of Security Research at HiddenLayer, talks about the complexities and security challenges in AI and machine learning models. Kasimir explains the concept of shadow genes and shadow logic, which involve identifying common subgraphs within neural networks to understand model ancestry and potential vulnerabilities, and emphasizes the importance of understanding the attack surface in AI integrations, scanning models for security threats, and evolving awareness in AI security practices to mitigate risks in deploying AI systems.
Announcements
Parting Question
In this episode of the AI Engineering Podcast Kasimir Schulz, Director of Security Research at HiddenLayer, talks about the complexities and security challenges in AI and machine learning models. Kasimir explains the concept of shadow genes and shadow logic, which involve identifying common subgraphs within neural networks to understand model ancestry and potential vulnerabilities, and emphasizes the importance of understanding the attack surface in AI integrations, scanning models for security threats, and evolving awareness in AI security practices to mitigate risks in deploying AI systems.
Announcements
- Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems
- Your host is Tobias Macey and today I'm interviewing Kasimir Schulz about the relationships between the various models on the market and how that information helps with selecting and protecting models for your applications
- Introduction
- How did you get involved in machine learning?
- Can you start by outlining the current state of the threat landscape for ML and AI systems?
- What are the main areas of overlap in risk profiles between prediction/classification and generative models? (primarily from an attack surface/methodology perspective)
- What are the significant points of divergence?
- What are some of the categories of potential damages that can be created through the deployment of compromised models?
- How does the landscape of foundation models introduce new challenges around supply chain security for organizations building with AI?
- You recently published your findings on the potential to inject subgraphs into model architectures that are invisible during normal operation of the model. Along with that you wrote about the subgraphs that are shared between different classes of models. What are the key learnings that you would like to highlight from that research?
- What action items can organizations and engineering teams take in light of that information?
- Platforms like HuggingFace offer numerous variations of popular models with variations around quantization, various levels of finetuning, model distillation, etc. That is obviously a benefit to knowledge sharing and ease of access, but how does that exacerbate the potential threat in the face of backdoored models?
- Beyond explicit backdoors in model architectures, there are numerous attack vectors to generative models in the form of prompt injection, "jailbreaking" of system prompts, etc. How does the knowledge of model ancestry help with identifying and mitigating risks from that class of threat?
- A common response to that threat is the introduction of model guardrails with pre- and post-filtering of prompts and responses. How can that approach help to address the potential threat of backdoored models as well?
- For a malicious actor that develops one of these attacks, what is the vector for introducing the compromised model into an organization?
- Once that model is in use, what are the possible means by which the malicious actor can detect its presence for purposes of exploitation?
- What are the most interesting, innovative, or unexpected ways that you have seen the information about model ancestry used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on ShadowLogic/ShadowGenes?
- What are some of the other means by which the operation of ML and AI systems introduce attack vectors to organizations running them?
Parting Question
- From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today?
- Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@aiengineeringpodcast.com with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers.
- HiddenLayer
- Zero-Day Vulnerability
- MCP Blog Post
- Python Pickle Object Serialization
- SafeTensors
- Deepseek
- Huggingface Transformers
- KROP == Knowledge Return Oriented Prompting
- XKCD "Little Bobby Tables"
- OWASP Top 10 For LLMs
- CVE AI Systems Working Group
- Refusal Vector Ablation
- Foundation Model
- ShadowLogic
- ShadowGenes
- Bytecode
- ResNet == Resideual Neural Network
- YOLO == You Only Look Once
- Netron
- BERT
- RoBERTA
- Shodan
- CTF == Capture The Flag
- Titan Bedrock Image Generator