Chris Lott

In this discussion, Chris Lott, Senior Director of Engineering at Qualcomm AI Research, dives into the complexities of accelerating large language model inference. He details the challenges of encoding and decoding, alongside hardware constraints like memory bandwidth and performance metrics. Lott shares innovative techniques for boosting efficiency, such as KV compression and speculative decoding. He also envisions the future of AI on edge devices, emphasizing the importance of small language models and integrated orchestrators for seamless user experiences.