AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
4min snip
# 182 - Alexa 2.0, MiniMax, Surskever raises $1B, SB 1047 approved
Last Week in AI
Beyond Logic: The Rise of Bandwidth in GPU Performance
The performance of GPUs has significantly improved over time, particularly in terms of floating point operations per second (FLOPS), which have been increasing at approximately three times per year. However, the components that facilitate communication and data transfer, namely interconnect bandwidth and on-GPU high bandwidth memory, have not scaled at the same rate. With interconnect bandwidth and high bandwidth memory improving at only 1.6x and 1.4x per year respectively, the disparity between logic performance and bandwidth is widening. This imbalance indicates that the industry is transitioning from a performance focus dominated by FLOPS to a bandwidth-bound reality, where insufficient memory and interconnect bandwidth are hindering the efficient training and operation of large models. Hence, enhancing communication capabilities among GPUs is becoming critical to fully leverage the advancements in computational power. The evolving landscape underscores the necessity of addressing bandwidth limitations for continued growth in GPU performance and the effective handling of complex computations.
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode