Grey Beards on Systems

Jim Handy, General Director at Objective Analysis and I were at Future of Memory and Storage 2026 Conference in Santa Clara earlier this month and there was a lot of news going around. Jim’s been on our show just about every year to discuss FMS news, And with the recent focus beyond flash, it’s even harder for one person to keep up. New tech was the main topic discussed at the show but the one underlying thread was the AI supercycle and the switch from AI training to inference. The supercycle and inference seems to be turning the IT industry on it’s head. HBM and DRAM were highlights of the show because they are in every accelerator/GPU. For whatever reason, there seems to be no price elasticity with respect to accelerators, DRAM or data centers for that matter. So the worlds memory fabs are moving from relatively low margin DRAM to the higher margin HBM. Starving the rest of the IT world for memory. Jason mentioned that fab HBM capacity is all contracted for out to the end of 2027. Speaking of which there was a lot of talk about KV (key-value) data offload (taking data out of HBM and storing it elsewhere while it’s unused) and the IO requirements needed to support an effective KV cache offload . One vendor said their current storage solution served 10M IOPs for a single HBF (high bandwidth flash storage) device but the real solution will need 100M 512B IOPs. It turns out that token and KV cache elements are relatively small and using 4K blocks wastes a lot of space. And of course with 1000s of processor elements accessing the KV cache during token processing, thinking and other AI inference activities, if the KV data elements are going to be anywhere but in HBM they need to be accessed very quickly. HBF, located on the accelerator board adjacent to HBM, was a hot topic last year but this year one could see some solutions coming to market. The one vendor discussed above talked about their liquid cooled HBF. But there were others with HBF storage of thier own. On the other hand there were a number of storage vendors talking about taking KV cache out off the accelerator and onto storage on the PCIe bus and using NVMe SSDs. Not HBF but maybe another tier of KV caching. However there was lot of talk about CXL shared memory pooling that could be used as a shared KV cache to also do this. Unclear which tech (CXL memory pooling or NVMe SSDs) will win this race but it just might be a combination of both of them. One vendor mentioned a new technology stacking DRAM directly ontop of (accelerator) compute modules. This would eliminate latency required to go off chip to an HBM and would use a chiplet interface. Doing so would create an intermediate tier of accelerator memory between compute SRAM and HBM, cheaper than SRAM and with more capacity. Hyperscalers are sitting on a lot of DDR4 DRAM which they are salvaging and moving to shared memory pools behind CXL switches. Jason has been talking about this for years but it’s becoming a reality. Jim mentioned that even without the AI supercycle driving data center build out there would likely have been a DRAM and NAND crunch coming anyway. This just happens every few years as fabs over produce driving prices down, which drives demand up and leads to fabs being unable to satisfy demand. Usually component price increases depress demand which allows the fabs enough time to increase production and start the cycle again. It’s just that the AI supercycle has driven all this like it’s on steroids. And price increases have yet to dampen demand. The hyperscalers don’t seem to care the price they have to pay to power AI inference. In fact some hyperscalers are going right to the fabs and contracting for all the chips coming off a fab line for a period of time. This seems to be the only way they can guarantee to support the compute needed for AI inference. For once there wasn’t much talk about increasing capacity or 3D NAND layers at the show. In prior shows SSD capacity and 3D NAND layer counts were always news. Last year we saw a glimmer of 3D layer increase backlash as the tradeoffs of more layers vs, more capacity per layer were discussed. This year 3D NAND layers and SSD capacities weren’t even a topic of discussion. The NAND and SSD world are moving to high bandwidth and high capacity flash, talking less about the technology behind them and their capacity. Jim mentioned one reason for the reluctance to discuss 3D layers increasing was that 3D NAND suppliers were running out of space for stairways. Turns out each layer in 3D NAND (and most are over 100 layers today) has to have a separate electronic connection outside of it and this requires building a stairway with one stair per layer. And as you get to 100 to 200 or more layers, the stairways themselves start consuming too much NAND floor space, reducing capacity. Finally, MRAM was another hot topic at the show. It turns out at current chip technology (2nm) levels, NOR gates out of normal TTL don’t work that well. MRAM seems impervious to this problem and is increasingly being used to replace TTL NOR gates in semiconductors. MRAM has been a long time vendor at the show but mostly in niche solutions. For example, MRAM seems pretty impervious to cosmic rays so it’s increasingly used in RAD hardened applications such as in space, nuclear reactors, et al. But this is a major new use case that could broaden MRAM technology ecosystem beyond the niche players. Watch for more in this space. Jim and the team at Objective Analysis have been busy this year churning out articles and in depth reports on NAND. memory, and compute business and recent technology innovations. Check out their website (link above) to learn more. Jim Handy, General Director Objective Analysis Jim Handy of Objective Analysis has over 35 years in the electronics industry, including 20 years as a leading semiconductor and SSD industry analyst. Early in his career he held marketing and design positions at leading semiconductor suppliers including Intel, National Semiconductor, and Infineon. A frequent presenter at trade shows, Mr. Handy is known for his technical depth, accurate forecasts, widespread industry presence and volume of publication. He has written hundreds of market reports, articles for trade journals, and white papers, and is frequently interviewed and quoted in the electronics trade press and other media.  He posts blogs at www.TheMemoryGuy.com, and www.TheSSDguy.com

I saw Jason Goldschmidt (LinkedIN), DELL Distinguished Engineer, present with another at SNIA StorageAI conference on the work of the Accelerated Object (AKA, S3 over RDMA) TWG (Technical Working Group) and thought it would be great to have him and the other TWG Co-chair, Nick Connolly (LinkedIn), ARM Principal Engineer, on the show to discuss what they are up to. Further, S3 and object storage in general is growing by leaps and bounds. Some AWS S3 statistics: 100s of Exabytes of objects stored, 500T objects, over 123 AZs, and currently AWS is doing about 200M S3 object requests/sec or ~ 1PB/sec of S3 object data transfer, world wide. S3 objects have become the goto store for training data, model checkpoints, RAGs and just about anything else that’s big, sequential, and needs to be stored for AI processing. As such, there’s been a ton of interest in providing a higher performing S3 object store to maintain and improve accelerator (GPU) utilization during AI processing. SNIA’s Accelerated Object TWG was formed last fall to address this pressing industry need. Listen to the podcast to learn more. Objects have been around seemly forever. I first learned of them at a Mass Storage Conference in the early 2000s, and at the time they were called BLOBs. And for a long time object storage was in search of a killer app. I thought a while back that data lakes would be that. But then AI emerged over the last decade and object storage just exploded.  Nowadays just about anything to do with AI models, i.e., training data, checkpoints, RAG databases, log files, etc. is stored as objects. And as the S3 stats above show, these files are large, over 200MB on avg (assuming 100EB of objects over 500T objects). So the number one problem has become how to get that data from storage to GPU and back again. Enter S3 over RDMA, or as SNIA calls it, Accelerated Object Storage. For a TWG that only formed last fall, they have been making amazing progress. Which to tell the truth is indicative of industry interest. I was at another conferenc a couple of weeks back where a major storage vendor, announced (pre-standardization) support for S3 over RDMA. And I find it hard to believe they’re the only one. Keith asked about how accelerated objects would help inferencing and agentic AI activity in particular. It turns out that a while back AWS and others offered support for tables (or databases) on object storage. So RAGs, are a prime example where accelerated object can help. RAG databases can be huge and searching them for prompt/query enterprise specific context can slow down agentic inferencing activity. Having a zero-copy, direct storage to GPU object data transfer can shave a lot of latency off a RAG request, as well as considerably speed up data transfer of large chunks of data. Then there’s KV (Key:Value) cache offload. KV cache’s are programmatically accessible context store built during inference PreFill phase and used throughout inference Decode phase and grow as agent/query context grows. But with today’s GPU memory limited, from 32GB to 192GB, there’s just not enough room to keep every prompt’s, every agentic tools context stored. This causes KV caches to be offloaded to memory, but even CPU memory has limits. As servers start to support 100s, if not 1000s of agentic tool request, prompts and queries, even CPU memory space can run out. Prior to KV cache offload when HBM and CPU memory ran out, prompt KV caches would be jettisoned. And so when the prompt starts up again, the GPU would have to do another PreFill phase to recalculate its KV store. But with KV cache offload to storage, a prompts KV store can be written to local SSD or object storage and read back when needed. As Nick and Jason mentioned, it’s a timing tradeof. How quickly can you retrieve a prompt’s KV cache from local SSD or accelerated object vs, how quickly can the GPU recompute the KV cache for the prompt from context tokens.. One would think this would be a no contest, computation always beats IO. But as KV stores become larger, and GPU activity becomes more multi-threaded, reading back an offloaded KV cache (directly to GPU) can be faster, sometimes much faster We didn’t spend as much time on AI training, but the advantages of accelerated object are even more obvious here. All we need mention is training data, may no longer need to be staged to local SSD, if it can be retrieved at RDMA speeds from local object storage. And all that checkpointing that goes on during LLM training over a gaggle of GPUs can be done a lot faster using RDMA writes to local object. Even so, S3 over RDMA really only helps data transfer speeds, by eliminating one copy (old way: read from object store to CPU memory, copy to GPU memory; new way: read directly from object store into GPU memory, hence zero-copy). There’s still has a lot of non-data transfer, or setup activity to be done by the CPU, such as, authentication, which storage, which GPU memory location, etc. but once all that’s in place the CPU can effectively step out of the picture and let (Ethernet RDMA) hardware take over. And of course S3 over RDMA, by eliminating one copy, frees up CPU cycles as well as speeding up data transfer. S3 over RDMA is intended to work over RoCEv1 . RoCEv2 also exists, which supports routed (internet) access to data over Ethernet. But at the moment accelerated object is an RoCEv1 targeted solution But how can one define a standard to enhance a protocol when the underlying protocol is not a standard. As we all know, S3 is not a standard, but rather is something AWS invented for their object access. Nick and Jason said it’s not impossible, but yes they have to deal with that as part of their work. In the mean time they and their team are quickly moving forward working on standardizing Accelerated Object and the AI world is taking notice. And there will be more info on Accelerated Object at the next SNIA Storage Developer Conference in USA, coming up this September in CA. Jason Goldschmidt, DELL Distinguished Engineer, SNIA Accelerated Object TWG, Co-Chair Jason is a Distinguished Engineer at Dell Technologies, working in the Storage Chief Technology Office. His role involves leading the development of next-generation storage and network protocols for both Public and Private Clouds. Jason also serves as Co-Chair of the SNIA Accelerated Object I/O Technical Working Group. Throughout his career, Jason has focused on enterprise storage and meeting the needs of customers who want to expand beyond the traditional on-premises data center. He enjoys collaborating with customers and partners to understand their requirements when transforming their business. In his personal life, Jason is passionate about running marathons and baking artisan bread. He lives in Newton, Massachusetts, with his wife and their three children. Nick Connolly, ARM Principal Engineer, SNIA Accelerated Object TWG, Co-chair Nick Connolly is a pioneer of software-defined storage and a Principal Software Engineer at Arm. He holds patents ranging from highly scalable algorithms through to data protection techniques and has delivered products generating over $1bn in software revenue. Nick serves as Co-Chair of the SNIA Accelerated Object I/O Technical Working Group and has previously been a Technical Lead for the CNCF Storage Technical Advisory Group. He is based in London.

Ted Weatherford (Lin), VP Business Development and John Carney (Lin), Distinguished Engineer, SW Architecture of Xsight Labs, presented at a recent AI Infrastructure Field Day (AIIFD4, see video of their session here) I attended and thought they had a great way to solve the need for high speed/software defined networking (SDN) in modern data centers. Turns out Xsight Labs is a fabless semiconductor company, specializing in SDN ASICs and currently have an X2 Switch ASIC and a E1 DPU ASIC out on the market today. They are the first vendor to have an 800Gbps DPU ASIC and their 12.8T X2 switch chip is focused on low power SDN for ToR, edge and extreme edge environments. Listen to the podcast to learn more. Ted said there are 6 primary chips which make up AI data centers: GPU, CPU, NIC, Scale up networking switch, Scale out networking switch, & DPU chips. And there are maybe 20 semiconductor dev groups around the world capable of developing these. Their current monolithic die chips use the TSMC N5 process. While their networking X2 switch chip has been out in volume for a while now and is in its second generation, their E1 DPU chip is brand new. We started discussing their X2 SDN switch chip. The X2 supports up to 128 ports/12.8T in 1RU (based on reference switch architecture) at under 200W and any port can configured to support 10G to 400G. In addition to their switch form factor and power envelope, the X2’s programability is a major strength. They have over 3000 dedicated, “MAP” core CPUs or Harvard (separate instructions and data memory and access paths) core CPUs to process, in parallel, the data flows in and out of the chip plus separate parser cores. The 3072 MAP cores are allocated statically to 64 parallel packet forwarding engines and 64 data plane units and there’s a 64MB packet buffer shared across all ports. They have had a couple of PoCs where customers managed to code X2 switch support for their application in a matter of weeks (days maybe). Moving on to the E1 DPU chip, the story gets even more interesting. Traditionally, DPUs have had a hybrid architecture where the data path processing is done in parallel cores and the management processing in a handful of RISC cores requiring clients to code in multiple architectural environments. Xsight labs took a different tack. The E1 has up to 64 N2 ARM cores and can literally run Linux or other OSs (at the same time) to support DPU processing. (Btw, the latest gen Apple M5 CPU only has 8 fast ARM cores and 4 slow ARM cores). On the server side of the DPU, it supports up to 40 PCIe Gen5 lanes and on the ethernet side 1-800Gbps port, 2-400Gbps ports, 4-200Gbps ports, etc.. The E1 also supports 4 DDR5 5200 MHz memory interfaces which means E1 can support TBs of memory!? John went into the software architecture a bit more on our podcast than at AIIFD4 and said it had 32MB of system cache. John said main memory would mostly be used to host static/slowly changing databases and tables. Actual instructions to support DPU IO would all reside in ARM l1-l2-l3 instruction caches during processing. The E1 operates the 800G port at line speed within 75W of power. At AIIFD4, they discussed their SONiC DASH benchmark VNET to VNET scenario results. SONiC DASH is an open source project started by Microsoft, used to assess smart switches under cloud service provider workloads. There are many levels for the SONiC DASH benchmark and their E1 DPU-X2 network (reference architecture) smart switch was the first smart switch to sustain Hero800 performance (800Gbps link support) using only a single DPU. This means their E1 DPU-X2 switch processed in excess of 14M TCP connections/second for over a minute and a half, all while sustaining over 120M TCP&UDP background data flows. As for customer wins, Ted mentioned each SpaceX Starlink V3 satellite uses multiple X2 chips and that the Open Flash Platform (OFP) organization is currently implementing an NFS server using a single E1 DPU and PBs of flash in a single sled. This means OFP is mapping a Linux file system across all the flash in the sled and presenting it as a NFS storage server out the front end. No server required… Ted Weatherford, VP Business Development, Xsight Labs Ted Weatherford is Vice President of Business Development at Xsight Labs, where he was hired to build out a commercial team, discover and close first strategic customer engagements, manages HW and SW partner ecosystems, and contribute to go-to-market activities for the company’s products and engineering services. Ted brings over 30 years of experience in product line management, business development, and strategic marketing in the semiconductor industry. Over his career, his work has contributed to hundreds of chip design-wins globally and generated more than $2 billion in revenue for leading merchant silicon suppliers including BRCM, INTL, CRDO, MTK, MRVL and many others John Carney, Distinguished Engineer, Software Architecture, Xsight Labs John is a Distinguished Engineer at Xsight Labs, focused on software architecture for Xsight’s E-series DPU products. Prior to Xsight, John held hardware and software technical leadership roles at Broadcom, Cisco and Juniper. John is an expert in high-performance networking data planes.

Well this year went fast. Keith, Jason and I sat down to try to make some sense of it all. AI is still on a tear and shows no end in sight. Questions abound on whether we are seeing signs of a bubble or not, our answer – maybe. We see it in GPU pricing. in AI startup valuations, and in enterprise interest. Some question whether the Enterprise is seeing any return from investments in AI but there’s no doubt they are investing. Inferencing on prem with training/fine tuning done in neo-clouds, has become the new norm. I thought we’d be mostly discussing agentic AI but it’s too early to for that yet. In other news, the real Broadcom VMware play is starting to emerge (if ever in doubt). It’s an all out focus on the (highly profitable) high end enterprises, and abandon the rest. And of course the latest weirdness to hit IT is DRAM pricing, but in reality it’s the price of anything going into AI mega-data centers that’s spiking. Listen to the podcast to learn more AI GPU pricing is still high, although we are starting to see some cracks in NVIDIA’s moat. AMD GPUs made a decent splash in the latest MLperf Training results and Google TPUs are starting to garner some in the enterprise. And NVIDIA GPUs are becoming less of a compute monster by focusing more with their latest GPU offerings on optimization for low precision compute, FP2 anyone, rather than just increasing compute. It seems memory bandwidth (in GPUs) is becoming more of a bottleneck than anything else IMHO. But NVIDIA CUDA is still an advantage. Grad students grew up on it, trained on it and are so familiar with it, it will take a long time to displace. Yeah, RoCM helps but, IT needs more. Open Sourcing all the CUDA code and its derivatives could be an answer, if anybody’s listening. Jason talked about AI rack and data center power requirements going through the roof and mentioned SMR (small modular [nuclear] reactors) as one solution. When buying a nuclear power plant is just not an option, SMRs can help. They can be trucked and installed (mostly) anywhere. Keith saw a truckload of SMRs on the highway on one of his road trips. And last but not least, Apple just announced RDMA over Thunderbolt. And the (Youtube) airwaves have been lighting up with studio Macs being clustered together with sufficient compute power to rival a DGX. Of course it’s Apples MLX running rather than CUDA, and only so many models work on MLX, but it’s a start at democratizing AI. VMware Broadrcom’s moves remind Jason of what IBM did with Z. Abandoning the low end, milk the high end forever. If you want vSphere better think about purchasing VCF. Keith mentioned if a company has a $100M cloud spend, they could save some serious money (~20%), going to VCF. But it’s not a lift and shift. Running a cloud on prem requires a different mindset than running apps in the cloud. Welcome to the pre-cloud era, where every IT shop did it all. Component Pricing Jason said that DRAM pricing has gone up 600% in a matter of weeks. Our consensus view is it’s all going to AI data centers. With servers having a TB of DRAM, GPUs with 160GB of HBM per, and LPDDR being gobbled up for mobile/edge compute everywhere is it any doubt that critical server (sub-) components are in high demand. Hopefully, the Fabs will start to produce more. But that assumes Fab’s have spare capacity and DRAM demand is function of price. There are hints that neither of these are true anymore. Mega data centers are not constrained by capital, yet, and most Fabs are operating flat out producing as many chips as they can. So DRAM pricing may continue to be a problem for some time to come. Speaking of memory, there was some discussion on memory tiering startups taking off with high priced memory. One enabler for that is the new UALink interconnect. It’s essentially an open source, chip-to-chip interconnect technology, over PCIe or Ethernet. UAlink solutions can connect very high speed components beyond the server itself to support a scale out network of accelerators, memory and CPUs in a single rack. It’s early yet but Meta specs for an OCP wide form factor rack was released in the AMD Helios OGP 72GPU rack that uses UALink tech today, More to come we’re sure. Keith Townsend, The CTO Advisor, Founder & Executive Strategist | Advisor to CIOs, CTOs & the Vendors Who Serve Them Keith Townsend (@CTOAdvisor) is a IT thought leader who has written articles for many industry publications, interviewed many industry heavyweights, worked with Silicon Valley startups, and engineered cloud infrastructure for large government organizations. Keith is the co-founder of The CTO Advisor, blogs at Virtualized Geek, and can be found on LinkedIN. Jason Collier, Principal Member Of Technical Staff at AMD, Data Center Solutions Group Jason Collier (@bocanuts) is a long time friend, technical guru and innovator who has over 25 years of experience as a serial entrepreneur in technology. He was founder and CTO of Scale Computing and has been an innovator in the field of hyperconvergence and an expert in virtualization, data storage, networking, cloud computing, data centers, and edge computing for years. He’s on LinkedIN. He’s currently working with AMD on new technology and he has been a GreyBeards on Storage co-host since the beginning of 2022

Keith and I attended AIFD7 a couple of weeks back and Articul8 AI presented at one session (see videos of their session here). Given all the press on LLMs and GenAI, there are only a few non-GenAI solutions in the market today. Articul8 is one of these and represents a different way of deploying AI for industry. Dr. Arun Subramaniyan, Founder and CEO of Articul8 (LinkedIn), discussed their approach on AI for industries at their session. With all the press on GenAI, agentic AI and LLMs, it’s hard to remember that AI has had a long history in helping various verticals address their challenges. Articul8 AI was founded only 2 years ago, but already has significant footprints in a number of industry sectors, such as aerospace, telecom, (electrical) energy, etc. Articul8 AI is all about deploying domain specific models trained to focus on select industry challenges. Listen to the podcast to learn more. Articul8 AI can operate on prem, in your VPC or in their own cloud . The solution is deployed on infrastructure sized to your organizations specific requirements and starts (processing) ingesting corporate data the moment it’s enabled. It can run on something as small as an 8GPU server to large clusters with many (1000s of) GPUs. Articul8 doesn’t host or store any corporate data on this infrastructure, just metadata describing data and relationships between them. Arun said that within 24 hours, Articul8 AI has enough of an organizations knowledge map, what they call the shape of data, to process up to 95% of an organizations requests. Their AIFD7 demo shows a sort of 3D visual of a knowledge map. And it’s interesting that every query or request changes the knowledge map in subtle ways. For the industries it supports, Articul8 AI is typically embedded into “systems of record”. There are very few AI solutions today like this. perhaps coding agents for software development firms and recommendation engines for online retailers, but that’s about it. For Articul8 AI to support a new domain or vertical, takes significant domain expertise and data. In some cases, they have partnered with industry associations to gain expertise and data. For those organizations that have contributed data or IP to support a new domain, Articul8 AI can share revenue from other organizations that adopt their solutions. One can see the current verticals Articul8 AI supports. One item of interest is their cross domain models. They have one cross-domain model trained to interpret and understand tables/spreadsheets/”structured image data”, another to understand logs or time series data and a third focused on converting text to database queries. Most GenAI/LLMs struggle to understand tables and spreadsheet data well. The other thing about tables and spreadsheets is that most corporations could not exist without them. By providing a cross domain table understanding model they have opened up vast troves of corp data which was just too inscrutable for LLM AI to understand and process before. Finally, Artucul8 AI has two offerings currently available on AWS Marketplace one of which is a LLM evaluation tool and the other a network topology log analyzer tool. The LLM evaluator, when provided a prompt, will return which current LLM could handle that prompt best and is callable via API. The topology service can analyze time series logs from networking and other gear and show network topology from logs alone. Dr. Arun Subramaniyan, Founder & CEO Articul8 AI Arun Subramaniyan is the founder & CEO of Articul8, where he is building a domain-specific GenAI Platform. Previously, he led the Cloud & AI Strategy team at Intel where he was responsible for establishing and driving the overall AI strategy globally, and was focused on democratizing AI in a sustainable fashion. Arun joined Intel from Amazon Web Services (AWS), where he led the Extreme-scale computing solution team spanning Machine Learning, Quantum Computing, High Performance Computing (HPC), Autonomous Vehicles, and Autonomous Computing. His team was responsible for developing solutions across all areas of HPC, quantum computing and large-scale machine learning applications, spanning a $1B+ portfolio, and he grew the businesses 2-3x over two years. Arun’s primary areas of research focus are Bayesian methods, global optimization, probabilistic deep learning for large scale applications, and distributed computing. He is an Executive Fellow at Harvard Business School, where he teaches courses on Generative AI for Business Leaders. He enjoys working at the intersection of massively parallel computing and modeling large-scale systems. Arun is a prolific researcher with a Ph.D. in Aerospace Engineering from Purdue University with 34 granted patents (60+ filed), 50+ international publications that have been cited more than 1600 times with a h-index of 16. He is also a recipient of the Hull Award from GE, which honors technologists for their outstanding technical impact.

SNIA’s Storage Developer Conference (SDC) was held last week in CA and although I didn’t attend I heard it was quite a gathering. Just prior to the show, I was talking with Jason about the challenges of storage for AI and he mentioned that SNIA had a new Storage.AI initiatdze focused on these issues. I called Dr. J Metz, Chair of SNIA & Technical Director @ AMD (@drjmetz, blog) and asked if the wanted to talk to us about SNIA’s new initiative. Storage.AI is a SNIA standards development community tasked with addressing the myriad problems AI has with data. Under its umbrella, a number of technical working groups (TWGs) will work on standards to improve AI data access, Just about every IT vendor in the universe is listed as a participating company in the initiative. Listen to the podcast to learn more. We started discussing Dr. J’s current roles at SNIA and AMD and how SDC went last week. It turns out, it was the best attended SDC ever and Dr. J’s keynote on Storage.AU was a highlight of the show. The storage/data needs for AI span a wide spectrum of activities or workloads. Dr. J spoke on the lengthy data pipeline, e.g. ingest, prep/clean, transform, train, checkpoint/reload, RAG upload/update and inference to name just a few. In all these diverse activities, storage’s job is getting the right data bits to the right process (GPU/accelerators for training) throughout the pipeline. Inferencing has somewhat less of a convoluted data journey but is still complex and performance critical. Te take just one component of the data pipeline checkpointing is a data intensive process. When training a multi-billion parameter model or, dare I say, multi-trillion parameter model with 10K to Million’s of GPUs, failure’s happen, often. Checkpoints are the only way model training can make progress in the face of significant GPU failures. And of course, any checkpoint needs to be reloaded to verify it’s correct. So checkpointing and reloading is an IO activity that happens constantly when models are trained. Checkpoints essentially save the current model parameters during training. Speeding up checkpoint/reload could increase AI model training throughput considerably And of course, GPUs and the power they consume are an expensive activity. When one has 1000’s to Millions of GPUs in a data center, having them sit idle is a vast waste of resources. Anything to help speed up accelerator data access could potentially save millions. In the old days compute, storage and networking were isolated/separate silos of technology. Nowadays, the walls between them have been blown away, mostly by the advent of AI. Dr. J talks about first principles, such as the speed of light that determines the time it takes for data to move from one place to another. These limits exist throughout IT infrastructure. But OS stacks surrounding these activities have spawned layer upon layer of software to do these actions. If one can wipe the slate clean, infrastructure activities can get closer to those first principles and reduce overhead SNIA has current TWGs focused on a number of activities that could help speed up AI IO. We talked about SNIA’s Smart Data Acceleration Initiative (SDXI), but there are others in process as well. But SNIA has also identified a few new ones they plan to fire up such as GPU direct access bypass and GPU-Initiated IO to address other gaps in Storage.AI. In today’s performance driven AI environments, proprietary solutions are often developed to address some of these same issues. We ended up discussing the role of standards vs. proprietary solutions in IT in general and in today’s AI infrastructure. Yes there’s a place for proprietary solutions and there’s also a place for standards. Sometimes they merge, sometimes not, but they can often help inform each other on industry trends and challenges. I thought that proprietary technologies always seem to emerge early and then transition to standards over time. Dr. J said it’s more of an ebb and flow between proprietary and standards, and mentioned as one example the ESCON-FC-FICON-Fabric proprietary/standards activities from last century. As always It was an interesting conversation with Dr. J and Jason and I look forward to seeing how SNIA’s Storage.AI evolves over time. Dr. J. Metz, Chair and Chief Executive of SNIA & Technical Director, AMD J is Technical Director for Systems Design for AMD where he works to coordinate and lead strategy on various industry initiatives related to systems architecture, including advanced networking and storage. He has a unique ability to dissect and explain complex concepts and strategies, and is passionate about the inner workings and application of emerging technologies.J has previously held roles in both startup and Fortune 100 companies as a Field CTO, R&D Engineer, Solutions Architect, and Systems Engineer. He is and has been a leader in several key industry standards groups, currently Chair of SNIA as well as the Chair of the Ultra Ethernet Consortium (UEC). Previously, he was on the board of the Fibre Channel Industry Association (FCIA) and Non-Volatile Memory Express (NVMe) organizations. A popular blogger and active on Twitter, his areas of expertise include both storage and networking for AI and HPC environments.Additionally, J is an entertaining presenter and prolific writer. He has won multiple awards as a speaker and author, writing over 300 articles and giving presentations and webinars attended by over 10,000 people. He earned his PhD from the University of Georgia.

Jim Handy, General Director at Objective Analysis and I were at FMS25 in Santa Clara last week and there was a lot of news going around. Jim’s been on our show just about every year to discuss FMS news, And with the recent focus beyond flash, it’s even harder for one person to keep up. Much of the discussion at FMS was on HBM4, new QLC capacity points, UAlink/UCe for chiplets, 100M IOP SSDs, and more. Listen to the podcast to learn more. Th.ere was not as much on CXL as in past shows. and ditto on increasing layer counts to drive more NAND capacity. A couple of years ago layer counts were all they talked about. And CXL was the major change to hit the data center. Jim’s view (and Jason’s) was that CXL was as a way for hyperscalers to make use of DDR4 DRAM but that need has passed now. As for layer counts they are still going up but not as fast. And the economics of 3D scaling now have to compete with 2D scaling and “virtual scaling”. But UAlink and UCe were active topics both of which are used to tie together chiplets in CPUs to build SoCs. SSD vendors are starting to use chiplet architectures to build their massive capacity SSDs and UAlink/UCe would be a way to architect these. SLC NAND is back to support very high performance SSDs or as a replacement for SCM (storage class memory or Optane). One vendor talked about reaching 100M (random 512B read) IOPS for a single SSD. Current SCL flash can do ~10M IOPS, next gen is speced to do ~30M and the one following would be 100M. One challenge is that current SSDs do 4Kbyte IO and it still takes a msec. or so to erase a page and reading a page isn’t that fast. But the performance is for read only activity. HBM4 was one topic at the show but the newest wrinkle was HB Flash, or putting SSDs behind HBM to support GPU caching (SSD to HBM to GPU). This would allow more data to be quickly accessed by a GPU. Jim also mentioned that there’s some interest in narrowing HBM access width, currently 1Kb and increasing to 2Kb with HBM4. This width, and all the pins it requires, limits how many HBM chips one can surround a GPU with. If HBM had a narrower interface more HBM chips could surround a GPU, increasing memory size and perhaps memory bandwidth. HBM4 seems to be going the wrong way but with narrower width HBM, they could easily double the number of HBM chips surrounding a GPU. They were also showing off a 40 SSD 2U chassis using E.2 form factor SSDs. Pretty impressive and given the capacity on offer a lot of storage per RU. Speaking of capacity one vendor announced a 246TB QLC SSD, roughly a 1/4PB in a single SSD. With 24 of these per 2U shelf, one could have a >1/10 Exabyte, (>100 PB) in a 40U rack. It looks like no end in sight for SSD capacities. And we aren’t even talking about PLC yet. At the other end of SSD capacity, it appears that M.2 SSDs were getting hotter on one side (controller side) than the other, throttling performance. So one vender decided to provide heat (liquid cooling) pipes between the two sides to equalize thermal load. Jim Pappas (lately of Intel) won the lifetime achievement award from FMS. Jim’s accomplishments span a wide swath of storage technology but at the award ceremony he waxed on his work on the USB connector. He said his will stipulates that once he is interned in the ground, they are to take out the casket and spin it around 180 degrees and put it back down again. There were quite a number of side topics not directly related to FMS25 on the podcast which were interesting in their own right, but I think i’ll leave it here. Jim Handy, General Director Objective Analysis Jim Handy of Objective Analysis has over 35 years in the electronics industry, including 20 years as a leading semiconductor and SSD industry analyst. Early in his career he held marketing and design positions at leading semiconductor suppliers including Intel, National Semiconductor, and Infineon. A frequent presenter at trade shows, Mr. Handy is known for his technical depth, accurate forecasts, widespread industry presence and volume of publication. He has written hundreds of market reports, articles for trade journals, and white papers, and is frequently interviewed and quoted in the electronics trade press and other media.  He posts blogs at www.TheMemoryGuy.com, and www.TheSSDguy.com

Luke Norris (@COentrepreneur), CEO and Co-Founder, Kamiwaza AI, is a serial entreprenaur in Silverthorne CO, where the company is headquartered.. They presented at AIFD6 a couple of weeks back and the GreyBeards thought it would be interesting to learn more about what they were doing, especially since we are broadening the scope of the podcast, to now be GreyBeards on Systems. Describing Kamiwaza AI is a bit of a challenge. They settled on “AI orchestration” for the enterprise but it’s much more than that. One of their key capabilities is an inference mesh which supports accessing data in locations throughout an enterprise across various data centers to do inferencing, and then gathering replies/responses together, aggregating them into one combined response. All this without violating HIPPA, GDPR or other data compliance regulations. Kamiwaza AI offer an opinionated AI stack, which consists of 155 components today and growing that supplies a single API to access any of their AI services. They support multi-node clusters and multiple clusters, located in different data centers, as well as the cloud. For instance, they are in the Azure marketplace and plans are to be in AWS and GCP soon. Most software vendors provide a proof of concept, Kamiwaza offers a pathway from PoC to production. Companies pre-pay to install their solution and then can use those funds when they purchase a license. And then there’s their (meta-)data catalogue. It resides in local databases (and replicated maybe) throughout the clusters and is used to identify meta data and location information about any data in the enterprise that’s been ingested into their system. Data can be ingested for enterprise RAG databases and other services. As this is done, location affinity and metadata about that data is registered to the data catalogue. That way Kamiwaza knows where all of an organization’s data is located, which RAG or other database it’s been ingested into and enough about the data to understand where it might be pertinent to answer a customer or service query. Maybe the easiest way to understand what Kamiwaza is, is to walk through a prompt.  A customer issues a prompt to a Kamiwaza endpoint which triggers, A search through their data catalog to identify what data can be used to answer that prompt. If all the data resides in one data center, the prompt can be handed off to the GenAI model and RAG services at that data center.  But if the prompt requires information from multiple data centers, Separate prompts are then distributed to each data center where RAG information germane to that prompt is located As each of these generate replies, their responses are sent back to an initiating/coordinating cluster Then all these responses are combined into a single reply to the customer’s prompt or service query. But the key point is that data located in each data center used to answer the prompt are NOT moved to other data centers. All prompting is done locally, at the data center where the data resides.  Only prompt replies/responses are sent to other data centers and then combined into one comprehensive answer.  Luke mentioned a BioPharma company that had genonome sequences located in various data regimes, some under GDPR, some under APAC equivalents, others under USA HIPPA requirements. They wanted to know information about how frequent a particular gene sequence occurred. They were able to issue this as a prompt at a single location which spun up separate, distributed prompts for each data center that held appropriate information. All those replies were then transmitted back to the originating prompt location and combined/summarized. Kamiwaza AI also has an AIaaS offering. Any paying customer is offered one (AI agentic) outcome per month per cluster license. Outcomes could effectively be any AI application they would like to perform. One outcome he mentioned included: A weather-risk researcher had tons of old weather data in a multitude of formats, over many locations, that had been recorded over time. They wanted to have access to all this data so they can tell when extreme weather events had occurred in the past. Kamiwaza AI assigned one of their partner AI experts to work with the researcher to have an AI agent comb through these archives, transform and clean all the old weather data into HTML data more amenable to analysis .  But that was just the start.. They really wanted to understand the risk of damage due to the extreme weather events. So the AI application/system was then directed to go and gather from news and insurance archives, any information that identified the extent of the damage from those weather events.  He said that today’s AgenticAI can implement a screen mouse click and perform any function that an application or a human could do on a screen. Agentic AI can also import an API and infer where an API call might be better to use than a screen GUI interaction. He mentioned that Kamiwaza can be used to generate and replace a lot of what enterprises do today with Robotics Process Automation (RPAs). Luke feels that anything an enterprise was doing with RPA’s can be done better with Kamiwaza AI agents. SaaS solution tasks are also something AgenticAI can easily displace . Luke said at one customer they went from using SAP APIs to provide information to SAP, to using APIs to extract information from SAP, to completely replacing the use of SAP for this task at the enterprise.  How much of this is fiction or real is subject of some debate in the industry. But Kamiwaza AI is pushing the envelope on what can and can’t be done. And with their AI aaS offering, customers are making use of AI like they never thought possible before. . Kamiwaza AI has a community edition, a free download that’s functionally restricted, and provides a desktop experience of Kamiwaza AI’s stack. Luke sees this as something a developer could use to develop to Kamiwaza APIs and test functionality before loading on the enterprise cluster.  We asked where they were finding the most success. Luke mentioned anyone that’s heavily regulated, where data movement and access were strictly constrained. And they were focused on large, multi-data center, enterprises. Luke mentioned that Kamiwaza AI has been doing a number of hackathons with AI Tinkerers around the world. He suggested prospects take a look at what they have done with them and perhaps join them in the next hackathon in their area. Luke Norris, CEO & Co-Founder, Kamiwaza AI Luke Norris is the co-founder of Kamiwaza.AI, driving enterprise AI innovation with a focus on secure, scalable GenAI deployments. With extensive experience raising over $100M in venture capital and leading global AI/ML deployments for Fortune 500 companies. Luke is passionate about enabling enterprises to unlock the full potential of AI with unmatched flexibility and efficiency.

It’s time once again for our annual YE GBoS podcast. This year we have Howard back making a guest appearance with our usual cast of Jason and Keith in attendance. And the topic de jour seemed to be AI rolling out to the enterprise and everywhere else in the IT world.  We led off with our discussion from last year, AI (again) but then it was all about new announcements, new capabilities and new functionality. This year it’s all about starting to take AI tools and functionality and make them available to help optimize organizational functionality. We talked some about RAGs and Chatbots but these seemed almost old school. Agentic AI Keith mentioned Agentic AI which purports to improve businesses by removing/optimizing intermediate steps in business processes. If one can improve human and business productivity by 10%, the impact on the US and world’s economies would  be staggering. And we’re not just talking about knowledge summarization, curation, or discussion, agentic AI takes actions that would have been previously done by a human, if done at all.   Manufacturers could use AI agents to forecast sales, allowing the business to optimize inventory positioning to better address customer needs.  Most, if not all, businesses have elaborate procedures which require a certain amount of human hand holding. Reducing human hand holding, even a little bit, with AI agents, that never slees, and can occasionally be trained to do better, could seriously help the bottom and top lines for any organization  We can see evidence of Agentic AI proliferating in SAAS solutions, i.e., SalesForce, SAP, Oracle and all others are spinning out Agentic AI services. I think it was Jason that mentioned GEICO, a US insurance company, is re-factoring, re-designing and re-implementing all their applications to take advantage of Agentic AI and other AI options.  AI’s impact on HW & SW infrastructure The AI rollout is having dramatic impacts on both software and hardware infrastructure. For example, customers are building their own OpenStack clouds to support AI training and inferencing. Keith mentioned that AWS just introduced S3 Tables, a fully managed services meant to store and analyze massive amounts of tabular data for analytics. Howard mentioned that AWS’s S3 Tables had to make a number of tradeoffs to use immutable S3 object storage. VAST’s Parquet database provides the service without using immutable objects. Software impacts are immense as AI becomes embedded in more and more applications and system infrastructure. But AI’s hardware impacts may be even more serious. Howard made mention of the power zero sum game, meaning that most data centers have a limited amount of power they support. Any power saved from other IT activities are immediately put to use to supply more power to AI training and infererencing. Most IT racks today support equipment that consumes 10-20Kw of power. AI servers will require much more Jason mentioned one 6u server with 8 GPUS that cost on the order of 1 Ferrari ($250K US), draws 10Kw of power, with each GPU having 2-400 GigE links not to mention the server itself having 2-400 GigE links. So a single 6U (GPU) server has 18-400GbE links or could need 7.2Tb of bandwidth. Unclear how many of these one could put in a rack but my guess is it’s not going to be fully populated. 6 of these servers would need >42Tb of bandwidth and over 60Kw of power and that’s not counting the networking and other infrastructure required to support all that bandwidth.   Speaking of other infrastructure, cooling is the other side of this power problem. It’s just thermodynamics, power use generates heat, that heat needs to be disposed of. And with 10Kw servers we are talking a lot of heat. Jason mentioned that at this year’s SC24 conference, the whole floor was showing off liquid cooling.  Liquid cooling was also prominent at OCP. At the OCP summit this year Microsoft was talking about deploying near term 150Kw racks and down the line 1Mw racks. AI’s power needs are why organizations around the world are building out new data centers in out of the way places that just so happen to have power and cooling nearby.  Organizations have an insatiable appetite for AI training data. And good (training) data is getting harder to find. Solidigm latest 122TB SSD may be coming along just when the data needs for AI are starting to take off. SCI is pivoting We could have gone on for hours on AI’s impact on IT infrastructure, but I had an announcement to make. Silverton Consulting will be pivoting away from storage to a new opportunity that is based in space. I discuss this on SCI’s website but the opportunities for LEO and beyond services are just exploding these days and we want to be a part of that.  What that means for GBoS is TBD. But we may be transitioning to something more broader than just storage. But heck we have been doing that for years. Stay tuned, it’s going to be one hell of a ride Jason Collier, Principal Member Of Technical Staff at AMD, Data Center and Embedded Solutions Business Group Jason Collier (@bocanuts) is a long time friend, technical guru and innovator who has over 25 years of experience as a serial entrepreneur in technology. He was founder and CTO of Scale Computing and has been an innovator in the field of hyperconvergence and an expert in virtualization, data storage, networking, cloud computing, data centers, and edge computing for years. He’s on LinkedIN. He’s currently working with AMD on new technology and he has been a GreyBeards on Storage co-host since the beginning of 2022 Howard Marks, Technologist Extraordinary and Plenipotentiary at VAST Data Howard Marks is Technologist Extraordinary and Plenipotentiary at VAST Data, where he explains engineering to customers and customer requirements to engineers. Before joining VAST, Howard was an independent consultant, analyst, and journalist, writing three books and over 200 articles on network and storage topics since 1987 and, most significantly, a founding co-host of the Greybeards on Storage podcast. Keith Townsend, President of The CTO Advisor, a Futurum Group Company Keith Townsend (@CTOAdvisor) is a IT thought leader who has written articles for many industry publications, interviewed many industry heavyweights, worked with Silicon Valley startups, and engineered cloud infrastructure for large government organizations. Keith is the co-founder of The CTO Advisor, blogs at Virtualized Geek, and can be found on LinkedIN.

Long time friend, Enrico Signoretti (LinkedIn), VP Product and Partnerships, Cubbit, used to be a common participant at Storage Field Day (SFD) events and I’ve known him since we first met there. Since then, he’s worked for a startup and a prominent analyst firms. But he’s back at another startup and this one looks like it’s got legs. Cubbit offers Distributed S3 compatible object storage that offers geo-distribution and geo-fencing for object data, in which the organization owns the hardware and Cubbit supplies the software. There’s a management component, the Coordinator, which could run on your hardware or as a SaaS service they provide but other than that, IT controls the rest of the system hardware. Listen to the podcast to learn more. Cubbit comes in 3 components: One or more Storage nodes which includes their agent software running ontop of a linux system with direct attached storage. One or more Gateway nodes which provides S3 protocol acces to the objects stored on storage nodes. Typical S3 access points https://S3.company_name, com/… points to either a load balancer, front end or one or more Gateway nodes. Gateway nodes provide the mapping between the bucket name/object identifier and where the data currently resides or will reside. One Coordinator node which provides the metadata to locate the data for objects, manage the storage nodes, gateways and monitor the service. The Coordinator node can be a SaaS service supplied by Cubbit or a VM/bare metal node running Cubbit Coordinator software. Metadata is protected internally within the Coordinator node. With these three components one can stand up a complete, geo-distributed/geo-fenced, S3 object storage system which the organization controls. Cubbit encrypts data as it at the gateway and decrypts data when accessed. Sign-on to the system uses standard security offerings. Security keys can be managed by Cubbit or by standard key management systems. All data for an object is protected by nested erasure codes. That is 1) erasure code within a data center/location over its storage drives and 2) erasure code across geographical locations/data centers.. With erasure coding across locations, customer with say 10 data center locations can have their data stored in such a fashion that as long as at least 8 data centers are online they still have access to their data, that is the Cubbit storage system can still provide data availability. Similarly for erasure coding within the data center/location or across storage drives, say with 12 drives per stripe, one could configure lets say 9+3 erasure coding, where as long as 9 of the drives still operate, data will be available. Please note the customer decides the number of locations to stripe across for erasure coding, and diet for the number of storage drives. The customer supplies all the storage node hardware. Some customers start with re-purposed servers/drives for their original configuration and then upgrade to higher performing storage-servers-networking as performance needs change. Storage nodes can be on prem, in the cloud or at the edge. For adequate performance gateways and storage nodes (and coordinator nodes) should be located close to one another. Although Coordinator nodes are not in the data path they are critical to initial object access. Gateways can provide a cache for faster local data access.. Cubbit has recommendations for Gateway server hardware. And similar to storage nodes, Gateways can operate at the edge, in the cloud or on prem. Use cases for the Distributed S3 storage include: As a backup target for data elsewhere As a geographically distributed/fenced object store. As a locally controlled object storage to feed AI training/inferencing activity. Most backup solutions support S3 object storage as a target for backups. Geographically distributed S3 storage means that customers control where object data is located. This could be split across a number of physical locations, the cloud or at the edge. Geographically fenced S3 storage means that the customer controls which of its many locations to store an object. For GDPR countries with multi-nation data center locations this could provide the compliance requirements to keep customer data within country. Cubbit’s distributed S3 objects storage is strongly consistent in that an object loaded into the system at any location is immediately available to any user accessing it through any other gateway. Access times vary but the data will be the same regardless of where you access it from. The system starts up through an Ansible playbook which asks a bunch of questions and loads and sets up the agent software for storage nodes, gateway nodes and where applicable, the coordinator node. At any time, customers can add more gateways or storage nodes or retire them. The system doesn’t perform automatic load balancing for new nodes but customers can migrate data off storage nodes and onto other ones through api calls/UI requests to the Coordinator. Cubbit storage supports multi-tenancy, so MSPs can offer their customers isolated access. Cubbit charges for their service on data storage under management. Note it has no egress charges, and you don’t pay for redundancy. But you do supply all the hardware used by the system. They offer a discount for M&E customers as the metadata to data ratio is much smaller (lots of large files) than most other S3 object stores (mix of small and large files). Cubbit is presently available only in Europe but will be coming to USA next year. So, if you are interested in geo-distributed/geo-fenced S3 object storage that you control and can be had for much cheaper than hyperscalar object storage, check it out. Enrico Signoretti, VP Products & Partnerships Enrico Signoretti has over 30 years of experience in the IT industry, having held various roles including IT manager, consultant, head of product strategy, IT analyst, and advisor. He is an internationally renowned visionary author, blogger, and speaker on next-generation technologies. Over the past four years, Enrico has kept his finger on the pulse of the evolving storage industry as the Head of Research Product Strategy at GigaOm. He has worked closely and built relationships with top visionaries, CTOs, and IT decision makers worldwide. Enrico has also contributed to leading global online sites (with over 40 million readers) for enterprise technology news.