Bernstein on Seven Memory Types: After HBM, DRAM and NAND Vie for the Next Trillion-Dollar Market

Over the past two years, the AI memory narrative has centered almost entirely on HBM. But as large models shift from training to large-scale inference, simply adding more HBM is no longer enough to solve all the problems.

In its latest global memory report, Bernstein points out that different AI workloads have markedly different memory requirements: training emphasizes compute and bandwidth, the decode phase of inference relies more on capacity, RAG needs large-scale databases, and agent workflows increase the load on both traditional and AI servers.

This means the changes brought by AI are cascading from HBM down to system DRAM, SSDs, HDDs, and even tape storage. Around the "memory wall," the industry is inserting new products between existing tiers, seeking a new balance among performance, capacity, and cost.

The ceiling of inference scale may hinge on KV Cache

During large-model training, GPUs and HBM remain at the core.

Training requires frequent reads of model parameters and intermediate data, demanding high compute power and memory bandwidth. But training a large model does not rely on HBM alone: raw datasets need to be stored on lower-cost media; before data enters the GPU, it typically must be cached and preprocessed by system DRAM and local SSDs; and training that lasts weeks or even months requires periodic checkpoint saves to avoid restarting from scratch due to hardware or software failures.

Thus, a single large training run actually engages the full hierarchy from HBM and system DRAM to local SSDs and network storage.

Entering the inference phase, memory requirements diverge further.

Inference can generally be split into prefill and decode. Prefill processes user input and generates the first token, performing large-scale matrix operations and leaning toward "compute-bound." In this phase, GPU utilization and HBM bandwidth matter more, and the common metric is time-to-first-token.

The decode phase is different. The model generates tokens one by one and, when producing a new token, must reference previously generated information. To avoid recomputation, systems typically store this data in KV Cache.

KV Cache has two key characteristics: its capacity grows linearly with context length, and each user requires an independent cache. Therefore, as context lengthens and concurrent users increase, both factors jointly drive up memory usage.

Bernstein believes that in large-scale AI deployments, KV Cache memory can exceed model weights, becoming the primary factor limiting concurrent users and context window size. How many users a model can serve and how long a context it can maintain ultimately directly affect revenue scale.

From this perspective, the competitive focus in the inference era is not just how much compute a chip can deliver, but also how cheaply the system can store and read an ever-expanding context.

RAG and agents push demand toward traditional memory

The proliferation of RAG and agents is spreading AI storage demand beyond HBM.

RAG mainly consists of database construction and database retrieval. When building the database, the system must process large volumes of unstructured data such as PDFs, web pages, and code, then convert them into searchable vectors and indexes. This process relies more on high-capacity SSDs and system DRAM, with HBM playing a relatively limited role.

Once the database is built, user queries are first converted into vectors and then matched against the database content. Vector generation can be done quickly on GPUs and HBM, but the actual search typically depends more on system DRAM. The retrieved results are then combined with the user's question and enter the normal prefill and decode flow.

Agent workflows impose an even heavier burden.

Traditional conversations are usually a single "input-model-output" call, whereas agents need to break goals into multiple steps, call other models or external tools, save intermediate results, and replan based on feedback. The output of each call may become the input for the next model call.

This simultaneously increases two types of demand: on one hand, tool calls and non-AI tasks require more CPUs and system memory; on the other, the continuous passing of context among multiple models rapidly expands prefill, decode, and KV Cache loads.

Therefore, the development of agent applications benefits not only GPUs and HBM but may also boost demand for server DRAM, enterprise SSDs, and lower-cost storage media.

Between HBM and SSDs, new memory tiers are emerging

Traditional server memory systems can be roughly divided into processor caches, system DRAM, local SSDs, and shared storage. AI servers add HBM to this structure, but HBM has limited capacity and high cost, making it difficult to hold all data.

The industry's current approach is to insert new products between different tiers.

CXL attempts to integrate physically dispersed memory into a shared resource pool, allowing CPUs, GPUs, and expansion devices to access DRAM more flexibly. Some products also use DRAM or SRAM as a cache combined with NAND, reducing cost while shortening access latency.

Nvidia's "Storage Next" initiative seeks to shift some storage management from the CPU to the GPU and give NAND latency, IOPS, and data access granularity closer to DRAM. Kioxia's GP series SSDs based on XL-FLASH are representative of this direction.

CMX primarily targets KV Cache. It deploys SSDs in independent data nodes, connected to compute nodes via DPUs, Ethernet, and switch chips. The goal is to share inference context across different GPUs, reduce duplicate storage, and break through the memory capacity limits of a single server.

These approaches all point to the same trend: AI systems cannot keep all active data in HBM long-term; data must be distributed across tiers according to access frequency and latency requirements.

Hot data stays in HBM, some context moves to system DRAM or high-performance SSDs, and colder data continues to sink to standard SSDs, HDDs, or even tape. The finer the memory hierarchy, the more likely the system can balance performance and cost.

New technologies are emerging rapidly, but commercialization remains uncertain

Around the "memory wall," the industry has proposed multiple new paths.

Samsung's zHBM plans to stack HBM on top of the processor, further shortening data transfer distances. However, this design must handle heat generated by the GPU while imposing higher requirements on wafer-level hybrid bonding yield and cost.

Nvidia's NVHBM has the base die designed by Nvidia and possibly manufactured by TSMC. This approach could reduce power consumption and increase bandwidth, but it may also diminish the design and manufacturing value of memory makers in HBM base dies. As products standardize, some added value may shift from memory makers to Nvidia and foundries.

HBF, promoted by SanDisk and SK Hynix, aims to use NAND to deliver bandwidth close to HBM while achieving larger capacity and lower unit cost. However, NAND and DRAM still have significant gaps in latency and performance, and HBF must cross multiple technology levels, making implementation far from easy.

Intel's ZAM attempts to rotate DRAM dies 90 degrees to improve heat dissipation, targeting commercialization by fiscal 2029; Qualcomm's HBC uses LPDDR and traditional packaging, sacrificing some performance to bypass CoWoS costs.

Additionally, PIM seeks to add compute capability directly into memory chips to reduce data movement between processors and memory. But this changes the existing computing architecture, requiring adaptation across processors, software, and networks, and it also disrupts the highly mature division of labor between logic and memory chips. Bernstein believes industry adoption remains limited.

Thus, the rapid increase in new solutions does not mean all paths will form large-scale markets. Compatibility with existing software and hardware ecosystems, cost advantages, and whether supply chain players can reach a balance of interests will determine the final commercialization outcome.

AI memory beneficiaries will not be limited to HBM makers

From an investment perspective, Bernstein's view is relatively clear: AI's pull on the memory industry is spreading from a few high-end products to more tiers.

HBM remains the core for training and high-performance inference, and Samsung Electronics, SK Hynix, and Micron will continue to benefit from high-bandwidth memory demand. But as inference scales up, the importance of system DRAM and NAND will rise. KV Cache overflow, RAG databases, and the large amounts of intermediate data generated by agents will also increase demand for SSDs and shared storage.

Colder data will continue to sink further. Bernstein notes that AI-driven data growth has already begun to benefit HDDs; in some scenarios, due to insufficient NAND and HDD capacity, tape—traditionally used mainly for archiving—is also seeing increased demand.

The report maintains "Outperform" ratings on Samsung Electronics, SK Hynix, Micron, SanDisk, Seagate, and Western Digital. Among them, Samsung Electronics, SK Hynix, and Micron correspond to DRAM and HBM, SanDisk benefits from NAND and HBF, and Seagate and Western Digital correspond to lower-cost high-capacity storage. Kioxia is rated "Underperform."

However, the core value of this report is not in listing a batch of new technology acronyms, but in redefining the boundaries of the AI memory market.

In the training era, bottlenecks were concentrated in GPUs and HBM; in the inference and agent era, bottlenecks are spreading across the entire memory hierarchy. The future competition in AI infrastructure depends both on how fast chips can compute and on whether data can flow efficiently among HBM, DRAM, NAND, and shared storage at sufficiently low cost.