NVIDIA Expands Memory Optimization for AI Systems
NVIDIA introduces full-stack codesign to tackle memory challenges in AI, improving efficiency across hardware and software.
What happened
NVIDIA is addressing the rising memory demands in AI by implementing full-stack codesign, which streamlines memory use throughout the AI development process.
The company has also introduced NVFP4, a format that reduces model weight storage compared to FP8, and has been adopted by several recent models.
Why it matters
As AI systems grow more complex, the memory hierarchy—particularly high-bandwidth memory (HBM)—faces increasing strain due to tasks like inference and agentic applications, which require retaining context across multiple interactions.
Key details
NVIDIA’s NVL72 architecture allows model weights to be spread across up to 72 GPUs, reducing the memory load per GPU. The GB300 NVL72 platform achieves up to 10x higher throughput per dollar than the H200 platform on specific workloads.
The Vera Rubin NVL72 platform offers 30x higher throughput per dollar than the GB300 NVL72, with a 3% increase in HBM capacity per GPU. NVIDIA’s CMX context memory storage is designed to efficiently reuse inactive context, reducing pressure on HBM.
Sources
Related stories
Meta Enhances Ad Retrieval Efficiency with New System
Meta's Andromeda system now scores ads at near ranking-level performance, handling over one billion ads per second.
Microsoft Research Introduces Synthetic Data Platform for Copilot
Microsoft Research has created a synthetic data platform to test Copilot at enterprise scale without using real customer data.
IBM Research Launches llm-d 1.0 for Open-Source AI Inference
IBM Research has released llm-d 1.0, a tool aimed at improving open-source AI inference for production use.
NVIDIA Unveils New Local AI Hardware at GTC Berlin
NVIDIA is introducing RTX Spark, DGX Spark, and DGX Station at GTC Berlin for local AI applications.
NVIDIA Launches RTX Spark Laptops With Local AI Capabilities
NVIDIA has launched RTX Spark laptops, offering local AI, gaming, and content creation through advanced hardware.
VLLM 0.32.0 Release Adds New Features and Optimizations
The vLLM 0.32.0 release includes new endpoints, tools, and GPU support for improved model performance.