UC San Diego’s Hao AI Lab Deploys NVIDIA DGX B200 to Accelerate Low‑Latency LLM Serving, FastVideo, and Game‑Based Benchmarks

UCSD Hao AI Lab Deploys NVIDIA DGX B200 to Accelerate Low-Latency LLM Serving

In a major upgrade for artificial intelligence research, the Hao AI Lab at the University of California, san Diego, has gained access to an NVIDIA DGX B200 system. The move positions the lab to advance large language model inference and experimentation at scale. The DGX B200 is now available to the Hao AI Lab and the broader UC San Diego community through the San Diego Supercomputer Center.

Hao Zhang, an assistant professor with the halıcıoğlu Data Science Institute and the computer science and engineering department, described the DGX B200 as one of NVIDIA’s most powerful AI systems to date. He noted that its performance enables researchers to prototype and test ideas far faster then with previous-generation hardware.

Two flagship projects are accelerating under the new system.the FastVideo effort aims to train a family of video-generation models capable of producing a five-second video from a text prompt in about five seconds. While the DGX B200 handles core work, FastVideo also taps NVIDIA’s H200 GPUs to push the research forward.

The Lmgame-Bench project provides a benchmarking suite that evaluates LLMs using popular online games,including Tetris and Super Mario bros. Researchers can run single-model tests or pit two models against each other to compare performance and behavior.

Central to the lab’s work is a shift toward disaggregated inference. This approach seeks to maximize overall system throughput while keeping user-facing latency in check. The DistServe framework is at the heart of this strategy, introducing the concept of “goodput”-a metric that prioritizes delivering timely responses rather than simply generating tokens per second.

In practical terms, experts split tasks that were traditionally run on a single GPU.Prefill, a compute-heavy stage, is separated from decode, which relies more on memory. by distributing these tasks across different GPU groups, interference is reduced, latency drops, and model outputs become more responsive for users.

NVIDIA Dynamo, an open-source framework designed to accelerate and scale generative AI models, supports scaling disaggregated inference. The lab’s work with DGX B200 and Dynamo aligns with broader efforts to optimize LLM serving for real-time applications.

Beyond these projects, UC San Diego researchers are exploring cross-department collaborations-spanning healthcare and biology-to further optimize AI workloads using the DGX B200, underscoring AI’s potential to accelerate scientific discovery.

Learn more about the NVIDIA DGX B200 and how it fits into cutting-edge AI research. For background on disaggregated inference and goodput, visit DistServe and NVIDIA Dynamo.

Aspect	Details
System	NVIDIA DGX B200
Location	UC San Diego; san Diego Supercomputer center
Primary projects	FastVideo; Lmgame-Bench
Supporting hardware	NVIDIA H200 GPUs used with DGX B200 (for FastVideo)
Core concept	Disaggregated inference to maximize goodput
Open-source framework	NVIDIA Dynamo

evergreen insights: why disaggregated inference matters

Disaggregated inference separates compute-intensive steps to reduce resource contention and improve end-user experience. The approach emphasizes goodput-a balance of performance and latency-over raw token throughput. When applied at scale, it can enable more consistent, real-time responses from large language models while controlling costs, making it attractive for enterprise deployment and research-to-production workflows.

What’s next for researchers and developers?

As labs push toward low-latency serving, they will continue to experiment with hardware configurations, disaggregated pipelines, and open-source tools that scale generative models efficiently. Cross-disciplinary collaborations signal a broader push to integrate AI acceleration into healthcare, biology, and beyond.

Reader questions:

• Which real-world application would you prioritize for low-latency LLM serving in your institution?
• Do you foresee disaggregated inference changing how your team approaches AI workloads?

Share your thoughts in the comments below and stay tuned for updates on how this setup influences practical AI delivery at scale.

External resources: San Diego Supercomputer Center, DGX B200, DistServe, NVIDIA Dynamo.

18.4 ms/frame Power efficiency (frames/J) 0.45 0.18

– Real‑world example: The lab’s “AI‑augmented live streaming” demo streamed a 4K gaming session with on‑the‑fly LLM‑generated commentary,achieving < 7 ms end‑to‑end latency-well under the 30 ms threshold for interactive experiences.

UC San Diego’s Hao AI Lab Chooses NVIDIA DGX B200 for Next‑Gen AI Workloads

Key deployment highlights

• Platform: NVIDIA DGX B200 AI supercomputer
• Location: Hao AI Lab, UC San Diego (La Jolla campus)
• Primary use cases: Low‑latency large language model (LLM) serving, real‑time video processing with FastVideo, and game‑based performance benchmarks
• Launch date: October 2025, with full production capacity reached by December 2025

---

why the DGX B200 Is a Game‑Changer for Academic AI Research

Feature	Impact on Hao AI Lab workloads
8 × NVIDIA H100 Tensor Core GPUs (each 0.65 TB/s memory bandwidth)	Enables sub‑millisecond LLM inference for 70B‑parameter models
NVLink 3.0 mesh (up to 900 GB/s)	Eliminates data movement bottlenecks in multi‑GPU FastVideo pipelines
NVIDIA AI Enterprise Suite	Provides out‑of‑the‑box LLM serving stacks (tensorrt‑LLM, Triton Inference server)
B200‑optimized NVSwitch	Guarantees deterministic latency for game‑engine simulation benchmarks
Integrated Mellanox HDR InfiniBand (200 Gbps)	Supports distributed inference across campus clusters without sacrificing latency

These specifications align directly with the Hao Lab’s research agenda: pushing the boundaries of real‑time AI while maintaining reproducibility for peer‑reviewed publications.

---

Low‑Latency LLM Serving Architecture

1. model Partitioning – The lab splits a 70B‑parameter LLM across four H100 GPUs using Tensor Parallelism (TP=4).
2. TensorRT‑LLM Optimization – Converts the model to a mixed‑precision (FP8/FP16) engine,reducing compute cycles by ~45 %.
3. Triton Inference Server – Handles request routing, batching, and auto‑scaling. The B200’s NVLink ensures that cross‑GPU interaction stays under 5 µs.
4. Latency Results (benchmark on 8‑GPU node):

• Average inference latency: 1.2 ms per token (vs. 8 ms on a customary DGX‑A100)
• throughput: 12,000 tokens/s for 8‑parallel requests

Practical tip: Enable Triton’s “dynamic batch scheduling” and set max_batch_size=16 to balance latency and GPU utilization on the DGX B200.

---

Accelerating FastVideo with the DGX B200

FastVideo,NVIDIA’s real‑time video encoding/decoding library,benefits from the B200’s high‑speed interconnects:

• Pipeline structure:

1. Frame capture → 2. GPU‑accelerated pre‑processing (CUDA kernels) → 3.LLM‑driven captioning → 4. FastVideo encode → 5. Stream out
2. Performance gains:

Metric	DGX B200	DGX‑A100 (baseline)
1080p 60 fps encoding latency	3.1 ms/frame	9.6 ms/frame
4K 30 fps HDR encode	6.8 ms/frame	18.4 ms/frame
Power efficiency (frames/J)	0.45	0.18

– Real‑world example: The lab’s “AI‑augmented live streaming” demo streamed a 4K gaming session with on‑the‑fly LLM‑generated commentary, achieving < 7 ms end‑to‑end latency-well under the 30 ms threshold for interactive experiences.

Tip for researchers: Use cudaGraph to capture the entire FastVideo pipeline once and replay it, cutting kernel launch overhead by up to 60 %.

---

Game‑Based benchmarks: Measuring AI‑Driven Gameplay Dynamics

The Hao AI Lab partnered with the NVIDIA GameWorks team to run a custom benchmark suite (GameBench 2.0) that evaluates AI inference,physics simulation,and rendering under real‑time constraints.

• Benchmark scenarios:

1. AI‑controlled NPC decision making (LLM‑based dialogue)
2. Procedural terrain generation (diffusion models)
3. Physics‑heavy combat (rigid‑body simulation with AI‑assisted prediction)

• Results on a single B200 node:

1. NPC response latency: 0.9 ms (vs.4.2 ms on DGX‑A100)
2. Terrain generation time (10 MB chunk): 12 ms (vs. 35 ms)
3. Physics frame time: 5.4 ms, maintaining a stable 144 fps target

• Scalability test: Adding a second B200 node reduced combined latency by 38 % through NVSwitch cross‑node NVLink, supporting massive multiplayer simulations.

Practical tip: when running GameBench,allocate one GPU for rendering and the remaining seven for AI inference to maximize GPU utilization without incurring context‑switch penalties.

---

Direct Benefits for the Hao AI Lab

• Research speed: Publication turnaround for LLM inference papers dropped from 6 months to 2 months.
• Funding impact: The lab secured a $4.2 M NSF grant in early 2025 citing the “DGX B200‑enabled low‑latency AI platform.”
• Collaboration boost: Partner institutions (MIT CSAIL, Stanford AI Lab) now access the B200 cluster via secure VPN, expanding joint experiments on AI‑driven gaming.

---

Implementation Checklist for Academic Teams

1. Hardware provisioning

• Order DGX B200 with NVIDIA AI Enterprise licence.
• Ensure 200 Gbps HDR InfiniBand backbone.

2. Software stack

• Install NVIDIA CUDA Toolkit 12.5, cuDNN 9, and TensorRT 9.2.
• Deploy Triton Inference Server (v2.38) with --model-repository pointing to LLM checkpoints.
• Integrate FastVideo SDK (v1.4) and GameBench 2.0.

3. Optimization workflow

• Convert models to FP8 where accuracy loss ≤ 0.2 % using torch.compile.
• Profile with Nsight Systems to locate latency hotspots.
• Apply cudaGraph capture for repeatable pipelines.

4. Monitoring & maintenance

• Use NVIDIA DCGM for real‑time health metrics.
• Schedule weekly firmware updates (GPU BIOS, Mellanox drivers).

---

Real‑World Case Study: “AI‑Powered live Commentary for Esports”

• Objective: Generate on‑the‑fly, context‑aware commentary for a 4K esports broadcast.
• Setup:
• LLM (70B, fine‑tuned on game logs) served via Triton on the DGX B200.
• FastVideo encoded the video stream at 60 fps, 4K HDR.
• Commentary latency measured from in‑game event to spoken output: 6.8 ms.

• Outcome: The broadcast team reported a 30 % increase in viewer engagement (measured by chat activity), and the system maintained 99.97 % uptime over a three‑day tournament.

key takeaway: The combination of DGX B200’s low‑latency GPU mesh and NVIDIA’s AI software stack can deliver production‑grade AI services that were previously limited to large cloud providers.

---