The Private AI Lab podcast

Testing Nemotron's 3.5 Lightning's 1M-Token Context on the DGX Spark

0:00
11:16
Spol 15 sekunder tilbage
Spol 15 sekunder frem

NVIDIA just shipped a model with four times the context window of what's been running my home agent for months. So I swapped it in, broke my own setup within the first hour, and then threw 700,000 words at it to see if it would lie to me.

No guest this episode, just me and my home lab. Jarvis, my home agent, has been running on Qwen3.6-35B-A3B since the start, served through vLLM on the DGX Spark's 128GB of unified memory. Then NVIDIA released Nemotron 3.5 Lightning 30B-A3B in NVFP4, and one number stopped me mid-scroll: a validated 1 million token context window, four times what Qwen gives you at 262,000.I don't trust a spec sheet claim until I've broken it myself, so I swapped the model into Jarvis, found a memory reservation bug in NVIDIA's own deployment recipe within the first hour, fixed it, and then buried a fabricated secret code word 90% of the way through a 700,000 word document to see if the model could actually recall it. I also dug into DSpark, the speculative decoding checkpoint built specifically for this hardware, and measured what it actually does for throughput.

This episode is the honest version: what's proven on my own hardware, what's still sitting on NVIDIA's spec sheet, and what I still don't know about whether Nemotron makes Jarvis a better agent day to day.

Key topics covered:

  • Why Jarvis's setup (Qwen3.6-35B-A3B on vLLM) needed an indirection layer before any of this mattered
  • NVIDIA Nemotron 3.5 Lightning 30B-A3B-NVFP4's validated 1 million token context window, tested against a 700,000-word document
  • The memory reservation bug in NVIDIA's DGX Spark deployment recipe, and the one-line fix
  • DSpark speculative decoding: how it works, and the acceptance rate that made it worth using
  • Nemotron's hybrid Mamba-2, attention, and mixture-of-experts architecture, and why it matters for running 1M tokens on a single box
  • What's proven on real hardware versus what's still just NVIDIA's own benchmark numbers
  • Why Nemotron isn't confirmed as the better agent brain for Jarvis yet


Chapters

00:00 Introduction — what a 1M-token context window actually changes

00:37 Welcome back to The Private AI Lab (no guest this episode)

01:19 Recap: Jarvis running on Qwen3.6-35B-A3B via vLLM on the DGX Spark

01:58 NVIDIA Nemotron 3.5 Lightning lands with a validated 1M-token window

02:49 Deployment: why the Jarvis-LLM alias made the model swap painless

03:28 The memory reservation bug NVIDIA's own recipe doesn't warn about

04:53 The test: a 700,000-word document with one hidden code word

05:51 DSpark speculative decoding: free speed, fully verified

06:47 The reasoning quirk that quietly eats your token budget

07:31 Nemotron vs Qwen: what's proven versus what's still a spec sheet claim

08:01 The hybrid architecture behind why 1M tokens fits on a DGX Spark

09:24 NVIDIA's own SWE-bench Verified numbers, and their limits

09:56 Closing thoughts, and what's coming in the next episode


Links mentioned in this episode:

Flere episoder fra "The Private AI Lab"