NVIDIA Nemotron 3.5 Lightning: 30B MoE-Mamba Hybrid for Agentic Execution

Executive Summary

NVIDIA released Nemotron 3.5 Lightning on August 11, 2026 — a 30B-parameter Mixture-of-Experts (MoE) model with only 3B active parameters per token, purpose-built for the high-volume execution layer of long-running AI agents. Unlike frontier reasoning models that excel at planning and orchestration, Lightning is designed for the repetitive work that dominates an agent's token budget: tool calls, result validation, sub-agent delegation, and routine code operations.

The model employs a novel hybrid architecture combining interleaved Mamba-2 state-space layers, MoE expert routing, and selective Transformer attention — a departure from the pure-attention designs of recent releases like Openai Gpt 5 6 Sol Terra Luna Ultra Multi Agent Cerebras Speed Frontier Coding Cyber Safety Stack 2026 08 21 and Qwen3 8 27b Dense Multimodal Hybrid Attention Local Deployment Apache 2026 08 20. It ships with three speculative decoding strategies (MTP, DFlash, DSpark), NVFP4 quantization for efficient inference, and support for up to 1M token context.

Licensed under OpenMDW-1.1, Nemotron 3.5 Lightning is fully open — weights, training data, and recipes — making it immediately available for commercial use, fine-tuning, and local deployment on everything from DGX Spark workstations to data-center GPU clusters. The release also introduces NeMo Switchyard, a model-routing library that dynamically assigns tasks across a pool of models, enabling "system of models" architectures where Lightning handles execution and frontier models handle planning.

1. Architecture: Mamba-2 + MoE + Attention Hybrid

Nemotron 3.5 Lightning's architecture represents a significant departure from the dense Transformer designs dominating the frontier landscape. The model combines three architectural paradigms:

• Mamba-2 State-Space Layers: Interleaved with MoE and attention layers, providing linear-time sequence modeling for long-context efficiency. Mamba's selective state-space mechanism processes sequences without the quadratic attention cost, enabling the 1M token context window.
• Mixture-of-Experts (MoE): 30B total parameters with only 3B active per token. A router directs each token to a subset of specialized expert networks, providing the capacity of a large dense model at the compute cost of a small one.
• Selective Attention Layers: Retained in strategic positions for tasks requiring global context awareness, such as multi-document aggregation and long-range retrieval.

The model also includes Multi-Token Prediction (MTP) heads — trained during a dedicated continued pre-training phase — which predict multiple future tokens simultaneously. These MTP heads serve dual purposes: providing richer training signals during fine-tuning and enabling speculative decoding at inference time.

Key architectural parameters:

Parameter	Value
Total parameters	30B
Active parameters per token	3B
Architecture type	MoE + Mamba-2 + Attention hybrid
Context length	Up to 1M tokens
Pre-training tokens	20T+
Training precision	NVFP4 (pre-training), BF16 (reference)
Inference precision	NVFP4, W4A16, BF16
Supported languages	English, Spanish, French, German, Italian, Japanese

2. Training Methodology: Five-Stage Pipeline

Nemotron 3.5 Lightning underwent a comprehensive five-stage training process from December 2025 through May 2026:

Stage 1: Pre-Training

Pre-trained on 20T+ tokens using an NVFP4 recipe with crawled and synthetic data spanning code, math, science, and general knowledge. Training used NVIDIA's Megatron-LM framework. Data cutoff: September 2025.

Stage 2: Multi-Token Prediction (MTP) Continuation

A dedicated continued pre-training phase trained the MTP heads to predict multiple future tokens, aligning them with the base model's distribution. A subsequent MTP-boosting phase further improved prediction accuracy.

Stage 3: Supervised Fine-Tuning (SFT)

Fine-tuned on synthetic data for code, math, science, tool calling, instruction following, and structured outputs. Special emphasis on long-range retrieval and multi-document aggregation capabilities.

Stage 4: Reinforcement Learning

Multi-environment RL using GRPO (Group Relative Policy Optimization) across math, code, science, instruction following, multi-step tool use, multi-turn conversations, and structured output environments. Utilized an asynchronous RL architecture that decouples training from inference and leverages MTP to accelerate rollout generation. Software: NeMo RL and NeMo Gym.

Stage 5: Post-Training Quantization (PTQ)

The NVFP4 checkpoint was produced via post-training quantization using NVIDIA Model Optimizer: Four-Over-Six NVFP4 (static MSE calibration variant) W4A16 on routed and shared experts, with FP8 per-tensor dynamic scales on Mamba projections and KV cache. Calibration used 1,000 samples at 32K token length from the Nemotron Ultra validation set.

3. Benchmark Performance

NVIDIA evaluated Nemotron 3.5 Lightning across a comprehensive suite covering general knowledge, reasoning, coding, agentic tasks, instruction following, and long-context retrieval. Results are measured under NVIDIA's consistent NeMo Gym / NeMo Evaluator SDK harness.

General Knowledge & Reasoning

Benchmark	Nemotron 3.5 Lightning (BF16)	Qwen 3.6 35B A3B	Gemma 4 26B A4B	Nemotron 3 Nano	Nemotron 3 Super
MMLU Pro	81.94	85.63	85.20	78.46	83.89
AA-Omniscience	17.50	19.47	22.17	20.15	26.68
GPQA Diamond (no tools)	75.44	83.40	79.61	74.05	78.60
HLE (text-only)	11.72	19.56	17.42	10.89	20.30
SciCode	32.60	35.33	40.28	30.08	35.11

Coding & Agentic Tasks

Benchmark	Nemotron 3.5 Lightning (BF16)	Qwen 3.6 35B A3B	Gemma 4 26B A4B	Nemotron 3 Nano	Nemotron 3 Super
SWE-bench Verified	51.56	70.12	57.40	34.08	63.08
SWE-bench Multilingual	39.33	63.40	43.40	14.07	49.80
Terminal-Bench 2.1	24.58	44.38	37.22	8.29	39.61
PinchBench	85.37	88.07	74.70	66.11	80.36
BrowseComp	36.97	48.74	26.30	13.74	22.77
τ³-bench (Banking)	9.28	10.52	14.02	7.01	12.37

Instruction Following & Long Context

Benchmark	Nemotron 3.5 Lightning (BF16)	Qwen 3.6 35B A3B	Gemma 4 26B A4B	Nemotron 3 Nano	Nemotron 3 Super
IFBench (loose)	71.88	63.71	77.25	72.17	71.92
AA-LCR (Long Context)	52.00	61.06	57.56	32.75	58.44

NVFP4 Quantization Impact

The NVFP4 checkpoint shows minimal degradation from BF16, with some benchmarks actually improving:

Benchmark	BF16	NVFP4	Delta
MMLU Pro	81.94	81.62	-0.32
GPQA Diamond	75.44	75.57	+0.13
SWE-bench Verified	51.56	52.80	+1.24
PinchBench	85.37	83.43	-1.94
IFBench (loose)	71.88	72.88	+1.00
τ³-bench (Banking)	9.28	9.48	+0.20

Key observation: On PinchBench, Nemotron 3.5 Lightning reaches 86% accuracy while completing 10,000 tasks 30% faster than Qwen 3.6 35B at similar accuracy — demonstrating that agent efficiency is about useful work completed per unit time, not just token generation speed.

4. Speculative Decoding: Three Strategies

Nemotron 3.5 Lightning ships with three speculative decoding approaches, each optimized for different deployment scenarios:

MTP (Multi-Token Prediction)

• Mechanism: The draft head is embedded within the target checkpoint — no separate model download required.
• Best for: Medium to high concurrency workloads.
• Trade-off: Optimal draft length decreases as concurrency increases.
• Implementation: Uses EAGLE algorithm with the model's own MTP heads.

DFlash

• Mechanism: A separate lightweight block-diffusion draft model that generates an entire draft block in one forward pass.
• Best for: Workloads where block-level speculation provides the best speedup.
• Block size: 6 on B200, 4 on H100/DGX Spark.
• Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash

DSpark

• Mechanism: A semi-autoregressive drafter that proposes a whole block of candidate tokens in a single forward pass from a parallel backbone.
• Best for: DGX Spark inference and low-concurrency data center deployments.
• Block size: 3 tokens.
• Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark

5. Deployment Options

Nemotron 3.5 Lightning supports deployment across a wide hardware spectrum, from consumer GPUs to data-center clusters:

Hardware Support Matrix

Hardware	Precision	Backend	Speculative Decoding	Validated Context
GB200 (Blackwell)	NVFP4 / FP4	vLLM + DSpark	DSpark (5 tokens)	1M
DGX Spark (GB10)	NVFP4 / W4A16	vLLM + Marlin	DSpark (3 tokens)	1M
RTX 5090	NVFP4	llama.cpp / Ollama	Built-in	VRAM-bound
H100 (single)	NVFP4 / W4A16	vLLM / SGLang	DSpark / None	256K-1M
H100 (8x, TP8+EP)	BF16	vLLM	None	1M
A100 (Ampere)	W4A16	vLLM	None	1M

Quick Start: DGX Spark (GB10)

export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark

vllm serve --model $MODEL_CKPT \
  --moe-backend marlin \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.85 \
  --speculative_config.num_speculative_tokens 3 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --reasoning-parser nemotron_v3 \
  --speculative_config.model $DSPARK_CKPT \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Quick Start: Single H100 (Max Throughput)

vllm serve --model $MODEL_CKPT \
  --max-num-seqs 256 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-backend flashinfer \
  --moe-backend humming \
  --linear-backend humming \
  --mamba-ssu-algorithm horizontal \
  --mamba-cache-mode align \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --reasoning-parser nemotron_v3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Local AI Deployment

For consumer hardware, Nemotron 3.5 Lightning is available via:

• llama.cpp: GGUF checkpoints at ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF
• Ollama: ollama run nemotron-3.5-lightning
• LM Studio: Direct model import
• Unsloth: Fine-tuning support

Reasoning Mode Control

The model supports configurable reasoning via chat template kwargs:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")

# Reasoning ON (default)
response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
    messages=[{"role": "user", "content": "Solve this step by step..."}],
    extra_body={"enable_thinking": True}
)

# Reasoning OFF (direct answers)
response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
    messages=[{"role": "user", "content": "What's 2+2?"}],
    extra_body={"enable_thinking": False}
)

6. NeMo Switchyard: Model Routing for Systems of Models

The Nemotron 3.5 Lightning release introduces NeMo Switchyard, a library for intelligent model routing that enables "system of models" architectures. Rather than sending every request to the most capable (and expensive) model, Switchyard dynamically routes each task to the best-suited model.

Routing Approaches

Tuning-free routers:

• LLM Classifier: Uses an LLM as a judge to classify requests and route to the appropriate model, maintaining session affinity across turns.
• Stage Router: Examines recent tool activity to determine how much model capability is needed — escalation for errors, efficient model for steady execution.
• Escalation Router: Starts with a lower-cost model and escalates to a frontier model when sustained difficulty is detected.

Tunable routers:

• Prefill Router: Extracts the LLM's residual stream during training to predict query complexity and maps it to accuracy labels for each candidate model, enabling learned cost-accuracy tradeoffs.

Real-World Results

• LangChain benchmark: Routing between Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router delivered 74% cost reduction compared with frontier-only baseline, sending just 7% of calls to the frontier model at ~6-point accuracy tradeoff.
• Cognition (Devin Desktop): Staged routing between Opus 5 and Kimi K2.7 achieved 50.6% on FrontierCode Main at $3.11 mean cost — within 2.8 percentage points of Opus 5 accuracy at ~28% lower cost.

7. Customization and Fine-Tuning

Nemotron 3.5 Lightning is designed for customization. The OpenMDW-1.1 license permits full commercial use, and NVIDIA provides comprehensive tooling:

• LoRA fine-tuning: Via NeMo Automodel and NeMo Megatron Bridge
• Full SFT: Using Megatron-LM with the published training recipes
• Reinforcement learning: Via NeMo RL and NeMo Gym
• Domain adaptation: Published cookbook includes Text2SQL, code review, and terminal automation examples

The release includes Nemotron-RL Agentic Terminal Pivot v1, an open agentic reinforcement learning dataset used to train coding agent capabilities.

Small model size (3B active parameters) means fine-tuning is faster, cheaper, and requires far more modest hardware than frontier models — making domain specialization practical for teams without massive compute budgets.

8. Positioning in the Nemotron Family

Nemotron 3.5 Lightning is the smallest member of the Nemotron 3 family, filling the execution layer below the larger models:

Model	Parameters	Active	Role	Best For
Nemotron 3 Ultra	350B+	~40B	Frontier reasoning	Complex planning, orchestration
Nemotron 3 Super	70B+	~7B	Mid-tier reasoning	Balanced planning and execution
Nemotron 3 Nano	15B+	~1.5B	Ultra-lightweight	Simple classification, routing
Nemotron 3.5 Lightning	30B	3B	Agentic execution	High-volume tool calls, code ops

This tiered approach enables the "system of models" paradigm: frontier models handle planning and complex reasoning, while Lightning handles the high-volume execution that dominates token budgets.

9. Comparison with Recent Releases

vs. Qwen 3.8 27B (Dense)

The recently released Qwen3 8 27b Dense Multimodal Hybrid Attention Local Deployment Apache 2026 08 20 Qwen 3.8 27B is a dense model with hybrid attention, offering strong multimodal capabilities under Apache 2.0. Key differences:

Aspect	Nemotron 3.5 Lightning	Qwen 3.8 27B
Architecture	MoE + Mamba-2 + Attention	Dense + Hybrid Attention
Active parameters	3B (of 30B total)	27B (all active)
Multimodal	Text-only	Multimodal
Speculative decoding	MTP + DFlash + DSpark	Standard
License	OpenMDW-1.1	Apache 2.0
Optimization focus	Agentic execution	General-purpose + local deployment
Context	1M tokens	Standard

Qwen 3.8 27B offers broader multimodal support and a more permissive license, while Nemotron 3.5 Lightning provides superior inference efficiency through MoE sparsity and specialized agentic training.

vs. GPT-5.6 (Frontier)

The Openai Gpt 5 6 Sol Terra Luna Ultra Multi Agent Cerebras Speed Frontier Coding Cyber Safety Stack 2026 08 21 GPT-5.6 represents the frontier of reasoning capability. Nemotron 3.5 Lightning is not a competitor but a complement — designed to handle the execution work that would be wasteful to send to a frontier model.

vs. GLM-5.3 (Coding Specialist)

The Zai Glm 5 3 Frontier Coding Emergent Cyber Capabilities 2436 Vulnerabilities Open Source Sota 2026 08 19 GLM-5.3 specializes in coding with emergent cyber capabilities. Nemotron 3.5 Lightning has strong coding benchmarks (51.56% SWE-bench Verified) but is optimized for broader agentic execution rather than deep coding specialization.

10. Key Takeaways

1. MoE-Mamba hybrid is production-ready: The combination of Mixture-of-Experts sparsity with Mamba-2 state-space layers delivers the capacity of a 30B model at the compute cost of a 3B model, with 1M context support.

2. Agentic execution is a distinct workload: Long-running agents spend most of their time on high-volume execution tasks. Purpose-building models for this layer — rather than over-provisioning with frontier models — delivers dramatic cost and latency improvements.

3. Speculative decoding is maturing: Three distinct strategies (MTP, DFlash, DSpark) with workload-specific optimization represent a significant step forward in inference efficiency.

4. Model routing is the future of agent architecture: NeMo Switchyard and the "system of models" paradigm enable dynamic task assignment, achieving 74% cost reduction in real-world benchmarks.

5. NVFP4 quantization is near-lossless: The NVFP4 checkpoint shows minimal degradation from BF16 (often improving on specific benchmarks), making it the recommended deployment format.

6. Open weights enable rapid customization: The OpenMDW-1.1 license, combined with published training recipes and datasets, makes domain adaptation practical for teams of all sizes.

11. Future Directions

Several areas warrant monitoring as the Nemotron ecosystem evolves:

• Mamba-2 adoption: If Nemotron 3.5 Lightning's hybrid architecture proves its value in production, expect more models to incorporate state-space layers for long-context efficiency.
• Speculative decoding standardization: The proliferation of MTP, DFlash, and DSpark suggests speculative decoding will become a standard feature of production models.
• Model routing maturity: NeMo Switchyard's early results (74% cost reduction) could make model routing a standard component of agent infrastructure.
• Local AI agentic workloads: With GGUF support and DGX Spark compatibility, Nemotron 3.5 Lightning could enable sophisticated local agent deployments on consumer hardware.
• Nemotron 4 timeline: Following the rapid evolution from Nemotron 3 to 3.5, the next iteration may further optimize the execution layer or introduce new architectural innovations.

References & Resources

• Official announcement: NVIDIA Developer Blog — Nemotron 3.5 Lightning
• BF16 model card: Hugging Face — NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
• NVFP4 model card: Hugging Face — NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
• GGUF checkpoints: Hugging Face — NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF
• NeMo Switchyard: GitHub — NVIDIA-NeMo/Switchyard
• Switchyard blog: NVIDIA Developer Blog — Route AI Agent Workloads
• Training recipes: GitHub — Nemotron 3.5 Lightning Cookbook
• Evaluation reproducibility: GitHub — NeMo Gym — Nemotron 3.5 Lightning Recipes
• RL dataset: Hugging Face — Nemotron-RL Agentic Terminal Pivot v1
• License: OpenMDW License Agreement v1.1
• Try it live: build.nvidia.com — Nemotron 3.5 Lightning
• OpenRouter: OpenRouter — Nemotron 3.5 Lightning (free)