GGUF Model Inference on macOS M3 Pro: Under the Hood with LM Studio, Ollama, and OpenAI-Compatible APIs

Executive Summary

Running large language models locally on Apple Silicon (M3 Pro) has become practical thanks to GGUF quantization and native support in tools like LM Studio and Ollama. This article explores the complete end-to-end execution pipeline—from tokenization through inference to GPU acceleration—when deploying a 35B parameter model like Qwen3.5-35B-A3B on consumer-grade MacBook Pro hardware.

Key Technical Stack:

• Model Format: GGUF (GPT-Generated Unified Format) quantized weights
• Hardware: MacBook Pro M3 Pro (12-core CPU: 6 performance + 6 efficiency cores; 18-core GPU; 18GB unified memory base)
• Inference Engines: LM Studio (GUI) + llama.cpp-compatible backends with Metal support
• API Layer: OpenAI-compatible REST endpoints
• GPU Backend: Metal Performance Shaders (MPS) / Metal Performance Shaders Graph (MPSGraph) via llama.cpp

The pipeline enables real-time inference (~10-50 tokens/second depending on quantization) on consumer hardware through careful memory management, quantization techniques, and OS-level optimizations.

---

I. The Full Inference Pipeline: From API Request to Token

Stage 1: API Request Reception

When you send a request to the OpenAI-compatible API (e.g., POST http://localhost:8000/v1/chat/completions), the LM Studio or Ollama HTTP server receives the JSON payload containing:

• messages: Conversation history
• model: Model identifier (e.g., "qwen3.5-35b-a3b")
• temperature, top_p: Sampling parameters
• max_tokens: Output length limit

Stage 2: Tokenization

What happens: The prompt is converted from human-readable text into integer token IDs using the model's vocabulary and BPE (Byte-Pair Encoding) tokenizer.

For Qwen3.5-35B-A3B specifically:

• Vocabulary size: 248,320 tokens (padded; Chinese + English + multilingual coverage)
• Tokenization is lossless: tokens can be perfectly reconstructed to original text
• Typical compression: 1 token ≈ 4 characters for English, 0.5-1 character for Chinese

Example:

Input: "What is machine learning?"
Tokenized: [1516, 318, 6438, 5555, 30]  # (simplified)
Token count: 5 tokens from 26 characters

---

II. GGUF Format & Quantization on M3 Pro

What is GGUF?

GGUF (GPT-Generated Unified Format) is a binary format optimized for inference. Unlike full-precision models (FP32/BF16), GGUF uses quantization to reduce model size while maintaining reasonable quality.

Quantization Levels for Qwen3.5-35B-A3B (Sparse MoE):

Important: Qwen3.5-35B-A3B uses Mixture-of-Experts (MoE) architecture with 256 total experts, 8 routed + 1 shared active per token, and Gated Delta Networks hybrid design. Only ~3B parameters activate per token (hence "A3B" suffix). Performance characteristics differ from dense models.

Format	Bit Depth	Size (35B sparse)	VRAM Needed (M3 Pro)	Quality	Speed
Full	FP32	~140 GB	❌ Not feasible	Perfect	~0.5 tok/s
Half	FP16	~70 GB	❌ Not feasible	Excellent	~2 tok/s
GPTQ	INT4	~8-10 GB	⚠️ Marginal (18GB base)	Good	~5-10 tok/s
GGUF-Q8	INT8	~9-12 GB	⚠️ Marginal	Excellent	~8-12 tok/s
GGUF-Q5	5-bit	~6-8 GB	✅ Comfortable	Good	~15-25 tok/s
GGUF-Q4	4-bit	~4-6 GB	✅ Excellent	Acceptable	~25-50 tok/s
GGUF-Q3	3-bit	~3-4 GB	✅ Excellent	Degraded	~40-80 tok/s

For M3 Pro (18GB unified memory, 150GB/s bandwidth):

• Q4 or Q5 recommended: Leaves 10-12GB for OS, runtime, and KV cache
• Q8 feasible: But limits context length and batch processing
• Anything larger: Requires aggressive paging, slow due to unified memory bandwidth

GGUF Loading Process

Key Optimization: Memory-Mapping (mmap)

• GGUF files are memory-mapped, not fully loaded into RAM
• Only the tensors needed for the current forward pass are loaded
• Dramatically reduces startup time (instant vs. 30+ seconds with FP32)

---

III. The Inference Loop: Token Generation Under the Hood

Step-by-Step Breakdown

Step 1: Token Embedding

Input tokens: [1, 5, 42, 103]
↓
Embedding matrix lookup (35B model has 152K × 4096 embedding)
↓
Embedding vectors: [[0.1, -0.3, ...], [0.2, 0.5, ...], ...]  # Shape: (4, 4096)

Step 2: Forward Pass Through Sparse MoE Architecture

• Input: Embedding vectors (sequence_length, hidden_dim) = (4, 4096)
• Process: Pass through Qwen3.5-35B-A3B's Mixture-of-Experts layers with Gated Delta Networks
  • Only 8 routed + 1 shared expert active per token (sparse routing)
  • Dramatically reduces computation vs. dense 35B model
  • Attention mechanism: O(n²) in sequence length (this is why KV cache matters)
  • Each active expert processes relevant subsets of the token
• Output: Hidden states (4, 4096)

Step 3: KV Cache Management (Critical for Speed)

Problem: Without caching, re-computing attention for every new token is expensive.

Solution: Cache the Key and Value matrices from previous tokens.

Memory Impact on M3 Pro:

• Cache per token: ~2 × (48 layers × 4096 × hidden_dim / heads)
• For Qwen3.5-35B: ~256KB per token
• At 1024 token context: ~256MB cache
• At max 4096 context: ~1GB cache (feasible with 18GB unified memory)

Step 4: Logits Computation

• Hidden state (4096,) → Output embedding weight matrix (4096, 152K)
• Result: Probability distribution over 152K vocabulary tokens
• Shape: (152K,) of floating-point logits

Step 5: Sampling with Temperature & Top-P

# Temperature: Controls randomness
# temperature = 0.0 → greedy (always pick max)
# temperature = 1.0 → normal distribution
# temperature = 2.0 → more random

probabilities = softmax(logits / temperature)

# Top-P (nucleus sampling): Only sample from top P% probability
# Filters out low-probability "tail" tokens
top_p = 0.9  # Use top 90% cumulative probability
filtered_probs = probs[cumsum(sorted_probs) <= top_p]
next_token = sample(filtered_probs)

Step 6: Select Next Token

• For greedy: argmax(probabilities)
• For sampling: sample(probabilities)
• Result: Single integer token ID

Step 7: Add to Output & Loop

• Append token to output sequence
• Check stopping condition: EOS token, max_tokens reached, or user stop signal
• If continuing, append new token to input and repeat

---

IV. Hardware Acceleration: Metal GPU on M3 Pro

Metal Performance Shaders (MPS)

M3 Pro Specs:

• CPU: 8-core ARM (2 performance + 6 efficiency cores)
• GPU: 12-core Metal GPU
• Unified Memory: 18GB shared (no PCIe transfer penalty)
• Memory Bandwidth: ~120 GB/s (vs. PCIe 4.0's 16 GB/s for dGPU)

Metal Performance Shaders (MPS) Execution

Key Optimizations for M3 Pro (150GB/s Bandwidth)

1. Operator Fusion: Multiple small operations combined into single MPS shader
2. Unified Memory: No data copies between CPU/GPU; direct shared access at 150GB/s
3. Asynchronous Execution: CPU queues GPU operations via MPS Graph while executing CPU-bound code
4. Memory Bandwidth Optimization: Careful tensor layout to maximize L2 cache hits
5. MoE-Aware Dispatch: Sparse expert routing reduces GPU load vs. dense model

---

V. Memory Management: Unified Memory Architecture

Unified Memory Management

With 150GB/s unified memory bandwidth and M3 Pro's architecture:

1. Efficient Access Pattern: No PCIe transfer penalty (unlike discrete GPUs)
2. Automatic Paging: macOS kernel optimizes data movement based on access patterns
3. Performance: Direct unified memory access far faster than discrete GPU VRAM over PCIe (16GB/s)

For Qwen3.5-35B-A3B (Q4 sparse MoE):

• Model weights (6GB total, ~0.5GB active per token): Sparse access pattern; efficient prefetching via MoE routing
• KV cache (1-2GB): Hot data; stays in fast memory during inference
• Activation tensors (1-2GB): Created and destroyed per forward pass; minimal overhead due to sparsity
• Expert buffers: Only 8 routed + 1 shared expert in flight, reducing memory pressure

---

VI. The OpenAI-Compatible API Layer

Request/Response Example

Request:

{
  "model": "qwen3.5-35b-a3b",
  "messages": [
    {"role": "user", "content": "What is machine learning?"}
  ],
  "temperature": 0.7,
  "top_p": 0.9,
  "max_tokens": 256,
  "stream": true
}

Backend: Served via llama.cpp with OpenAI-compatible API (LM Studio or direct llama.cpp server)

Response (Server-Sent Events):

data: {"choices":[{"delta":{"content":"Machine"},"finish_reason":null,"index":0}]}

data: {"choices":[{"delta":{"content":" learning"},"finish_reason":null,"index":0}]}

data: {"choices":[{"delta":{"content":" is"},"finish_reason":null,"index":0}]}

...

data: {"choices":[{"delta":{"content":"."},"finish_reason":"stop","index":0}]}

Note: Ollama currently does not support Qwen3.5 GGUF due to separate mmproj vision files. Use llama.cpp-compatible backends directly (LM Studio or llama.cpp CLI).

---

VII. Performance Characteristics on M3 Pro

Typical Throughput (Qwen3.5-35B-A3B on M3 Pro)

Quantization	Context Length	Tokens/Second	Latency (First Token)	Memory Used
Q4	1024	20-35	100-200ms	6-7 GB
Q4	4096	15-25	200-400ms	7-9 GB
Q5	1024	12-20	150-250ms	7-8 GB
Q5	4096	10-15	250-500ms	8-10 GB
Q8	1024	8-12	200-300ms	9-11 GB

MoE Sparsity Advantage: Per-token active parameters (~3B for A3B) reduce memory footprint vs. equivalent dense 35B model. Routing logic adds minimal overhead.

Factors Affecting Speed:

• Expert routing overhead: MoE gating adds ~5-10% compute vs. dense equivalent
• Batch size: 1 token at a time (streaming) vs. prefill batch (multi-token input)
• Temperature/Sampling: Higher randomness = slightly more compute
• Context length: Attention is O(n²); longer context = slower
• Background activity: Other processes reduce GPU availability

Energy Efficiency

M3 Pro's efficiency cores enable:

• Idle power draw: <5W when not computing
• Active inference power: 15-25W (vs. 150-300W for discrete GPU servers)
• All-day local inference: Feasible with battery management

---

VIII. Practical Example: Full Request Flow

Request

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-35b-a3b",
    "messages": [{"role": "user", "content": "Explain photosynthesis briefly."}],
    "temperature": 0.7,
    "max_tokens": 100,
    "stream": true
  }'

What Happens on M3 Pro

1. Tokenization (CPU): "Explain photosynthesis briefly." → [2397, 4629, 20882, 8234, 13] (5 tokens)

2. Embedding (GPU): Token IDs → 5 × 4096 embedding vectors

3. Forward Pass (GPU): Through 48 transformer layers:

   • Each layer: ~20M FLOPs for seq_len=5, hidden=4096
   • Total: ~1B FLOPs per forward pass
   • With Metal optimization: ~500ms first token, ~50ms per subsequent token

4. Sampling (CPU): Generate 100 tokens with temperature 0.7

5. Streaming Response:

   Photosynthesis is the process by which plants, algae, and some bacteria convert sunlight into chemical energy...
   

6. Total Time: ~5-6 seconds wall-clock (100 tokens at 20 tok/sec)

---

IX. Conclusion: Local LLM Inference on Consumer Hardware

Why This Matters

1. Privacy: No data leaves your device
2. Cost: One-time hardware vs. per-token API pricing
3. Control: Local model serving, customization, fine-tuning
4. Latency: 50-100ms token latency beats cloud API round-trip (500ms+)

Limitations

1. Hardware constraints: 18GB memory limits to ~35B parameter models
2. Speed tradeoff: Slower than A100 clusters but acceptable for many applications
3. Quantization cost: Q4/Q5 introduces ~5-10% quality degradation vs. FP32

Apple Silicon Roadmap (2026+)

• M5 Pro and beyond: Continued memory and compute increases enabling larger models
• Speculative decoding: Faster generation through multi-token prediction
• Sparse MoE optimizations: Hardware-accelerated expert routing for efficient sparse models
• Adaptive quantization: Dynamic precision based on layer importance
• Multi-device support: Inference distribution across multiple Apple Silicon devices

---

References & Sources

• GGUF Format: Specification in llama.cpp repository
• llama.cpp: High-performance LLM inference engine with Metal support (https://github.com/ggerganov/llama.cpp)
• LM Studio: GUI wrapper for local LLM inference (uses llama.cpp backend)
• Metal Performance Shaders (MPS): Apple's GPU programming framework
• Metal Performance Shaders Graph (MPSGraph): Modern computation graph framework (replaces deprecated MLCompute)
• Qwen3.5-35B-A3B: Model card on Hugging Face — Sparse MoE architecture, 256 experts (8 routed + 1 shared active), Gated Delta Networks, 248K vocabulary
• Apple Silicon specifications: M3 Pro: 12-core CPU (6P+6E), 18-core GPU, 150GB/s unified memory bandwidth

---

Published: April 16, 2026
Classification: Technical Deep-Dive · Hardware Architecture
Status: Complete ✓

This article documents the complete inference pipeline for running quantized 35B parameter models on consumer-grade Apple Silicon, revealing the sophisticated interplay between quantization, GPU acceleration, memory management, and API compatibility that enables practical local LLM serving in 2026.