HOW-TO: Deploy a Local LLM API Server with vLLM

Overview

vLLM is the de facto standard for high-performance open-source LLM serving. Built at UC Berkeley, it combines PagedAttention for efficient KV-cache management with continuous batching to keep GPUs maximally utilized under concurrent load.

What you'll learn:

• Installation via uv/pip and Docker
• Launching an OpenAI-compatible API server in one command
• Multi-GPU deployment with tensor parallelism
• Quantization strategies (FP8, INT4, AWQ, GGUF) for VRAM-constrained setups
• Performance tuning: batching, prefix caching, speculative decoding
• Production hardening: health checks, monitoring, load balancing
• Real-world configurations for common hardware (single 24GB GPU, dual-GPU, cloud instances)

Why vLLM matters:

• 2-4x throughput over naive PyTorch serving (PagedAttention eliminates KV-cache fragmentation)
• OpenAI-compatible API — drop-in replacement for openai Python SDK
• 200+ model architectures supported out of the box (Llama, Qwen, Gemma, DeepSeek, Mixtral, multimodal)
• Production-scale: Used by xAI, LinkedIn, Cursor, Google Cloud, and thousands of startups
• Hardware flexibility: NVIDIA, AMD, Intel Gaudi, Apple Silicon, Google TPU

Prerequisites:

• NVIDIA GPU with CUDA 12.1+ (or AMD ROCm, Intel ARC, Apple Silicon)
• Minimum 8GB VRAM for small models; 24GB+ recommended for 7B+ models
• Docker + NVIDIA Container Toolkit (for containerized deployment)
• Python 3.10+ (for native installation)

---

Architecture Overview

vLLM sits between your application and the model weights, managing the entire inference pipeline:

Key Components

Component	Role	Impact
PagedAttention	Manages KV-cache in fixed-size blocks (like OS virtual memory)	Eliminates memory fragmentation; 2-4x throughput gain
Continuous Batching	Dynamically adds/removes requests from active batch	GPU utilization 85-92% under concurrent load
Chunked Prefill	Splits long prompts across multiple iterations	Prevents latency spikes on long-context requests
Prefix Caching	Reuses KV-cache for shared prompt prefixes	10-50x speedup for repeated system prompts
Speculative Decoding	Uses a small draft model to propose tokens	1.5-2x throughput with minimal accuracy loss

---

Step 1: Installation

Option A: Native Installation (Recommended for Development)

Using uv (fastest, recommended):

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a virtual environment
uv venv vllm-env
source vllm-env/bin/activate

# Install vLLM with CUDA support
uv pip install vllm

# Verify installation
python -c "import vllm; print(vllm.__version__)"

Using pip:

pip install vllm

Option B: Docker (Recommended for Production)

vLLM provides official Docker images with pre-built CUDA kernels:

# Pull the latest stable image
docker pull vllm/vllm-openai:latest

# Or pin to a specific version (recommended for production)
docker pull vllm/vllm-openai:release-0.8.2

NVIDIA Container Toolkit setup (required for GPU access in Docker):

# Ubuntu/Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.key
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.key] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU access in Docker
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

---

Step 2: Quick Start — Single GPU

Launch a Server (Native)

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9

Launch a Server (Docker)

docker run --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    vllm/vllm-openai:latest \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9

Test the API

# List available models
curl http://localhost:8000/v1/models | python -m json.tool

# Chat completion
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is PagedAttention?"}
        ],
        "temperature": 0.7,
        "max_tokens": 256
    }' | python -m json.tool

# Streaming response
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [
            {"role": "user", "content": "Explain continuous batching in 3 sentences."}
        ],
        "stream": true
    }'

Python SDK Integration

from openai import OpenAI

# Point to your local vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # vLLM doesn't require auth by default
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a Python function to compute fibonacci numbers."}
    ],
    temperature=0.7,
    max_tokens=512
)

print(response.choices[0].message.content)

---

Step 3: Multi-GPU Deployment

Tensor Parallelism (Single Node, Multiple GPUs)

For models too large for one GPU (e.g., 70B+ models), split the model across GPUs:

# 70B model across 4 GPUs
docker run --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    vllm/vllm-openai:latest \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

GPU memory requirements by model size:

Model	Precision	1 GPU	2 GPUs	4 GPUs	8 GPUs
8B	FP16	✅ 16GB	—	—	—
8B	INT4	✅ 6GB	—	—	—
70B	FP16	❌	❌	✅ 4×24GB	✅
70B	INT4	❌	✅ 2×24GB	✅	✅
405B (MoE)	FP16	❌	❌	❌	✅ 8×80GB
405B (MoE)	FP8	❌	❌	✅ 4×80GB	✅

Data Parallelism (Multiple Replicas)

For handling more concurrent requests, run multiple replicas behind a load balancer:

# Replica 1
docker run --gpus all -p 8001:8000 \
    vllm/vllm-openai:latest --model meta-llama/Llama-3.1-8B-Instruct --port 8000

# Replica 2
docker run --gpus all -p 8002:8000 \
    vllm/vllm-openai:latest --model meta-llama/Llama-3.1-8B-Instruct --port 8000

# Nginx reverse proxy (nginx.conf snippet)
# upstream vllm_backend {
#     server localhost:8001;
#     server localhost:8002;
# }
# server {
#     listen 8000;
#     location / {
#         proxy_pass http://vllm_backend;
#         proxy_set_header Host $host;
#         proxy_buffering off;
#         proxy_cache off;
#     }
# }

---

Step 4: Quantization for VRAM-Constrained Setups

Quantization reduces model size and memory usage with minimal quality loss:

# AWQ 4-bit (best quality/size trade-off)
vllm serve meta-llama/Llama-3.1-8B-Instruct-AWQ \
    --quantization awq \
    --gpu-memory-utilization 0.9

# GPTQ 4-bit
vllm serve TheBloke/Llama-2-7B-Chat-GPTQ \
    --quantization gptq \
    --gpu-memory-utilization 0.9

# FP8 (NVIDIA H100/A100 with native FP8 support)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --quantization fp8 \
    --quantization-param-path /path/to/fp8_config.json

# GGUF (community quantized models)
vllm serve TheBloke/Llama-2-7B-Chat-GGUF \
    --quantization gguf \
    --gpu-memory-utilization 0.9

# INT4 via compressed-tensors
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --quantization compressed-tensors \
    --quantization-param-path /path/to/config.json

Quantization comparison:

Method	Precision	Size Reduction	Quality Loss	Best For
AWQ	4-bit	~75%	Minimal	General purpose
GPTQ	4-bit	~75%	Minimal	Legacy models
FP8	8-bit	~50%	Negligible	H100/A100 hardware
GGUF	2-6 bit	50-85%	Variable	Edge deployment
Compressed-Tensors	4-bit	~75%	Minimal	Production flexibility

---

Step 5: Performance Tuning

Key Parameters

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.95 \
    --enable-prefix-caching \
    --swap-space 4 \
    --max-logprobs 20 \
    --disable-log-requests

Parameter	Default	Recommended	Effect
--max-model-len	2048	4096-8192	Max context window; higher = more VRAM
--max-num-seqs	256	128-512	Max concurrent requests
--max-num-batched-tokens	Auto	16384-32768	Max tokens per batch iteration
--gpu-memory-utilization	0.9	0.85-0.95	Fraction of VRAM for KV cache
--enable-prefix-caching	Off	On	Reuse KV-cache for shared prefixes
--swap-space	4	2-8	CPU swap space (GB) for paged-out sequences
--enforce-eager	Off	On (debug)	Disable CUDA graphs for debugging

Speculative Decoding

Speed up generation by using a small draft model to propose tokens:

# N-gram speculative decoding (no extra model needed)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-model ngram \
    --num-speculative-tokens 5

# EAGLE draft model (better quality)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-model meta-llama/Llama-3.1-8B-Instruct-EAGLE \
    --num-speculative-tokens 5

Chunked Prefill for Long Contexts

Prevents latency spikes when processing long prompts:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --max-model-len 32768 \
    --max-num-batched-tokens 8192 \
    --chunked-prefill-enable

---

Step 6: Production Hardening

Health Checks

# Readiness probe (returns 200 when model is loaded)
curl http://localhost:8000/health

# Detailed health info
curl http://localhost:8000/metrics | head -50

Docker Compose for Production

# docker-compose.yml
version: '3.8'

services:
  vllm:
    image: vllm/vllm-openai:release-0.8.2
    container_name: vllm-server
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    volumes:
      - huggingface-cache:/root/.cache/huggingface
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --host 0.0.0.0
      --port 8000
      --max-model-len 8192
      --gpu-memory-utilization 0.9
      --enable-prefix-caching
      --max-num-seqs 256
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"

volumes:
  huggingface-cache:

Authentication (Nginx Reverse Proxy)

# nginx.conf — add API key authentication
upstream vllm {
    server 127.0.0.1:8000;
}

server {
    listen 443 ssl http2;
    server_name api.your-domain.com;

    ssl_certificate /etc/letsencrypt/live/api.your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.your-domain.com/privkey.pem;

    location /v1/ {
        # Simple API key check
        if ($http_authorization != "Bearer YOUR_API_KEY") {
            return 401;
        }

        proxy_pass http://vllm;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
    }

    location /health {
        proxy_pass http://vllm;
        allow all;
    }
}

Monitoring with Prometheus

vLLM exposes Prometheus metrics at /metrics:

# Example metrics
curl http://localhost:8000/metrics | grep -E "vllm:|gpu_"

Key metrics to monitor:

• vllm:num_requests_running — active requests
• vllm:num_requests_queued — queued requests (high = need more capacity)
• vllm:gpu_cache_usage_perc — KV cache utilization
• vllm:time_per_output_token_seconds — generation latency
• vllm:time_to_first_token_seconds — TTFT (time to first token)

---

Step 7: Real-World Configurations

Configuration 1: Single RTX 4090 (24GB) — Development

# 8B model, INT4 quantized — fits comfortably
docker run --gpus all -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model meta-llama/Llama-3.1-8B-Instruct-AWQ \
    --quantization awq \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --enable-prefix-caching

Capacity: ~20-30 concurrent requests, 50-80 tok/s per request

Configuration 2: Dual RTX 4090 (48GB) — Small Team

# 70B model across 2 GPUs, INT4
docker run --gpus all -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model meta-llama/Llama-3.1-70B-Instruct-AWQ \
    --quantization awq \
    --tensor-parallel-size 2 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.85

Capacity: ~10-15 concurrent requests, 20-40 tok/s per request

Configuration 3: A100 80GB — Production

# 70B model, FP16 — full precision
docker run --gpus all -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 2 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --enable-prefix-caching \
    --max-num-seqs 512 \
    --max-num-batched-tokens 32768

Capacity: ~50-100 concurrent requests, 40-80 tok/s per request

Configuration 4: H100 80GB — High-Throughput Production

# 70B model, FP8 — maximum throughput
docker run --gpus all -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --quantization fp8 \
    --tensor-parallel-size 2 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.95 \
    --enable-prefix-caching \
    --max-num-seqs 1024 \
    --speculative-model ngram \
    --num-speculative-tokens 5

Capacity: ~200+ concurrent requests, 100+ tok/s per request

---

Troubleshooting

Common Issues

Problem	Cause	Fix
CUDA out of memory	Model too large for VRAM	Reduce --gpu-memory-utilization, use quantization, reduce --max-model-len
Failed to load model	Missing Hugging Face token	Set HUGGING_FACE_HUB_TOKEN env var for gated models
NCCL error (multi-GPU)	GPU interconnect issue	Ensure NVLink/PCIe is functional; try --enforce-eager to debug
Slow first request	Model loading + warmup	Expected; subsequent requests use cached KV blocks
Torch not compiled with CUDA	Wrong PyTorch build	Use official vLLM Docker image or install torch with CUDA
High TTFT on long prompts	Prefill bottleneck	Enable --chunked-prefill-enable

Debug Mode

# Run with verbose logging
vllm serve ... --log-level debug

# Disable CUDA graphs for debugging (slower but easier to debug)
vllm serve ... --enforce-eager

# Profile memory usage
vllm serve ... --disable-log-requests --max-log-prob-len 0

---

Integration with Prior Work

This guide builds on our existing research:

• Vllm Vs Sglang Llm Serving Comparison 2026 05 07 — When to choose vLLM vs SGLang
• Gguf Inference Macos M3 Lmstudio Ollama 2026 04 16 — Local inference on consumer hardware
• Howto Multi Model Routing Layer — Route requests to different vLLM instances
• Inference Optimization Quantization Sparsity Speculative Decoding 2026 05 12 — Deep dive on the optimization techniques vLLM uses

---

References

• vLLM Documentation
• vLLM GitHub
• PagedAttention Paper (SOSP 2023)
• vLLM Blog
• vLLM Forum
• NVIDIA Container Toolkit

---

Next Steps

After deploying your vLLM server, consider:

1. Build a routing layer — Howto Multi Model Routing Layer to route requests to the optimal model
2. Set up monitoring — Prometheus + Grafana dashboards for latency and throughput
3. Add authentication — API keys, rate limiting, and usage tracking
4. Explore disaggregated serving — Separate prefill and decode for ultra-long contexts
5. Try multi-LoRA — Serve multiple fine-tuned adapters on one base model