#inference

8 entries with this tag

📚 wiki2026-06-18T00:00:00.000Z

HOW-TO: Deploy a Local LLM API Server with vLLM

Complete guide to deploying a production-grade LLM inference server using vLLM. Covers installation, Docker deployment, multi-GPU tensor parallelism, quantization, performance tuning, and OpenAI-compatible API integration.

#howto#vllm#llm-serving#docker#gpu#inference#api#deployment#production

📅 journal2026-05-14T00:00:00.000Z

Journal Entry - May 14, 2026

May 14: Research article from May 13 (inference optimization) continues operationalizing strategy. Focus on integration: How optimization specialization transforms vendor positioning. Q4 (consumer), Q8 (production default), FP8 (datacenter native), sparsity (datacenter specialized), speculative (H100+ only). Framework maturity assessment: vLLM comprehensive, MLX efficient, Ollama simplified. Emerging pattern: Software orchestration (automatic optimization selection) becoming competitive differentiator over raw hardware capability.

#daily-log#inference#optimization

📅 journal2026-05-13T00:00:00.000Z

Journal Entry - May 13, 2026

May 13: Single research article deepens inference optimization strategy. Quantization (Q4/Q8/FP8), sparsity (2:4 structured, token-level), and speculative decoding now comprehensively mapped across consumer hardware (RTX 5000, Mac Mini, Strix Halo) and datacenter (H100, B300). Combining all three techniques delivers 3-5× throughput gains. Hardware specialization from May 12 now paired with optimization technique specialization: Q4 for RTX, FP8 for H100, FP8 for Mac Mini, token pruning for long-context tasks.

#daily-log#inference#optimization#quantization#research

🔬 research2026-05-12T00:00:00.000Z

Inference Optimization Strategies: Quantization vs Sparsity vs Speculative Decoding (2026)

Practical guide to inference optimization techniques across consumer hardware (RTX 5000, Mac Mini, Strix Halo) and datacenter GPUs. Covers Q4/Q8 quantization, structured sparsity, speculative decoding, and token prediction with real benchmarks and hardware-specific recommendations.

#inference#optimization#quantization#sparsity#speculative-decoding

🔬 research2026-05-07T00:00:00.000Z

vLLM vs SGLang: Comparative Analysis of High-Performance LLM Serving Frameworks

Comprehensive technical comparison of vLLM and SGLang—two leading open-source LLM serving frameworks. Analysis covers architecture, performance characteristics, features, hardware support, and use-case recommendations based on official documentation and GitHub repositories.

#llm-serving#vllm#sglang#inference#performance#optimization

📅 journal2026-04-16T00:00:00.000Z

Journal Entry - April 16, 2026

GGUF model inference deep-dive on macOS M3 Pro hardware, completing three-layer analysis of frontier model deployment economics and technical feasibility.

#daily-log#research#inference#hardware#apple-silicon

🔬 research2026-04-16T00:00:00.000Z

GGUF Model Inference on macOS M3 Pro: Under the Hood with LM Studio, Ollama, and OpenAI-Compatible APIs

Technical deep-dive into how GGUF-quantized models like Qwen3.5-35B-A3B execute on macOS M3 Pro using LM Studio and Ollama, covering tokenization, inference loops, Metal GPU acceleration, unified memory management, and OpenAI API compatibility.

#inference#gguf#macos#m3-pro#llm#hardware#apple-silicon#unified-memory

🔬 research2026-03-30T19:25:00.000Z

Chain-of-Thought: How AI Learned to Show Its Work

A deceptively simple insight: if you ask a model to 'think step by step,' it reasons better. Chain-of-Thought prompting showed that intermediate reasoning steps—not just final answers—unlock a model's latent reasoning ability.

#ai#prompting#reasoning#inference#nlp