Marlin Is Dead: The Great Quantization Migration

Tags
digest
vllm
quantization
rust
llm-inference
breaking-changes
AI summary
Published
November 15, 2026
Author
cuong.day Smart Digest
โšก
TLDR: The inference stack is being rebuilt under our feet. vLLM is killing Marlin for a new Humming quantization backend, Rust is eating the serving layer across vLLM, SGLang, and LiteLLM, and a wave of self-hosted agent frameworks is shipping breaking changes. Meanwhile Nvidia stands accused of engineering GPU scarcity. If you run inference in production, today's digest is a migration checklist.
Here's the thing: November 15, 2026 isn't a day of shiny launches - it's a day of *deprecations*. The old batch API in llama.cpp is dead by year-end, Marlin kernels are being retired, and Opus 5.5 just shipped a regression that spiked answer correction rates from 5.2% to 25.4%. Underneath the churn, three forces are converging: Rust as the new systems language for inference, 4-bit quantization as the new default, and a local-first rebellion against cloud AI. Let's unpack it.

Why is everyone migrating off Marlin right now?

๐Ÿ”ฅ
Breaking: Marlin is deprecated in vLLM (Issue #65560). Users must re-quantize models or update inference configs to migrate to Humming before the legacy kernels are removed.
Humming is vLLM's new quantization backend, and it's a generational leap. It handles INT4/INT8 GPTQ/AWQ, FP8, MXFP4/6/8, and NVFP4 across both Hopper and Blackwell architectures - replacing Marlin, which was essentially a single-format kernel optimized for an older world. This isn't a drop-in swap; it's a strategic re-architecture of how models get served.
The migration pressure is real because the model zoo moved on. Qwen3.6-27B-FP8 crashes SGLang and corrupts tool calls in llama.cpp/Ollama. DeepSeek-V4.1-Flash has prompt reuse corruption bugs in llama.cpp and crashes Ollama. GLM-5.3-Flash FP4-Local is vLLM-only for now. MiniMax-M3.1 MoE gets native support in vLLM (cuTile NVFP4), llama.cpp, and Unsloth, with an opt-in cuTile NVFP4 W4A4 runner in SGLang for SM12x RTX 50-series GPUs.
  • Marlin - legacy kernel, deprecated. Re-quantize or update configs now.
  • Humming - new backend covering INT4/INT8, FP8, MXFP4/6/8, NVFP4 on Hopper + Blackwell.
  • ZAOERIS AXIOM MoE (79.43B) - sparse MoE proposed for native vLLM support (Issue #67412).
  • Edge0-145B - open-source model claiming frontier capability on consumer hardware via aggressive quantization.
  • End-to-End 4-Bit Training - Thinking Machines deep dive showing ultra-low-bit training is becoming viable.
Worth watching: Humming's cross-architecture coverage means one quantization pipeline now serves Nvidia Blackwell SM120 (RTX PRO 6000), AMD MI355X/gfx942 (via FlyDSL hitting 21-31% of MXFP8 peak), and eventually Intel NPU (Core Ultra, requested in Ollama #19535) and Ascend 910B (experimental SGLang NPU support for Qwen3.8-27B GDN hybrid models). The multi-vendor inference dream just got less theoretical.

Is Rust taking over the AI inference stack?

๐Ÿฆ€
Pattern: vLLM, SGLang, and LiteLLM are all converging on Rust backends in the same week. This is no longer experimental - it's the new production baseline.
VLLM_USE_RUST_FRONTEND is now stable enough for production, with the caveat that the Rust and Python frontends will *diverge* in API behavior over time. SGLang is migrating its control API to SGLANG_RUST_SERVER with Ollama/Anthropic/Vertex-compatible routes, and LiteLLM shipped signed v1.110.0-rc.3 with Cosign supply-chain integrity, spend logging, and congestion-aware queuing.
Why Rust? Memory safety without GC pauses, fearless concurrency for the KV cache management problem, and predictable latency under load. For agent workloads that hammer inference endpoints with thousands of concurrent tool calls, this matters more than raw FLOPS.
  • vLLM - Rust frontend stable; Blackwell SM120 support; sparse MLA and FlyDSL kernels on MI355X.
  • SGLang 1.0 - major release emphasizing stability, explicit config errors over silent changes, backward compat policy.
  • LiteLLM v1.110.0-rc.3 - Cosign-signed releases, spend logging, congestion-aware queuing.
  • Codewhale - fully open-source, auditable Rust coding agent with multi-agent task splitting.
  • Swizzle Runtime v0.1.0 - pure Rust local inference hitting 209 tok/s and 43ms TTFT on a 4GB laptop GPU.
Hot take: the Python-first era of LLM tooling is ending. Python will remain the *modeling* layer, but the *serving* layer is being rewritten in Rust right now.

Which AI coding tools shipped breaking changes today?

โš ๏ธ
Opus 5.5 regression: Answer correction rate spiked from 5.2% to 25.4% after the October 19 update (Issue #105112). If your code generation quality dropped recently, this is why.
The CLI agent space is in a weird middle phase - lots of hardening, not many releases. Claude Code is fighting memory leaks (fseventsd on macOS), resource governance for agent process trees, and MCP session state drift. OpenAI Codex rolled out rollout budget enforcement (PRs #61577-#61583) plus request ID tracing, but the community is loud about cross-device session resumption bugs.
  • Gemini CLI - autonomous subagent initiation, AST-aware navigation, security guardrails blocking destructive commands.
  • Gemini 4 Hydrogen - agentic coding model focused on fast, self-correcting code generation.
  • GitHub Copilot CLI - macOS 12.7.6 backward compatibility and session resilience against memory leaks.
  • OpenCode - ephemeral sessions, TUI polish, plugin lifecycle control post-v2.3.0; new nightly build.
  • Qwen Code - nightly v0.27.2-nightly.20261113.6f754be654; leading on CI/CD rigor and shared agent loop migration.
  • Pi - fullscreen copy, tool output truncation, Docker template for local-first workflows.
  • llama_batch deprecated (PR #32306) - migrate to llama_batch_ext by end-of-year.
The MCP layer is showing its age. Ecosystem-wide tool drift and discovery failures (Claude Code #97369, Codex #53169) highlight how fragile plugin contracts really are. Selvedge (SQLite-backed rejected-approach memory) and compact-memory (symbolic notation for agent state compression, Issue #1329) are community patches for problems the spec should have solved.

Is the self-hosted agent stack ready for production?

๐Ÿ 
Local-first is no longer niche. Driven by Qwen3 and DeepSeek-V3, plus remove.bg shutting down December 1st, developers are voting with their terminals for self-hosted AI.
The agent framework space is fragmenting *productively*. OpenClaw shipped v2026.11.5-beta.3 with reduced memory pressure during transcript compaction and adaptive session hydration - streaming only selected hydration data with configurable summary limits. That's the kind of boring infrastructure work that makes 100K-token sessions actually usable.
  • Hermes Agent v0.23.1+canary - unified gateway architecture handling all local conversations via a single process; enables profile lifecycle but introduces session stability regressions.
  • QwenPaw v2.3.0 - AgentScope's plugin-driven mobile+cloud agent; empty stream retries, UTF-8 legacy file loading fixes; Olostep proposed as web_search provider (PR #8741).
  • ZeroClaw - WASM-sandboxed, SSH-native runtime pursuing multi-user identity injection and encrypted credentials.
  • IronClaw - NEAR AI agent project with zero activity in 24h; flagged as stagnation risk.
  • Agent Resource Governance - cross-tool demand for memory caps, process teardown guarantees, predictable turn limits.
  • Shotoku - local-first action governance and budget caps for agent safety.
  • Agweb - lightweight open-source browser built specifically for autonomous agents.
The SEC-MARATHON security hardening initiative is driving in-process secret redaction and approval gates across Hermes Agent. Combined with webapp-testing's command injection protection and the growing prompt injection safety discourse, agent security is finally getting the attention it deserves.

What's driving the token efficiency revolution?

๐Ÿ“‰
Token costs are being attacked from every angle: caveman cuts usage up to 65% via prompt simplification, headroom compresses tool outputs 20-95%, and affaan-m/ECC optimizes agent harnesses for efficiency.
This is the quiet revolution. As agents get more autonomous, they generate *massive* tool output, logs, and context. The new optimization layer treats tokens like a scarce resource to be managed, not wasted.
  • caveman - viral proxy reducing LLM token usage up to 65% via prompt simplification.
  • headroom - compresses tool outputs and logs before LLM ingestion, cutting tokens 20-95%.
  • affaan-m/ECC - agent harness optimizer gaining massive traction for performance and token efficiency.
  • claude-mem - persistent cross-session context storage with behavior compression.
  • Ragmir - open-source retrieval layer pulling context from code, PDFs, Office files.
  • firecrawl - open-source crawler API for real-time web data in RAG pipelines; free tier + self-hostable.
  • Apple Faster Retrieval for RAG - new indexing methods for speed; upcoming Mac app for local AI models.
Worth watching: temperature=0 determinism analysis showed 17 of 5,000 greedy tokens flipped - proving deterministic sampling isn't fully reproducible due to floating-point behavior. If you're benchmarking agents, this matters.

๐Ÿ“Š Inference Engine Migration Cheat Sheet

๐Ÿ“Š Engine | Breaking Change | Migration Action

  • **vLLM** โ€” **Marlin** deprecated for **Humming** โ€” Re-quantize models; update inference configs (Issue #65560)
  • **vLLM** โ€” **VLLM_USE_RUST_FRONTEND** now stable โ€” Enable in production; expect API divergence from Python
  • **SGLang** โ€” Control API moving to **SGLANG_RUST_SERVER** โ€” Adopt rust-server for high-throughput agent workloads
  • **SGLang 1.0** โ€” Explicit config errors replace silent changes โ€” Audit configs before upgrading
  • **llama.cpp** โ€” **llama_batch** deprecated (PR #32306) โ€” Migrate to **llama_batch_ext** by end-of-year
  • **LiteLLM** โ€” Signed releases via **Cosign** โ€” Verify signatures in CI/CD pipelines
  • **Hermes Agent** โ€” Unified gateway architecture (canary) โ€” Test session persistence and API routing carefully

โšก Quick Bites

  • Nvidia - allegations of engineering GPU scarcity by funding AI startups to hoard chips; antitrust debate ignites.
  • Jensen Huang - departure from Nvidia speculated to link to alleged smuggling of 1.5M banned AI GPUs to China.
  • Claude Meetings - integrates Claude into live calls for real-time task extraction; 293 votes on Product Hunt.
  • Filament by Galaxy - open-source data movement tool for terminal/agent workflows; top voted with 306 votes.
  • qorqbot - proactive enterprise agent handling repetitive business tasks without prompting.
  • Pocket Studio for Supabase - full database control on Android.
  • Trainn MCP - generates product videos and courses from any AI chat.
  • GAtrends - keyword spike alerts via Telegram, Claude, ChatGPT.
  • Paxper - visual cloud infrastructure builder with change review and audit trails.
  • Burnstate - proactive API usage monitoring agent.
  • Expo Simulators - cloud simulators for testing AI agents in realistic environments.
  • libreagi - privacy-focused platform preserving open-source AI models without vendor lock-in.
  • prompts.chat - community prompt library now self-hostable as privacy-first alternative.
  • open-webui - friendly interface for local LLMs via Ollama and OpenAI API.
  • RAGFlow - evolving into agent-native knowledge infrastructure.
  • LlamaIndex - document processing powering RAG pipelines with multi-vector-DB integration.
  • LangChain - de facto agent engineering platform with deep RAG and multi-agent integration.
  • AutoGPT - foundational project for autonomous AI agent research.
  • career-ops - open-source AI job search agent; runs locally or as Claude plugin.
  • CowAgent - lightweight personal assistant with memory and self-evolution.
  • nanobot - minimal, modular agent framework for low-resource and edge deployment.
  • ppt-master - LLM-powered PowerPoint generation with animations and narration.
  • daily_stock_analysis - LLM stock analysis with real-time news and zero-cost scheduled runs.
  • Vibe-Trading - AI trading agent for retail investors.
  • MoneyPrinterTurbo - viral HD short video generation from keywords.
  • alibaba/open-code-review - enterprise-grade hybrid LLM + deterministic code review.
  • qwen-code - growing enterprise demand for self-hosted AI coding tools.
  • Ollama - community integration wave (PipesHub, SynapCores, ForgeMind, inference-vitals) plus critical VRAM accounting fix.
  • Unsloth - native support for DeepSeek-V4.1, MiniMax-M3.1; voice mode and beta desktop update.
  • KV Cache - key fix for slow inference of large models on constrained VRAM.
  • Reward gaming - agent leaderboards can be gamed, producing misleading scores.
  • University of Michigan energy study - non-English text generation consumes significantly more energy.
  • Gรถdel notebooks AI discovery - AI revealed hundreds of long-lost proved theorems.
  • AI literary scams - The Atlantic exposรฉ on AI-generated books flooding Amazon.
  • Jonathan Blow critique - argument that AI hasn't delivered transformative change in software development.
  • Jane Street - NYC datacenter nearly drowned by false smoke alarm; infrastructure reliability risk.
  • ML Modules for Mere Mortals - modular ML design patterns for non-experts.
  • The Little Book of Generative AI Foundations - ArXiv primer on generative models.
  • agents-radar - auto-generation system producing this digest.

โ“ FAQ: Today's AI News Explained

  • Q: Why is Marlin being deprecated in vLLM? - Marlin was a single-format quantization kernel that couldn't keep up with the explosion of new formats (FP8, MXFP4, NVFP4). Humming replaces it with unified support across INT4/INT8 GPTQ/AWQ, FP8, MXFP4/6/8, and NVFP4 on both Hopper and Blackwell. Users must re-quantize models or update configs (Issue #65560).
  • Q: What happened to Opus 5.5's code generation quality? - After the October 19 update, the answer correction rate spiked from 5.2% to 25.4% (Issue #105112). This is a confirmed regression that erodes trust in code generation. If you noticed degraded output, this is the root cause.
  • Q: Should I switch to the Rust frontend in vLLM? - Yes, VLLM_USE_RUST_FRONTEND is now stable for production. But be aware the Rust and Python frontends will *diverge* in API behavior over time, so plan your migration carefully and pin versions.
  • Q: What is the unified gateway architecture in Hermes Agent? - It's a refactor consolidating all local conversations into a single process, enabling profile lifecycle management. The v0.23.1+canary build introduces it but brings session stability regressions - test thoroughly before adopting.
  • Q: Why are so many AI tools moving to Rust? - Memory safety without GC pauses, fearless concurrency for KV cache management, and predictable latency under load. vLLM, SGLang, LiteLLM, and new agents like Codewhale and Swizzle Runtime all converged on Rust this week.
  • Q: What's driving the local-first AI movement? - Privacy requirements, regulated environments, low connectivity, and the shutdown of services like remove.bg (December 1st). Models like Qwen3, DeepSeek-V3, and Edge0-145B make self-hosted viable on consumer hardware.
๐Ÿ”ฎ Editor's Take: The Marlin-to-Humming migration is the clearest signal yet that inference infrastructure is consolidating around *format flexibility* over *single-format optimization*. Teams that treat quantization as a one-time setup are going to get burned repeatedly. The winners here will be the ones who build re-quantization into their CI/CD pipelines - everyone else is one deprecation notice away from an outage.