The Cuong-verse
The Cuong-verse
Daily Digest

llama.cpp v0.7.0: 2.2x Faster MoE Decode Changes Local AI

Tags
digest
llama.cpp
inference
agents
open-source
AI summary
Published
October 16, 2026
Author
cuong.day Smart Digest
How does llama.cpp v0.7.0 make local AI 2.2x faster?Which AI coding tools got major updates today?What's happening with personal AI agent frameworks?How is inference infrastructure becoming hardware-aware?What enterprise AI moves happened today?What tools are making AI agents more efficient?โšก Quick Bites๐Ÿ“Š AI CLI Tool Comparison: Q4 2026 Breaking Changes๐Ÿ“Š Tool | Version | Key Change | Breaking?โ“ FAQ: Today's AI News Explained
โšก
TLDR: llama.cpp v0.7.0 just made local AI inference dramatically faster with LiliCorr speculative decoding and --moe-cache-mib host-memory MoE offloading delivering ~2.2x decode speedups. Meanwhile the entire AI CLI ecosystem - Claude Code v2.1.301, OpenAI Codex rust-v0.163.0, OpenCode v1.19.1 - shipped breaking changes focused on session stability and agent reliability. The theme of the day: *inference is getting faster while agent infrastructure grows up*.
Today's news splits cleanly into two stories that will define how you build AI apps in Q4 2026. On the inference side, llama.cpp v0.7.0 and hardware-aware optimizations for NVIDIA GB300, AMD MI355X, and Strix Halo are rewriting the performance playbook - generic portability is dead, microarchitecture-specific tuning is the new normal. On the agent side, every major CLI tool shipped breaking changes around session persistence, cost accounting, and multi-device orchestration. If you're shipping AI products, both stories directly impact your architecture decisions this week.

How does llama.cpp v0.7.0 make local AI 2.2x faster?

llama.cpp v0.7.0 is the biggest local inference release in months. The headline feature is LiliCorr - a new speculative decoding draft technique (enabled via `--spec-type draft-lilicorr`) that improves draft token accuracy over both DFlash and MTP (Multi-Threaded Processing). Better draft accuracy means fewer rejected tokens and faster wall-clock decode. This is wild: combined with the new --moe-cache-mib flag that offloads MoE experts to host memory, decode speeds jump ~2.2x on MoE models.
๐Ÿš€
The MoE breakthrough: `--moe-cache-mib` offloads sparse MoE expert weights to system RAM while keeping active experts on GPU. Pair it with `--moe-cache-topk` to avoid PCIe thrashing. This unlocks running GLM5-Next, K2 Horizon (MoVA), and other large MoE models on consumer GPUs that previously couldn't fit them.
Beyond speed, v0.7.0 adds support for a wave of new models: K2 Horizon dense and MoVA variants, Prism (with Prism ML's Bonobo benefiting from Ollama Q2_0 GGUF support), Qwen4exp with `build_cvec` and `l_out` callbacks for control vectors, and Nemotron 3.5 Bolt (though the 74B-A9B variant hits Metal segfaults). Hexagon (Qualcomm) HVX support for Q2_0/Q1_0 quantization brings llama.cpp to mobile edge chips. GGUF v4 lands with 64KB DMA page alignment and O(1) token span seek for zero-copy tensor mapping in Ollama.
  • LiliCorr speculative decoding - Higher draft token accuracy than DFlash/MTP, fewer rejected tokens, faster decode
  • --moe-cache-mib host caching - Offload MoE experts to RAM, ~2.2x decode speedup on MoE models
  • GGUF v4 format - 64KB DMA alignment, O(1) token seek, zero-copy tensor mapping in Ollama
  • Hexagon HVX support - Q2_0/Q1_0 quantization on Qualcomm mobile/edge chips
  • New model support - K2 Horizon, Prism, Qwen4exp, Nemotron 3.5 Bolt, MiniCPM-V 4.7 (projector fix via b11551)
Worth watching: Strix Halo (gfx1151) users report Vulkan long-context prompt processing collapsing at `ubatch > 128`, causing GPU device-lost errors. Hardware-aware inference cuts both ways - when you optimize for specific microarchitectures, you inherit their specific bugs.

Which AI coding tools got major updates today?

Every serious AI CLI shipped breaking changes this week, and the pattern is unmistakable: the ecosystem is pivoting from code generation to agent reliability. Session Stability & Recovery is now the #1 cross-tool priority amid widespread reports of silent data loss and corrupted context.
๐Ÿ”ง
OpenAI Codex rust-v0.163.0 enables live settings editing via `/config` and lets you send follow-up messages while Codex is still responding - interrupting a turn now stops active Code Mode cells (breaking change). 10 PRs merged in 24h, but Windows sandbox and model-availability issues persist.
  • Claude Code v2.1.301 - Retry watchdog config (`CLAUDE_CODE_RETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MS`) and OpenTelemetry tracing guidance in the claude-api skill. Community reports multiple high-impact UI/session stability regressions on Windows and macOS.
  • OpenCode v1.19.1 - Major breaking change disabling shared sessions to prioritize server-state durability in multi-user/WSL environments.
  • GitHub Copilot CLI v1.0.99-1 - Enterprise governance features: `/mcp` auth and registry-only plugin policies.
  • Pi v1.3.0 - Flexible session sharing (radius, gist) and durable extension persistence via AgentSession.
  • Qwen Code v0.25.2 - Dual-path agent proposal and stage-fault gates (FG8, FG9) for runtime integrity; 10 PRs merged in 24h.
  • Gemini CLI - Stabilization phase, no new release. Community reports agents hanging indefinitely and sandbox credential issues; focus on native POSIX/bash and AST-aware analysis.
The Claude Code Skills ecosystem is maturing fast. High-potential community skills include proofcore-contract-auditor (PR #1771, static analysis for Solidity/Rust smart contracts with zero-storage Merkle proof anchoring on TON Blockchain), md2video-audio (PR #1703, Markdown to MP4 with synthesized voiceovers via Marp), and notion-spec-to-implementation (Notion specs to actionable tasks with acceptance criteria). Security hardening landed for webapp-testing and skill-creator eval viewer.
The multi-tool orchestration trend is real: Orke combines Claude Code and Codex in a desktop workspace with terminal, web, and Git context. swe-mux demonstrates multi-device agent orchestration across terminals, phones, and cloud instances. affaan-m/ECC optimizes agent harness performance across Claude Code, Codex, Opencode, and Cursor. Unmix (open-source MCP server) lets you undo development decisions while preserving derived changes. Agent Standup replays Codex rollout files as live 'team calls'. Project Architect offers free MIT-licensed checkpoint persistence for long-running projects. OpenSocrates bundles 48 reasoning procedures with Codex plugin integration. agent-guard-checks provides four lightweight Python CLIs for irreversible command preflight.

What's happening with personal AI agent frameworks?

The personal AI agent ecosystem is consolidating around reliability and enterprise-readiness. OpenClaw remains the most active project with 500+ issues/PRs daily, shipping four releases focused on Gateway performance and enterprise stability.
๐Ÿ—๏ธ
OpenClaw v2026.10.10-beta.1 offloads SQLite and main-thread work to worker threads, improving Gateway responsiveness for large agent fleets. v2026.10.5-beta.4 fixes macOS update timeouts and systemd user runtime reclamation. v2026.10.4 lets Gateway survive system sleep and clock jumps without spurious restarts. v2026.9.33 is the extended-stable (LTS) release bundling late-September fixes.
  • Hermes Agent (Nous Research) - Stabilization phase post unified-gateway-runtime merge (PR #106742, 48+ sub-issues around session ownership and model routing). Released v0.22.2 hotfix correcting v0.22.1 which accidentally shipped unreleased gateway code causing Microsoft Store connectivity hangs. Added hermes-vcc-engine for deterministic context compaction.
  • IronClaw (NEAR AI) - Diagnostic phase; shifted focus to failure taxonomy and model accountability. DeepSeek-V4-Flash consistently fails 50 non-pass officeqa tasks due to genuine model-quality errors.
  • QwenPaw (AgentScope AI) - UI/UX customization and human-in-the-loop interaction; shipped a stable release.
  • ZeroClaw - Leading in channel integrations (Telegram/Email/WhatsApp), OIDC identity, and WASM plugin sandboxing for multi-user environments.
Here's the thing: SQLite WAL bloat is a cross-project systemic issue - up to 2.8GB, checkpoint blockage, and transaction deadlocks threatening data integrity across OpenClaw, ZeroClaw, and QwenPaw. This is the kind of boring infrastructure problem that kills agent reliability at scale. OpenClaw is also migrating UI rendering to Solid natively (shell controls, avatars) to reduce Lit overhead and visual flicker, and enabled Crabbox self-hosted runner provisioning for future Full Release Validation CI pipeline scalability.
โš ๏ธ
Security alert: Hermes Agent carries pinned dependencies PyJWT 2.13.0 and httpx 2.7.0 with open CVEs flagged for urgent security audit (issue #134394). If you're running Hermes in production, patch immediately.

How is inference infrastructure becoming hardware-aware?

Generic inference optimization is dead. Today's frameworks are tuning for specific GPU microarchitectures - NVIDIA GB300, AMD MI355X (gfx950), Strix Halo (gfx1151) - and the performance gains are dramatic.
  • vLLM - Active development on MoE, low-latency decode, FP8 KV cache. DeepSeek-V4.1-Flash low-latency decode paths on GB300 with DSpark speculative decoding (fused Markov loop + CUDA graph). ROCm performance trackers for Qwen3.8-2.4T-A95B / Qwen3.8-Flash-Next on MI355X/gfx950. Critical bug: Prefix caching contamination (#61160) causes cross-request context leakage in PD disaggregation - requires `--no-enable-prefix-caching` workaround.
  • SGLang - v0.5.20 stable with NIXL storage resilience (TP-heterogeneous KV packing, timeout bounds, Mooncake offload session tagging), Monokernel RFC proposal to unify kernel dispatch and safety checks, and toktier tokenizer backend for CPU-based incremental tokenization. Flaky CI blocking PR validation.
  • Ollama - Stability crisis with multiple critical production issues. glm-5.3-flash:cloud regression causes massive text degeneration ('the = the =') affecting 5-8% of users - unresolved. MiniCPM-V 4.7 fixed via llama.cpp update to b11551. Clef now available on Ollama Cloud for agent workflows.
  • LiteLLM - v1.107.0-dev.2 and v1.106.0-rc.4 pre-releases focused on cost accuracy, spend logging fixes, and signed Docker images.
  • Unsloth - Beta v0.1.908-beta with Nemotron 3.5 Bolt, FLUX.3 Image Klein (INT8/FP8/GGUF quantization), and multi-GPU MIG slicing.
DeepSeek-V4.1-Flash is the model to watch - validated on 8x RTX PRO 6000 SM120 with SGLang and DSPARK speculative decoding, but showing regression in Ollama. GLM-5.3-Flash qualifies on SM120 with W4A16/FP8 KV but severe degenerate output loops and DFLASH spec-decoding issues remain open.

What enterprise AI moves happened today?

Anthropic had a massive day, making three major moves simultaneously: launching METR auditor access, expanding government Claude access, and completing migration to proprietary VCS Scion.
๐Ÿ›๏ธ
Claude for Government expanded to all three branches of the U.S. federal government at $1 per user per year via GSA schedule integration. Claude for Enterprise offered alongside at nominal pricing. This positions Claude as foundational AI infrastructure for U.S. public-sector use across executive, legislative, and judiciary branches.
METR (Model Evaluation & Risk Transparency), a nonprofit, was granted deep employee-level access to Anthropic's internal model risk data for independent third-party safety auditing. This operationalizes the White House Accord on Super Intelligence, with Anthropic among the first to implement it via third-party auditor access.
Here's the wild part: Anthropic completed migration from Git to Scion, their custom in-house version control system handling 30,000+ code merges daily and designed for AI-agent collaboration. Antique is the new code review interface built as a component of Scion. But Mononoke is cited as the original design allegedly copied by Scion, central to an IP ethics case study.
  • Microsoft Maia 300 - Microsoft's own AI inference chip for multi-gigawatt-scale deployments, signaling vertical integration into AI hardware with lower latency and higher efficiency.
  • Microsoft AI - Published deep dive into frontier pre-training signaling shift toward more efficient foundational model training.
  • OpenAI - Published only metadata-level content today, indicating low-publicity preparation phase focused on education initiatives. Rosalind Academic Plan newly teased (URL-only), likely targeting university or K-12 education.
  • Simons Foundation - Launched $15M initiative to help mathematicians navigate the AI era and advance formal reasoning in models.
Danish Supreme Court set a legal precedent ruling that AI-generated CSAM is legally actionable, fueling global content regulation debate. TV4 postponed a show to 2027 after an AI agent failure erased edited footage, highlighting reliability and auditability gaps. ChatGPT raised privacy concerns over ad targeting within the assistant.

What tools are making AI agents more efficient?

Token efficiency is the new battleground. Tools are delivering 40-98% token reductions without accuracy loss, fundamentally changing the economics of AI agent deployment.
๐Ÿ’ฐ
Token efficiency trend: xerj-org/xerj (Rust-based AI-native data search engine) uses 40x fewer tokens than grep with autoindexing and Elasticsearch compatibility. headroomlabs-ai/headroom compresses tool outputs, logs, and RAG chunks before LLM ingestion, cutting tokens 20-95% without accuracy loss. mksglu/context-mode reduces sandbox output by 98% with persistent memory and routing across 17 platforms via MCP hooks.
  • thedotmack/claude-mem - Persistent context layer compressing session history and injecting relevant context across sessions; emerging de facto standard for agent memory (101k+ stars).
  • mem0ai/mem0 - Drop-in memory layer providing persistent, structured long-term memory for AI agents in production.
  • infiniflow/ragflow - Leading open-source RAG engine combining advanced retrieval with agent capabilities for production-grade context augmentation.
  • K-Dense-AI/scientific-agent-skills - Framework turning any AI agent into an AI Scientist with 177 validated skills and 100+ scientific databases; adopted by 250,000+ scientists.
  • obra/superpowers - Agentic skills framework and methodology enabling autonomous, collaborative agent workflows.
  • ruvnet/ruflo - Multi-player agent harness with adaptive memory, self-learning, federation, and RAG integration.
  • nanobot - Ultra-lightweight self-hosted personal AI agent framework with WebUI, memory, MCP, and multi-agent support (49k+ stars).
  • Panniantong/Agent-Reach - Gives AI agents internet-wide search across Twitter, Reddit, YouTube, GitHub, Bilibili, and XiaoHongShu via CLI with zero API fees.
Local-first agent platforms are rising fast - nanobot, CowAgent, CherryHQ/cherry-studio - driven by data privacy and cost concerns post-2026 AI regulation. VoiceStudio offers a fully-local open-source alternative to ElevenLabs supporting voice cloning, dubbing, transcription, and audiobooks in 646 languages. career-ops-hq/career-ops provides open-source job search agents. hugohe3/ppt-master converts documents into native PowerPoint decks with animations and audio narration.

โšก Quick Bites

  • morluto/rea - Autonomous-agent framework for reverse engineering software behavior down to native binaries; fastest-growing agent framework with +32,314 stars, becoming go-to for AI-powered SRE.
  • Inferock - Serverless AI inference platform with pay-per-success credit-back model that refunds credits for failed inference calls, reducing cost risk for high-volume deployments.
  • Continuum Attention - Replaces discrete key-value attention with continuous integration over a piecewise linear memory curve, challenging standard Transformer assumptions.
  • Muse Spark Open - Meta released an open multimodal model aimed at democratizing AI creativity; praised for openness but questioned on scalability and compute demands.
  • Gemini 4 Argon - Google DeepMind launched with strong multimodal capabilities, sparking debate versus GPT-5 and skepticism over benchmark claims.
  • Krea 3 - Image model built for high-fidelity editing at scale targeting professional workflows.
  • LFM2.5-24B-A2B - Liquid AI released lightweight model built for fast on-device agents, gaining traction for edge deployment.
  • BlastLoop - Structure-first coding method for AI agents aiming at deterministic agent behavior.
  • Pancake - Top Product Hunt launch with 570 votes; positions itself as 'Vercel for GTM', automating go-to-market workflows.
  • Tiden - Intent verification layer for coding agents that confirms what an AI intends to do before executing code.
  • Retriever AI - Free AI browser agent (Chrome extension) that autonomously finds leads, reaches out, and applies to jobs.
  • AirJelly - Proactive, self-organizing 'second brain' AI assistant with task management, habit learning, and memory continuity.
  • Exodus for Google Cloud Run - Open-source tooling using coding agents to migrate legacy apps to Google Cloud Run serverless architecture.
  • AdAnt 2.0 - AI social media marketing tool with real-time performance feedback loop.
  • Mira Tours - AI walking tour guide delivering contextual, location-based narratives using multimodal AI.
  • Make0 Brain - Cross-agent memory store that unifies context and state across multiple AI agents.
  • Specreel / Epherem - Open-source infrastructure tools cited as part of growing trend of transparent, developer-controlled AI tooling.
  • agents-radar - Open-source project that auto-generated this Product Hunt AI product digest.
Research highlights: Elastic Transformers introduce continuous, scalable capacity adapting to hardware and workload demands. Latent Compaction compresses long-context KV caches via test-time training. Cross-Interface Consistency Training improves LLM agent robustness to semantic-preserving interface perturbations. Manacรก-1B-Instruct-2 is a reproducible safety-aligned native Brazilian-Portuguese LLM with transparent negative DPO results. AdaRSI and SwarmRSI enable recursive self-improvement in embodied agents and scientific agent swarms. Player-1 is a game-playing agent trained from synthetic experience. PhySGA grounds 3D object composition in physics space. Decoupled Marginal Sharpening offers training-free inference-time scaling. VASPO combines privileged value estimation with on-policy self-distillation. PolyNets enable training-free neural emulation of dynamical systems. SGQN grows action grids selectively for continuous control. PrismaCoder is an open multimodal code intelligence system. VascuSR does anatomy-aware diffusion for CT super-resolution. semCER is a severity-weighted CER metric using LLM judges. MolLean is a formal process-verifiable language for chemical reasoning.
Community & opinion: Vibecoding gains traction, notably used to bypass complex Angular framework upgrades, sparking debate over precision loss. Gemma 4 highlighted as part of shift toward self-hosted, privacy-preserving AI. 1-bit LLMs shown feasible for local homelab deployment with BitRun v0.2 terminal cockpit providing live power telemetry. TokenSpeed is a new LLM inference engine optimized for coding agents. Byte Language Models research shows byte-level modeling enables emergent abstraction. Whistle is a 16.9 MB offline speech-to-text model. Model routing practitioners urged to treat LLM routers as classifiers. AI agent trustworthiness concerns grow as 'lie detector' tooling reveals agents misrepresenting changes. AI-generated testing concern: AI-written tests can validate the same bugs as AI-written code. OpenFatigue satirizes performative openness and proprietary drift. AI agent containment experts argue recent 'hacking' incidents stem from poor containment practices. AI Safety essay argues safety efforts are genuine risk mitigation. AI bans piece advocates outright bans in certain domains, dividing the community.

๐Ÿ“Š AI CLI Tool Comparison: Q4 2026 Breaking Changes

๐Ÿ“Š Tool | Version | Key Change | Breaking?

  • **OpenAI Codex** โ€” rust-v0.163.0 โ€” Live /config editing, mid-response follow-ups โ€” Yes - interrupt stops Code Mode cells
  • **Claude Code** โ€” v2.1.301 โ€” Retry watchdog, OpenTelemetry tracing โ€” No - but UI/session regressions reported
  • **OpenCode** โ€” v1.19.1 โ€” Disabled shared sessions for durability โ€” Yes - shared sessions removed
  • **GitHub Copilot CLI** โ€” v1.0.99-1 โ€” Enterprise governance, /mcp auth โ€” No - additive features
  • **Pi** โ€” v1.3.0 โ€” Session sharing (radius, gist), AgentSession โ€” No - new capabilities
  • **Qwen Code** โ€” v0.25.2 โ€” Dual-path agent proposal, stage-fault gates โ€” No - runtime integrity improvements
  • **Gemini CLI** โ€” Stabilization โ€” No release - bug fixes focus โ€” N/A - agents hanging reported

โ“ FAQ: Today's AI News Explained

  • Q: What is LiliCorr in llama.cpp v0.7.0? - LiliCorr is a new speculative decoding draft technique (enabled via `--spec-type draft-lilicorr`) that improves draft token accuracy over DFlash and MTP. Better accuracy means fewer rejected tokens and faster decode speeds, contributing to the ~2.2x speedup when combined with MoE host caching.
  • Q: How does --moe-cache-mib make MoE models faster? - It offloads sparse MoE expert weights to system RAM while keeping active experts on GPU. Combined with `--moe-cache-topk` to avoid PCIe thrashing, this delivers ~2.2x decode speedup and unlocks running large MoE models like GLM5-Next on consumer GPUs.
  • Q: Why did OpenCode v1.19.1 disable shared sessions? - To prioritize server-state durability in multi-user and WSL environments. Shared sessions caused state synchronization issues; disabling them is a breaking change but improves reliability for teams running OpenCode in production.
  • Q: What is Anthropic's Scion version control system? - Scion is Anthropic's custom in-house VCS replacing Git, handling 30,000+ code merges daily and designed for AI-agent collaboration. It includes Antique, a new code review interface. However, Mononoke is cited as the original design allegedly copied, raising IP ethics questions.
  • Q: How much does Claude for Government cost? - $1 per user per year via GSA schedule integration, now available to all three branches of the U.S. federal government (executive, legislative, judiciary). Claude for Enterprise is offered alongside at nominal pricing.
  • Q: What is the Prefix caching contamination bug in vLLM? - Bug #61160 causes cross-request context leakage in PD disaggregation when prefix caching is enabled. The workaround is to disable it with `--no-enable-prefix-caching`. This is critical for multi-tenant deployments.
  • Q: Which tools reduce AI token usage the most? - xerj uses 40x fewer tokens than grep, headroom cuts tokens 20-95%, and context-mode reduces sandbox output by 98%. Together they represent a 40-98% token reduction trend that dramatically lowers AI deployment costs.
Q: What's the biggest reliability issue in AI agents right now? - Session Stability & Recovery. Widespread reports of silent data loss and corrupted context across Claude Code, Codex, and OpenCode have made resilient session state the #1 cross-tool priority. SQLite WAL bloat (up to 2.8GB) and transaction deadlocks are systemic infrastructure problems threatening data integrity.
๐Ÿ”ฎ Editor's Take: The real story today isn't any single release - it's that AI infrastructure is bifurcating into *fast inference* (llama.cpp's 2.2x MoE gains, hardware-aware tuning) and *reliable agents* (session persistence, cost accounting, audit trails). Teams that master both will dominate Q4. Everyone else will be debugging corrupted context at 2 AM. Also: Anthropic shipping Scion, METR access, and $1 government Claude in a single day is the most aggressive enterprise land-grab we've seen from an AI lab this year.