Running 177B Models at Home on a 2015 Dell Workstation

This is the build log and runbook behind my local LLM box: a retired Dell Precision T7910, three used Tesla V100s, and the path I took to run a 177B-parameter model at 524K context on 2017 silicon.

What this is: the full build log of my local LLM rig, start to finish. I replaced the CPUs, added three datacenter GPUs to a workstation, fought the power and cooling, and got llama.cpp serving a 177B Qwen hybrid and a 284B DeepSeek at home. I use llama.cpp for inference and V100 GPUs for compute. If you want to know whether a retired workstation can carry modern models, this is the honest answer.

Every number below is measured on this box at real context fill. Nothing here is a spec sheet, and nothing is a benchmark from someone else’s hardware. Where I quote a community figure instead of my own measurement, I say so.

1. Why do this at all

Three reasons drove the build.

Ownership. I run long agentic conversations, 90K tokens and up. I want those on my own hardware. No API logging, no content filter, no deprecation notices. The models I use daily are uncensored repacks, and I want to keep them that way.

Long context. The house standard ended up at 262,144 tokens, and the current record is 524,288. Renting that from a provider is expensive, and it comes with limits I do not control.

Old hardware is worth it. The industry dropped Volta years ago. Getting a 2026 model to run well on 2017 silicon turns out to be a solvable problem, and it’s an interesting one.

One design decision shaped everything. This box runs big models for humans and long conversations. It does not run the agent loop. Small local models hallucinate tool results, so the agent loop stays on frontier APIs. The T7910 is the heavy local brain. That is its job.

One working rule, now that it comes up constantly: measure everything, trust nothing you did not measure. Every number in this post was measured on this box at real context fill. The times I trusted a document instead of the machine, I got burned. I show you where that happened.

2. What is in the box

The T7910 is a dual-socket Broadwell-EP workstation: two E5-2697A v4s, 16 cores each, DDR4-2400, 16 DIMM slots across 4 channels per CPU, and 7 PCIe slots. It was never meant to be a GPU server. It has the slots, the CPUs, and it keeps running, which is enough.

What the box serves right now:

• Qwen3.8-Flash-Next, a 177B-parameter hybrid MoE, at 524,288 tokens of context across two 262K slots.
• DeepSeek-V4-Flash-0731 “Quality128”, a 284B MoE with 13B active, on the cache-compression path with a CPU-side speculative drafter.

The Qwen model uses 87.4 GiB of the 96 GiB of VRAM. It has vision support, an OpenAI-compatible API on port 8081, an OpenWebUI front end, and a network endpoint my other machines reach over the LAN.

Slot layout

The box started with two V100s plus the P400. Adding the third V100 meant moving the P400 out of its original slot on CPU1, a real x16 slot, and down to the lowest bus slot so the V100 could take the good one.

• V100 #1 and #2 on CPU0 (03:00.0, 04:00.0)
• V100 #3 on CPU1 (A1:00.0), on a right-angle riser to reach the top of the chassis
• P400 at 02:00.0

That last layout caused a subtle problem. Because of how CUDA enumerates devices, the P400 sometimes shows up as CUDA device 0 in nvidia-smi. Keep that in mind. It bites later.

Two checks are mandatory every time you add or move a GPU:

1. Re-check nvidia-smi indices. The display GPU quietly re-includes itself after any re-enumeration.
2. Run llama-server --list-devices before writing any launcher. Windows re-enumerates PCIe on boot. Cards land on new paths or leave phantom devices behind. Those are harmless and removable with pnputil /remove-device.

3. What it cost

Here is every part and what it paid for it. The chassis and the RAM upgrade are listed separately because the machine’s current state differs from how it arrived: the chassis shipped with the E5-2643v3s and 32 GB, the 2697A v4s went in later, and the 128 GB RDIMM lot is the upgrade I have in hand.

Used (eBay)

New (Amazon)

Memory upgrade (bought, install pending)

Total build cost

The chassis arrived with 32 GB (8×4 GB) and the E5-2643v3s. What it runs now is the 2697A pair and the three V100s. The 128 GB RDIMM lot is the upgrade in hand.

4. The Volta problem

This is the most important section if you want to copy the build. Read it before you buy anything.

The V100 is sm_70, and the industry has been dropping Volta support for two years. Here is where things stand:

• vLLM is dead on Volta. v0.20+ dropped sm_70 entirely (SM70 (Volta) is deprecated). The last usable version is 0.18.x, and even that loses to llama.cpp for single-user work: no Marlin or GPTQ-AWQ kernels before Ampere, FP16 weights only, no speculative decoding path, no native Windows.
• TensorRT-LLM needs sm_80 or newer. Classic TensorRT needs sm_75 or newer. EXL2 needs Turing or newer. All dead on Volta.
• CUDA 13 removed Maxwell, Pascal, and Volta from the toolchain. PyTorch cu128 and cu129 wheels dropped sm_70. cu126 is the last line that supports it.
• MXFP4 native kernels are Blackwell-only. On a V100, anything 4-bit goes through the DP4A fallback.

llama.cpp is the only actively maintained engine that still supports Volta. That is not a compromise. It is why the project exists. But it has two consequences.

First, prebuilt binaries do not include sm_70. The official release zips ship sm_86 and sm_89 only. On a V100 they fail with failed to load model even though the CUDA backend loads fine, which sends you debugging in the wrong direction. Every build on this box, roughly a dozen of them, was compiled from source with -DGGML_CUDA_ARCHITECTURES=70. I use official prebuilts everywhere else. On Volta there is no choice.

Second, Windows versus Linux barely matters for this workload. For fully offloaded, GPU-bound llama.cpp on TCC-mode Teslas, the difference is 0 to 3 percent. The big Linux wins, 10 to 25 percent, only show up in CPU-bound partial-offload and WSL2 setups. I stayed on Windows Server because this is a daily driver that launches from a visible desktop .bat window. The Ubuntu dual-boot plan exists for a different reason: some engines other than llama.cpp refuse sm_70 on Windows entirely.

The build command

Shared build fingerprint across all forks: Release, Ninja, MSVC 14.44, CUDA 12.8, GGML_CUDA=ON, GGML_CUDA_ARCHITECTURES=70, GGML_NATIVE=OFF, GGML_CPU_ALL_VARIANTS=OFF. The qwen4exp build also sets GGML_CUDA_FA_ALL_QUANTS=ON.

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_ARCHITECTURES=70 -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF

Three patches that made modern models load

Getting 2026’s FP4 and MXFP4-heavy models to run on 2017 silicon took three separate fixes. The DeepSeek build needs all three. The Qwen build needs none of them.

1. GGML_CUDA_FORCE_MMQ=ON. Routes quantized GEMMs, MXFP4 and friends, to the DP4A path instead of cuBLAS tensor-ops. Fixes CUBLAS_STATUS_NOT_SUPPORTED on batch 64+ quantized GEMMs.
2. A source patch in ggml-cuda.cu: change CUBLAS_GEMM_DEFAULT_TENSOR_OP to CUBLAS_GEMM_DEFAULT at all three call sites. F16 dense GEMMs still crashed during big prefills. DEFAULT lets cuBLAS pick a kernel that works.
3. GGML_CUDA_GRAPHS=OFF. cudaGraphInstantiate runs out of memory after single prefills longer than about 23K tokens.

The fork zoo

Upstream llama.cpp moves slower than model architectures do, so the box accumulated a pile of source builds, each with one job. A MoE-cache fork is the workhorse on the DeepSeek path. The qwen4exp build handles Qwen3.8-Flash-Next with the MTP graph wired in, one upstream PR rolled back and another rolled in.

One cleanup is worth mentioning. At one point I deleted about 930 GB of stale build trees and old models, leaving one serving build and one model. What made it safe: move the runtime dependencies into the surviving tree and patch the launcher before deleting anything, then verify the server after.

5. Cooling, or the 82-degree problem

The third V100 is a passive-heatsink Tesla. Datacenter cards assume a wall of chassis airflow. A workstation assumes its GPUs have fans. Card #3 sat at the top-right of the chassis, the worst airflow spot in the case, and for weeks nobody noticed, because short OpenWebUI sessions never heated it up.

Then I started running sustained agent workloads, and the truth came out. GPU3 idled at 82 °C pulling about 54 W. During decode it sat in software thermal slowdown, SM clock pinned at 135 to 142 MHz against a 1380 MHz maximum. A three-way layer split that should have decoded around 23 t/s was crawling at about 11 t/s.

The GPU was fine. It was running hot.

The fix was mechanical: put a blower fan on GPU3 identical to the ones already on GPUs 1 and 2, each on a 3D-printed adapter shroud that seats a standard 97×33 blower on the V100’s heatsink. The fans draw 12 V from a SATA power connector through a fan-power splitter. Results, verified with nvidia-smi under real load across all three cards:

• Idle: 82 °C down to 36-39 °C on all three V100s
• SM clock: 135-142 MHz throttled up to 1260 MHz, no throttle events

The lesson generalizes. A passive datacenter card in a workstation chassis is fine until you measure it under sustained load. Idle temps and short benchmarks hid this for weeks.

6. Power

The T7910’s mainboard power is a proprietary 28-pin connector plus two 8-pin EPS for the CPUs. The Dell PSU has no 24-pin ATX connector at all. It plugs edge-on into a power distribution board. That shaped every power decision:

• V100s take EPS 8-pin, not PCIe 8-pin. Tesla uses the CPU pinout. I had to adapt the Dell PCIe cables to the V100’s female connector with COMeap 8-pin CPU-to-EPS adapters.
• The stock Dell 1300W PSU feeds the board plus two V100s from its distribution board outputs.
• A fourth V100 was attempted off a SATA-to-12V adapter branch. The branch died. A second branch died later during the cooling work.

I learned that lesson twice so I would not have to learn it a third time: SATA power adapters are not a power delivery strategy. The 4-GPU plan survives, but with a proper second PSU instead of adapters.

7. The doc said 64 GB of RAM. The board had 32 GB.

Every planning document for the DeepSeek-V4-Flash runs assumed 64 GB of RAM. It said so in writing. When I finally enumerated the DIMMs with Win32_PhysicalMemory: 8× 4 GB Hynix 1Rx4 @2400, 32 GB total. The doc was wrong, the board was half empty, and RAM was the thing gating the 284B-class models I wanted to run.

Broadwell-EP memory rules, for anyone speccing the same upgrade:

• 1 DIMM per channel (8 sticks or fewer) runs at 2400 MT/s
• Any channel at 2 DIMMs drops the whole socket to 2133. Adding sticks never gets 2400 back.
• 32 GB sticks were priced out, so 16 GB 2Rx4 PC4-2400T ECC RDIMM is the part to buy
• 8× 16 GB gives 128 GB @ 2400 (what I picked); 12× gives 192 GB @ 2133; 16× gives 256 GB @ 2133

I bought a 12-stick lot of Micron 16 GB 2400T at $757.70, about $63.14 a stick. Plan: install 8 for 128 GB at 2400, keep 4 as tested spares in case a model bigger than 150 GB shows up. It is in hand but not installed, so the box is still on 32 GB.

One stick in the lot was dead, with an amber light. The diagnostic routine, learned over several evenings:

• “Worked before, won’t POST after a failed upgrade” usually means sticky memory training. Power drain, CMOS battery out, hold the power button 15 to 60 seconds, wait, then let it retrain for 2 to 3 minutes. Do not interrupt training.
• Move the suspect stick into a known-good occupied slot. That separates a dead stick from a dead slot.
• Booting one stick on a CPU2 with no memory of its own is valid (UPI forwarding). Never a POST blocker.
• A CMOS clear resets BIOS fan settings to Auto. Set them back after.
• Slot map: DIMM1-4 per CPU are the slots that get exercised. DIMM5-8 are invisible until populated. A dead secondary slot hides until you need it.

Drives: inventory before you delete

The worst day of the project produced a standing rule. On 08-29, about 88 GB got deleted while the intended target drive sat empty, because nobody inventoried the drives first. The rule since then, which has saved me twice: inventory every drive before deleting anything, and run an orphan audit.

8. The physics of fast inference on old cards

Every config decision in this build comes down to one equation, verified over and over:

Decode speed ceiling = single-GPU HBM bandwidth / total weight bytes.

A V100’s HBM2 gives you about 900 GB/s per card. Whatever you force that bus to read per token sets your speed. Everything else follows from that.

Layer split is a relay race

Splitting a model across two GPUs by layers means GPU0 computes its half of the layers, hands off to GPU1, and waits. Only one GPU works at any instant. nvidia-smi shows alternating 100/0 and 0/100, which is expected. The two HBM2 pools never add together. Consequences:

• A model that fits on one card gets slower when you split it. Measured on a 17.9 GB Q4: 51.6 t/s single-card versus 38.6 split at short context; 29.9 versus 21.1 at 62K fill; speculative draft acceptance 62% versus 34.5%.
• F16 is a trap on V100. An F16 27B model decodes at about 7.6 t/s at 100K+ fill, versus 30.5 t/s for Q8_0. Half the speed, because it is bound by HBM2 bandwidth and F16 doubles the bytes. Q8_0 GGUF is the right format for Volta: full quality, about 82% of the bandwidth ceiling. BF16 does not even run natively on sm_70. There is no bf16 hardware, so raw BF16 gets converted in-kernel per layer.
• Custom quants fall back to generic kernels. One “K_P” quant hit only about 64% of ceiling. Do not predict speed from a quant’s file size. Benchmark it.

The one multi-GPU speedup that works is --split-mode tensor (build b10545 or newer). On Qwen3.8-27B Q8_0 across 2× V100 at 110K real fill: 30.5/31.5 t/s versus about 22 t/s for layer split, a 40% gain, with prefill at 565 versus 518. Splitting the KV cache across cards halves per-card attention reads at deep fill, which is exactly the deep-context bottleneck. One caveat: on 3 devices it hits an AllReduce power-of-2 assert unless your build has the upstream fix. My current fork does not, so the 3-card rig runs layer split.

Hybrid architectures change the KV math

Modern hybrid models (Qwen’s Gated-DeltaNet lines) made KV cache nearly free. On Qwen3.6-27B, only 16 of 64 layers are real gated attention carrying KV. The other 48 are recurrent linear-attention blocks holding constant state, about 12 MB total, independent of context length.

The rule for every hybrid since: count only the full-attention layers when budgeting KV. This is why the 524K record later cost about 7.5 GiB instead of doubling.

The flip side: recurrent layers are sequential over the sequence dimension and cannot be tensor-parallelized. --split-mode row crashes at load on every build I tested. Hybrids are layer-split or nothing. MoE hybrids doubly so, since they have both experts and SSM layers.

Speculative decoding

Volta decode is bandwidth-bound, so anything that avoids decoding one token at a time is worth real money:

• MTP (baked-in multi-token-prediction heads). The heads ship inside the GGUF, no separate draft model needed. On bandwidth-bound Volta, MTP is a requirement, not a bonus.
• DSpark drafter. DeepSeek-V4-Flash’s native fast path: --spec-type draft-dspark, roughly 2× throughput. The drafter architecture string has to match exactly or you get unknown model architecture at load.
• MoE cache compression (--moe-cache auto). Caches expert output per head-group so repeated expert calls skip compute. The memory-side complement to speculative decoding. Combined with the DSpark drafter, it is the biggest single win on the DeepSeek path.
• The sleep-idle VRAM leak. MTP draft context does not get freed on --sleep-idle-seconds cycles and accumulates until OOM. Fixed upstream but not in my forks, so MTP launchers never carry the sleep flag.

House standards

Settled early, never overturned:

• Quant floor: Q8-tier for dense quality, Q5_K_M as the absolute floor. No AWQ, nothing sub-Q8 for anything that needs to be good.
• KV cache: q8_0 floor. q4_0 KV was a single-card compromise. The quality loss compounds with context length, and long-context retrieval fails first.
• Benchmarks lie at short context. Report tokens/sec at real context fill (90K+), never short-context warm-cache. A short-context number on this hardware can be off by 2×.
• Uncensored or nothing. Every daily-driver model on this box is an abliterated or otherwise uncensored repack.

9. The models

DeepSeek-V4-Flash-0731 “Quality128”: the 284B that fits

The 284B/13B-active DeepSeek Flash was the defining model of the middle of the project: agentic depth, 1M native context, and a Q4-class quality ceiling because the official weights are FP4/FP8 mixed at source (MXFP4 experts, so Q5/Q6/Q8 on the experts is wasted bytes).

• Fit math first. At 96 GB of VRAM, the roughly 96 GB “Quality128” quant (GGUF v3, arch deepseek4) fits fully in VRAM with the drafter. The trick was surgical expert eviction: --override-tensor regexes push only expert tensors to RAM while attention stays on GPU. Never --n-cpu-moe, which evicts the wrong end and starves GPU0.
• Quality128 + DSpark became the daily. An abliterated repack with MXFP4 kept on the most sensitive expert layers, plus the DSpark drafter running on CPU. --moe-cache auto with GGML_CUDA_MOE_CACHE_ADMIT_AFTER=64 gates cache admission until a layer-pair has been seen 64 times. The 6.9 GB dflash drafter must run on CPU, not GPU, or the cache will not engage. That costs 5-7 GB of the 32 GB of system RAM.
• The honest verdict. At 2-bit, DeepSeek is not smarter than a Q8_0 27B dense for chat. 13B active at 2-bit is roughly half the clean compute of 27B dense at Q8. Its advantages (agentic depth, 1M context) only show up at Q3/Q4 quality. DS4 is the agent brain for long autonomous runs; a dense Q8 27B is the fast chat brain. Upgrading to a proper Q4_K (153 GB, already downloaded and hash-verified) is gated on RAM, which is exactly what drove the 128 GB purchase.

Also evaluated and rejected: GLM-5.3 (not viable on this hardware), a TensorSharp .NET engine (missing sm_70 flash-attention kernels even after a real rebuild; llama.cpp won 39.9 t/s versus OOM), and DwarfStar (good for sm_70 verification, rejected as an agent backend because small local models hallucinate tool results).

Qwen3.8-Flash-Next: 177B on 96 GB, the current daily

This architecture is strange in a good way. 177B total: a 125B transformer with only 6B active per token, plus a 51B n-gram PLE (per-layer token embeddings in a 20M-row hash table, a RAM/disk structure touched every token), plus 4B of MTP. 48 layers in repeating Gated-DeltaNet/MoE blocks with sparse attention. MoE with 512 experts, 10 routed plus 1 shared. Native context 262,144.

First contact was a red herring. The Q4_K_XL quant loaded (86.3 GiB across three cards) and prefilled to 112,640 tokens, 86% of the way, before a CUDA OOM that needed just 1.6 GB more. The entire crash chain turned out to be the CUDA_VISIBLE_DEVICES device-map bug (section 10). With the env var gone: Q4_K_XL at 262K survives 113K fills at 11.47 t/s decode and 129.6 t/s prefill. The 262K launcher went live.

Then, on 09-03, the record. The v2-mtp build (MTP graph finally wired, one upstream PR rolled back, another rolled in) loaded the IQ4_XS 92 GB (98.4 GB decimal) file with a Q4_K_M MTP draft head (2.5 GB) at -c 524288, two slots of 262,144 each, and generated. Idle VRAM: 87.4 GiB of 96 GiB. Doubling 262K to 524K cost only about 7.5 GiB because of the architecture: of 48 layers, only every fourth (12 layers) carries real KV plus the indexer cache. The other 36 are SSM-recurrent with fixed state that does not scale with context. On a dense model this would have been impossible.

The 262K launcher was retired the same day.

10. Windows CUDA problems you will also have

Collected in one place so they cost you hours instead of days:

• The LLAMA_ARG_DEVICE poison. llama.cpp parses LLAMA_ARG_* env vars before CLI args, and an empty LLAMA_ARG_DEVICE makes every launcher in every new cmd window abort instantly. The symptom: all models crash identically from .bat double-clicks while working fine from a clean shell. It looks exactly like a global hardware failure. Fix: reg delete HKCU\Environment /v LLAMA_ARG_DEVICE /f. Prevention is now absolute: never setx any LLAMA_ARG_*. Pass -dev CUDA0,... in the .bat.
• CUDA_VISIBLE_DEVICES breaks the qwen4exp device map. That build enumerates devices in its own order (V100s by PCI ascending, P400 last), so setting CUDA_VISIBLE_DEVICES=1,2,3 re-maps the ordinals and the P400 sneaks in while a V100 gets dropped, giving an instant 38 GB allocation OOM. On that build, never set the variable. Bare -dev CUDA0,CUDA1,CUDA2 already picks the three V100s. Verify with -lv 4. This was the root cause of the entire 177B crash chain.
• Zombie CUDA processes. Killed processes can hold VRAM without appearing in tasklist. nvidia-smi shows phantom compute PIDs. Fix: wmic process where processid=<pid> delete, and reboot if that fails.
• The stale-server trap. “Cannot load the model” from a launcher means check for a stale llama-server holding the port or VRAM first. Always taskkill /F /IM llama-server.exe by image name, never by wrapper PID. That burned me twice: a shell wrapper survived a “kill” and kept serving a benchmark.
• CUDA DLL discipline. Fork builds often ship no CUDA DLLs in bin/. Copy cublas64_12.dll, cublasLt64_12.dll, and cudart64_12.dll from a known-good build, or every launch dies instantly.
• Building while the server runs. The exe is locked on Windows. Build into a separate directory, kill the server, copy the DLLs, re-point the desktop bat.
• Health check pitfall. Never use /v1/completions to verify a template-configured server. It bypasses the Jinja template entirely, so template fixes silently vanish and raw <think> blocks leak out. Use /v1/chat/completions only.

Chat templates

The official Qwen chat templates were buggy. I catalogued 15 distinct bugs: agentic stalling from empty-think poisoning, KV cache invalidation, 80% throughput drops from deep Jinja nesting, tool-call crashes. The fix was the community qwen-fixed-v22 Jinja template, passed via --jinja --chat-template-file, mandatory on every Qwen launcher. The subtlest bug: llama.cpp’s C++ Qwen handler hardcodes enable_thinking=true into the template variables, which makes the template’s else false fallback dead code. Fixed by hardcoding {%- set enable_thinking = false %} and re-enabling thinking per request. Restart the server after template edits.

Downloading 100+ GB models without corrupting them

Big model pulls are routine, so the download rules are battle-tested:

• Use aria2c, but on HuggingFace Xet-backed repos it must run single-connection (-x 1 -s 1 --file-allocation=none). Multi-connection eats 403s and 416s mid-download and can leave a file that looks byte-perfect but is corrupt. Multi-connection is fine on non-Xet repos with retries on. If ETA inflates more than 5× while connections drop, kill it and restart. It does not self-heal.
• Getting 403s with the default user agent: send a browser User-Agent header.
• Verification is a hash, not a size. SHA-256 against the HF LFS OID is definitive. A GGUF header parse plus byte size is not enough, because a corrupted tail passes both.
• Grep the binary for architecture support before committing to a 50 GB download. GGUF v2 versus v3 matters for build compatibility.

Defending the configs from other AIs

A recurring problem with running this project alongside AI assistants: other AI sessions have edited my launchers and fabricated binary paths, inflated benchmark numbers by 40 to 50%, and written broken flags. The contamination checklist is now permanent: binary path must be the fork’s build\bin, thread counts sane for the box, flash-attn present (q4 V-cache segfaults without it on some forks), MTP never added to models without MTP layers (failed to create MTP context is fatal at load, and probing it on the production server is forbidden for exactly that reason), and context sizes within the VRAM budget with 3+ GB of headroom. Trust, but grep.

11. The daily launchers

Two launchers, one box. One server at a time on :8081, --alias main, --host 0.0.0.0. Both builds are source-compiled: Release, Ninja, MSVC 14.44, CUDA 12.8, GGML_CUDA=ON, GGML_CUDA_ARCHITECTURES=70, GGML_NATIVE=OFF.

The Qwen 524K MTP daily (qwen4exp)

The current daily, the one that beats the 524K record. Binary at C:\llama-cpp-qwen4exp-v2-mtp\build\bin\llama-server.exe, fork tree llama-cpp-qwen4exp-v2-mtp at HEAD f65b746ef (2026-08-29). Model Qwen3.8-Flash-Next-Uncensored.IQ4_XS.gguf (92 GB / 98.4 GB decimal, GGUF v3, arch qwen4exp), draft mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf (2.5 GB, native MTP heads), vision mmproj-f16 (863 MB).

@echo off
title Qwen3.8-Flash-Next UNCENSORED IQ4_XS - 524K MTP DAILY (8081)
echo ==================================================
echo  v2-mtp build: #28123 rollback + #28023 + MTP graph
echo  IQ4_XS 98.4GB + mtp-Q4_K_M draft (2.62GB)
echo  524K total ctx - 2 slots x 262144 each - draft-mtp n3
echo  p-min 0.75, -devd pinned
echo  VERIFIED 09-03: loads + generates @524K
echo ==================================================
echo.

cd /d C:\llama-cpp-qwen4exp-v2-mtp\build\bin

set CUDA_VISIBLE_DEVICES=
set LLAMA_ATTN_ROT_DISABLE=1

llama-server.exe ^
  -m "E:\models\Qwen3.8-Flash-Next-Uncensored\Qwen3.8-Flash-Next-Uncensored.IQ4_XS.gguf" ^
  -md "E:\models\Qwen3.8-Flash-Next-Uncensored\mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf" ^
  --mmproj "E:\models\Qwen3.8-Flash-Next-Uncensored\Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf" ^
  --image-min-tokens 1024 ^
  --alias main ^
  --host 0.0.0.0 ^
  --port 8081 ^
  -dev CUDA0,CUDA1,CUDA2 ^
  -devd CUDA0,CUDA1,CUDA2 ^
  -ngld 999 ^
  --split-mode layer ^
  --load-mode none ^
  --no-host ^
  --fit off ^
  -ot "per_layer_token_embd.weight=CPU,v\.blk\..*=CUDA2" ^
  --flash-attn on ^
  -ctk q8_0 -ctv q8_0 ^
  -c 524288 ^
  -np 2 ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 3 ^
  --spec-draft-p-min 0.75 ^
  --reasoning on --reasoning-effort medium --reasoning-preserve ^
  --reasoning-budget 4000 ^
  --reasoning-budget-message "Reasoning budget exceeded. Stop thinking and answer now." ^
  --reasoning-format deepseek

echo.
echo Server exited with code %errorlevel%
pause

Line by line: the device env cleared (poison prevention). -dev explicit on all three V100s. Draft device pinned with -devd (the P400 trap). --fit off paired with -ot overrides, because they abort together otherwise; the PLE embedding table is parked on CPU and vision blocks pinned to CUDA2. q8_0 KV both directions. Flash attention on. --host 0.0.0.0, because loopback-only broke OpenWebUI once. The MTP draft head at n3 with p-min 0.75. The reasoning budget with its “stop thinking” message, because an unbudgeted run once went 30 minutes without answering. And a visible window with a version banner, so I always know what is running.

One reasoning note: effort medium with budget 4000 was a deliberate step down from xhigh. Only xhigh, medium, and low are valid values.

The DeepSeek cache-compression daily (moecache)

Designed for long autonomous runs. Binary at C:\llama-cpp-moecache\build\bin\llama-server.exe, fork tree llama-cpp-moecache branch moe-cache-v2-pr at HEAD e3096b046 (2026-08-06). Model DeepSeek-V4-Flash-0731-Abliterated-DS4-Quality128.gguf (96 GB, GGUF v3, arch deepseek4), draft ...-llamacpp-DSpark-support.gguf (6.9 GB, arch dflash, runs on CPU).

set CUDA_VISIBLE_DEVICES=0,1,2
set GGML_CUDA_FORCE_MMQ=ON
set GGML_CUDA_GRAPHS=OFF
set GGML_CUDA_MOE_CACHE_ADMIT_AFTER=64

C:\llama-cpp-moecache\build\bin\llama-server.exe ^
  -m "E:\models\DeepSeek-V4-Flash-0731\Quality128\DeepSeek-V4-Flash-0731-Abliterated-DS4-Quality128.gguf" ^
  --alias main ^
  --host 0.0.0.0 --port 8081 ^
  --jinja ^
  -c 262144 ^
  -fa on ^
  -fit on ^
  -dev CUDA0,CUDA1,CUDA2 ^
  --split-mode layer ^
  --cpu-moe-draft ^
  --moe-cache auto ^
  -t 56 -tb 56 ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --batch-size 2048 --ubatch-size 1024 ^
  --load-mode none ^
  -np 1 ^
  --spec-type draft-dspark,ngram-mod ^
  --reasoning on --reasoning-effort xhigh --reasoning-preserve ^
  --reasoning-format deepseek ^
  --reasoning-budget 4096 ^
  --reasoning-budget-message "The thinking budget is exhausted. Stop deliberating and immediately give your best final answer or make your next tool call." ^
  --spec-draft-model "E:\models\DeepSeek-V4-Flash-0731\Quality128\DeepSeek-V4-Flash-0731-Abliterated-DS4-Quality128-llamacpp-DSpark-support.gguf" ^
  --spec-draft-n-max 5 ^
  --spec-draft-p-min 0.7 ^
  --metrics

--moe-cache auto caches expert outputs per attention head-group so repeated expert calls skip compute; GGML_CUDA_MOE_CACHE_ADMIT_AFTER=64 gates cache admission until a layer-pair has been seen 64 times. The DSpark drafter must run on CPU (--cpu-moe-draft, -devd unset): a GPU-resident drafter breaks cache engagement because the main model’s cache warms between draft verifications. Composite spec draft-dspark,ngram-mod, n-max 5, p-min 0.7.

Note the Volta stability env in this one: GGML_CUDA_FORCE_MMQ=ON, GGML_CUDA_GRAPHS=OFF (the DeepSeek path needs all three fixes). The qwen4exp build needs none of them.

Daily operations

• Launch: double-click the desktop .bat. Always a visible cmd window, never a hidden background process. Hard rule.
• Stop: a kill bat that taskkills by image name. One port, 8081, one model server at a time. Never parallel test instances.
• Benchmarks: llama-bench sets baselines only; it has no speculative decoding flags. Real MTP numbers come from the server API: launch with --spec-type draft-mtp, time /v1/chat/completions, and check draft_n_accepted against draft_n. 70 to 95% is the healthy band. Report at real fill.

12. The scoreboard

All measured on this box, decode numbers at real context fill where stated.

• GPU3 cooling fix: 82 °C with throttle at 135-142 MHz (~11 t/s) to 36-39 °C at 1260 MHz, ~23 t/s restored
• DeepSeek “Quality128” + DSpark (daily): ~33 t/s at high context, high drafter acceptance
• DeepSeek moe-cache + DSpark-on-CPU: ~+188% over baseline (moe-cache alone ~+46%, community numbers for the cache delta)
• Qwen3.8-Flash-Next Q4_K_XL @262K: 11.47 t/s decode @113K real fill, 129.6 t/s prefill
• Qwen3.8-Flash-Next IQ4_XS @524K (daily): loads and generates, 87.4 GiB / 96 GiB, 2 slots × 262,144
• Context records: 262,144 (Aug) to 524,288 (Sep 03)

13. What’s next

1. 128 GB RAM. The lot is bought; 8 sticks go in for 128 GB at 2400. That opens the gate for the DeepSeek Q4_K (153 GB, already downloaded and hash-verified) with real expert spill.
2. The 4-GPU transplant. The T7910 chassis physically maxes out at 3 double-slot Teslas. Pick up a 4th V100 and power it properly: the Dell 1300W feeds the board plus 2 GPUs, and the other 2 get a second PSU. The open problem is the power, because the Dell PSU has no 24-pin to tap, so a plain dual-PSU adapter board will not work. Either a second complete PSU or gutting the chassis to fit a 4th V100 and a small PSU into the case. End state: 128 GB VRAM. The architecture is decided, the parts are not picked yet.
3. Ubuntu dual-boot. The spare drive is wiped and waiting. For engines that refuse Volta on Windows, and for CPU-bound workloads where Linux actually wins.

What I learned

Verify against the machine, not the document. The doc said 64 GB of RAM; the board had 32. The third GPU idled at 82 °C for weeks because nobody ran it hard enough to notice. The fourth GPU plan died on a SATA power adapter. Every wrong assumption had to be found by measuring, and the machine was always right.

On Volta, bandwidth is the whole game. Every optimization in this build, Q8_0 over F16, q8_0 KV over q4_0, MTP and DSpark and cache compression and draft heads, tensor split where the architecture allows it, IQ4_XS over bigger-but-slower, is the same rule restated: read fewer bytes per token, and keep the bus busy while you do it. HBM2 from 2017 does not care how good your model is. It cares how many gigabytes you make it move.

Old hardware is the point. vLLM, TensorRT-LLM, EXL2, and CUDA 13 all dropped Volta. llama.cpp still supports sm_70, and that is why this project exists. A 2015 workstation with 2017 datacenter GPUs is running a 2026 177B hybrid model at half a million tokens of context, on Windows, from a desktop .bat file. No rented instance is more mine than that.

Build window: about nine weeks, mid-June to September 2026. All benchmarks measured on the described hardware at the stated context fill. Some DeepSeek throughput deltas combine my own A/B measurements with community reference data; where I cite a community number I am relying on the source, not a clean single benchmark. Your mileage will vary. Your hashes should not.