An inference engine, in progress, for running models larger than a machine's memory at high speed. The first model is DeepSeek-V4.1-Flash at full precision, on a single NVIDIA DGX Spark.
The weights take 476 GB, about four times the machine's memory, so the routed experts stay on the SSD. A predictor guesses which experts the next layers need, and a global scheduler coordinates SSD reads, the expert cache and GPU work so that the weights land before the GPU needs them.
C · C++ · CUDA · Python
STBLarge on one DGX Spark. ★ marks where the engine's own techniques act; memory and SSD bars are drawn to scale.
Speed
Category
Benchmark
tok/s
Code
c-hashtable, py-lru
29.0, 24.7
Prose
en-story, zh-essay
16.5, 16.7
Chat
chat
18.9
Tool calls
BFCL v4, 3 samples
24.5, 19.8, 22.0
Multi-turn agent
τ-bench, 22 turns
20.4
Long input
LongBench v2, 2 samples
17.9, 18.4
Long output
2 samples
17.1, 17.0
Average
13 samples
20.23
Generation speed with greedy decoding, prefill excluded. Each sample was run once from a cold start on Nov 14, 2026, and every output matched the engine's saved reference token for token.
Engine version
Average, tok/s
Nov 7
15.08
Nov 12
19.60
Nov 13
19.95
Nov 14
20.23
Earlier versions of the engine on the same 13 samples. The expert cache sets much of the speed: in a test on Nov 9, leaving 20, 35 or 50 GiB of memory to other programs lowered the average by 8%, 18% and 29%.