WASTE: Running a 2.78T-Parameter Model on a Laptop
WASTE is a new inference engine that runs the complete open-weights Kimi K3 model — 2.78 trillion parameters — on a consumer MacBook Pro with 64GB of RAM. It streams the model from NVMe storage, keeping only the essential parts in memory, and achieves 0.49–0.54 tokens per second. This is not a distilled or pruned variant; it's the full model, converted into a 982 GiB container.
The engine is written in C with zero runtime dependencies beyond libc and pthreads. It uses a technique called "expert streaming": a mixture-of-experts model activates only about 4% of its weights per token, so WASTE keeps the model trunk in RAM and reads the selected experts directly from disk. Each expert is stored in a 4 KiB-aligned record, so routing to an expert costs exactly one pread() call — no seeks, no extra reads.
Measured Performance
| Model | Container Size | Minimum RAM | Speed |
|---|---|---|---|
| Kimi K3 2.78T | 982 GiB | 29.05 GiB | 0.49–0.54 tok/s |
| Kimi-Linear 48B | 19 GiB | 1.87 GiB | 10.7 tok/s |
These numbers were measured on a 64GB MacBook Pro. The engine enforces a RAM budget, refusing to start below 29.05 GiB. On a 64GB machine, it automatically allocates a 46GB budget, with 17.56GB used for an expert cache. That cache is critical: it holds one token's working set (17.0 GB), which is the minimum needed to get any cache hits.
The table below shows what happens as the budget increases:
| Budget | Expert Cache | Hit Rate | Decode Speed |
|---|---|---|---|
| 32 GB | 3.32 GB | 0% | 0.50 tok/s |
| 46 GB | 17.32 GB | 17% | 0.54 tok/s |
| 52 GB | 23.32 GB | 31% | 0.04 tok/s |
| 58 GB | 29.32 GB | 39% | 0.02 tok/s |
The cliff at 52GB and above is due to the OS paging out the expert cache. The engine stays within its budget, but the machine doesn't have enough free RAM, so cache hits become page faults. The default budget calculation steps down in whole working-set multiples, picking the largest that fits under seven-eighths of RAM. That's why on a 64GB machine it chooses 46GB, not 52GB.
How It Works
The model is converted once into a .waste container: a JSON manifest, a resident trunk, and one expert bank per layer. Experts are stored using residual vector quantization — three stages of 256-entry codebooks over 8-dimensional vectors, giving 3.00 bits per weight. The matrix is never materialized; instead, the engine builds a table of partial dot products per token, making each expert row three table reads and two adds.
The trunk stays at 4 and 8 bits. A 3-bit trunk was tested but collapsed output quality. Reads bypass the page cache (F_NOCACHE on macOS, O_DIRECT on Linux) to avoid unrealistic cache behavior. Each record's header is checked on read, and a CRC32 check is available with --verify.
Code Example
Embedding WASTE is straightforward:
waste_cfg cfg;
waste_cfg_init(&cfg);
cfg.ram_budget_bytes = 46ULL << 30; /* hard ceiling */
waste_ctx *ctx;
if (waste_open("/path/to/k3.waste", &cfg, &ctx) != WASTE_OK) return 1;
waste_generate(ctx, ids, n, ¶ms, on_token, user);
waste_close(ctx);
Storage Requirements
Storage speed is not a detail. A token reads 17GB of experts. On the internal SSD (12.78 GB/s), the model streams; over USB (0.94 GB/s), a single token takes 13 seconds. The container must be on internal NVMe. You'll also need 982GB for the container plus 1.42TB of staging space for the original shards during conversion.
Why It Matters
WASTE demonstrates that frontier-scale models can run on consumer hardware. The author notes: "We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine." The engine is correct: logits agree with a PyTorch reference to 3.6e-06, and the vision tower matches to 2.3e-06.
The project is open-source, and the repository includes detailed efficiency notes (docs/EFFICIENCY.md) and a log of mistakes (docs/LEARNED.md). For developers without a terabyte of disk, the same engine runs Kimi-Linear-48B at 10.7 tok/s from a 19GB container.
What's Next
The author's next steps: "from here the question is engineering rather than feasibility." They've already identified that reads still take 55% of decode time, so faster NVMe or more RAM would help. They also invite counter-examples to their claim of being the first.
If you want to try it, start with Kimi-Linear-48B to test the waters before committing a terabyte. The performance is slow, but the fact that it runs at all is a milestone.
