Run Kimi K3 Locally on M1 Max: 14.6s per Token with Deltafin

Deltafin enables running Moonshot AI's 1.56TB Mixture-of-Experts model Kimi K3 on a single M1 Max Mac. Full install requires 1.7TB disk, achieves 14.6s/token decode. Streaming mode uses 215GB but is network-bound at ~3 min/token.

4 min readJul 29, 2026

Run Kimi K3 Locally on M1 Max: 14.6s per Token with Deltafin

Deltafin: Running Kimi K3 on a Single M1 Max

Moonshot AI's Kimi K3 is a 1.56 trillion parameter Mixture-of-Experts model. Its sheer size (1.56 TB of MXFP4 weights) would normally require a datacenter. Deltafin, an open-source inference engine, makes it possible to run this model on a single workstation — even an aging M1 Max Mac.

The Key Insight: MoE Sparsity

K3 has 82,432 routed experts across 92 layers. Per token, the router selects only 16 experts per layer (top-16 of 896). That's roughly 1.8% of expert weights touched per token. The resident spine (attention, shared experts, embeddings) is ~114 GB (quantized to int8 ~60 GB) and is read sequentially from disk. This design makes local inference feasible despite the massive total size.

Installation and Setup

Deltafin requires Python 3.12+, a C/C++ compiler, and PyTorch. The setup is straightforward:

git clone https://github.com/gavamedia/deltafin.git
cd deltafin
python3 -m venv venv
./venv/bin/pip install torch numpy safetensors tiktoken ml_dtypes blobfile &#34;transformers==4.56.2&#34; einops tokenizers
python3 tools/build_native.py
./venv/bin/python tools/setup_k3.py --full

The build_native.py script compiles native libraries (.dylib or .so) with host-specific ISA flags (e.g., AVX2/FMA3 on x86-64-v3, NEON on aarch64). It validates symbols and ABI before installing.

Two Operating Modes

Mode	Disk Needed	Download Time	Speed (M1 Max)
`--full`	~1.7 TB	5–10 hours	14.6 s/token decode
`--stream`	~215 GB	~30 minutes	~3+ min/token (network bound)

The critical difference: every token reads 16 experts × 92 layers = 25.8 GB of expert data. From local NVMe that's ~4 seconds; over the network it's minutes. Streaming is a good trial mode; you can upgrade later with fetch_experts_all.py without reinstalling.

Performance Benchmarks

On a single M1 Max (10-core CPU, 32-core GPU, 64 GB RAM, internal NVMe) with full install, int8 spine, Metal MoE, fp32 numerics, greedy decoding, and tracing off:

Metric	Value
Prefill (5-token prompt)	28.0 s median (range 24.9–37.9 s)
Steady decode	0.0687 token/s (14.6 s/token)
3-token generation (model time)	56.5 s median
Fresh-process wall time	64.1 s median
Streaming decode	~3 min/token (network bound)

This is an early Apple Silicon machine — newer hardware with higher memory bandwidth should perform better. The maintainer notes this is a "conservative reference point."

Usage

Deltafin provides a CLI and an OpenAI-compatible API server.

CLI example:

./venv/bin/python tools/kimi_run.py --chat --prompt &#34;What are the three largest moons of Saturn?&#34;

API server:

./venv/bin/python tools/serve_openai.py --port 8000

Then use any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url=&#34;http://127.0.0.1:8000/v1&#34;, api_key=&#34;none&#34;)
r = client.chat.completions.create(
    model=&#34;deltafin-kimi-k3&#34;,
    messages=[{&#34;role&#34;: &#34;user&#34;, &#34;content&#34;: &#34;Hello!&#34;}])
print(r.choices[0].message.content)

Caveats: answers take minutes; set client timeouts to hours. Streaming installs are much slower for chat (prefill touches many experts). Only greedy decoding is supported (temperature/top_p ignored). One request at a time.

Configuration

Deltafin auto-detects the best device (MPS, CUDA, CPU) and uses int8 spine if built. Override via environment variables:

K3_DEV: set device (mps, cuda, cpu)
K3_MOE: MoE backend (metal, cpu)
K3_SPINE: weight type (int8, bf16)
K3_MOE_TOP_K: number of routed experts (default 16)
K3_CPU_MOE_BATCH: CPU worker ring size

Why This Matters

Running a 1.56T parameter model on a single consumer machine is a milestone. The Mixture-of-Experts architecture, combined with aggressive quantization (MXFP4) and sparse activation, enables local inference that was previously impossible. For developers, this means experimenting with state-of-the-art LLMs without cloud costs, though patience is required: 0.07 tokens/second is not interactive.

Next Steps

If you have 1.7 TB free disk space and a recent Mac or Linux machine, try the full install. For a quick test, use streaming mode. Monitor the GitHub repo for updates — the M1 Max benchmark already shows 87× improvement in prefill since the first version.

Editor's Take

I've been watching the local LLM space since llama.cpp, and Deltafin is a different beast. The 1.7 TB disk requirement is brutal, but the fact that it works at all on an M1 Max is impressive. I tried the streaming mode and waited 3 minutes per token — that's not useful for chat, but for batch processing or research it's a start. I think the real breakthrough will come when someone ports the MXFP4 MoE kernel to CUDA, which the maintainer explicitly says doesn't exist yet.

— DevDigest Editorial

Key Takeaways

•Use full install for usable speed (14.6 s/token) if you have 1.7 TB free; streaming is only for testing.
•Set K3_MOE_TOP_K lower than 16 to reduce expert I/O at the cost of quality — experiment with your use case.
•The int8 spine conversion halves per-token I/O with no quality loss; always enable it.
•For API usage, set client timeouts to hours and use max_tokens to limit generation length.

Why It Matters

This is the first time a 1.56 trillion parameter model can run on a single consumer-grade machine. For developers, it means experimenting with frontier AI locally without cloud costs, though at extremely slow speeds. The technical approach (MXFP4 quantization, sparse expert activation) is instructive for anyone building large model inference pipelines.

#apple-silicon#mixture of experts#MXFP4#Kimi K3#Deltafin

Get the weekly digest

Every Sunday - top tech stories, industry breakthroughs, and developer tools delivered to your inbox.

No spam, unsubscribe anytime.