Swiftlet delivers the first native on-device run of an 80B-parameter MoE model on a Mac, and a 35B model on an iPhone. The open-source Swift + Metal runtime streams Mixture-of-Experts (MoE) weights from SSD on demand, keeping only the dense core resident in RAM. The result: Qwen3.6-35B-A3B runs in 2.6GB RAM, and Qwen3-Next-80B-A3B in just 4.3GB, with decode speeds of 7-11 tok/s and 4.5-5 tok/s respectively on an M5 Mac. The 35B also runs on an iPhone 17 in ~2.5GB RAM at ~1 tok/s — a first for a model of this class.

How It Works

These Qwen hybrid models activate only ~3B parameters per token. Each layer routes every token to 10 of 512 experts (80B) or 8 of 256 (35B). Swiftlet keeps the dense weights resident: attention, DeltaNet projections, routers, shared experts, and embeddings — about 1.3GB (35B) or 2.5GB (80B) at 4-bit. It then repacks the thousands of routed experts into fixed-stride blobs in a .qpack container, so fetching one expert is exactly one pread from SSD — no mmap, no page-cache thrash.

A bounded cache with LFU plus recency eviction holds hot experts. Cache size barely affects speed: measured hit rates of 43-70% at the same throughput, because Apple SSDs absorb the misses. The entire forward pass runs on Metal with runtime-compiled shaders, eliminating the need for a Metal toolchain at build time and enabling the same code to ship on iOS.

Seventy-five percent of layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so there's no growing KV cache for those layers at any context length.

Getting Started

Clone and build on a Mac:

git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release

Download a model (resumable from Hugging Face):

.build/release/swiftlet-repack \
--from-hf Leonickson/Qwen3.6-35B-A3B-qpack \
--output ~/models/qwen3.6-35b.qpack

Chat with it:

.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \
"Who wrote One Hundred Years of Solitude?" "What language did he write it in?"

Or run an OpenAI-compatible server:

.build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080

Four Ways to Use It

  • Swift package: Add SwiftletCore to any macOS/iOS app. Use SwiftletSession for chat with streaming deltas, conversation caching, sampling with repetition control, and memory-pressure handling.
  • CLI: swiftlet chat, swiftlet generate, swiftlet-repack for building containers from MLX checkpoints.
  • Server: swiftlet-server speaks the OpenAI chat-completions API on loopback.
  • iOS app: Priv AI embeds SwiftletCore. End users tap Download and chat on-device. The app is open source at leonickson1/localLLM.

Correctness

Every layer of the forward pass — Gated DeltaNet recurrence, gated GQA attention, sparse MoE routing — is validated against mlx-lm reference implementations with per-layer fixtures in f32 and int4 quantized form. Incremental decoding is verified against whole-sequence processing. Metal kernels are tested against the exact CPU reference, and fast/scalar GPU kernels produce identical outputs. Containers are byte-verifiable against source checkpoints. Streaming placement never changes model semantics: an expert answers identically from cache or disk.

Relationship to TurboFieldfare

TurboFieldfare proved the expert-streaming thesis for Gemma on Macs. Swiftlet adopts its design lessons: stream experts with pread into a bounded slot pool, evict with LFU + recency, pack at fixed stride, install by routing downloaded bytes directly into final container positions, and compile shaders at runtime. Everything else is built from scratch in ~10k lines of Swift/Metal, supporting the Qwen hybrid stack with Gated DeltaNet, gated GQA, and high-sparsity MoE with a shared expert. It adds MLX affine int4/int8 group quantization compute in Metal, byte-addressed kernels with 64-bit offsets, a cooperative simdgroup GEMV fast path, and explicit hazard management.

Honest Expectations

Only ~3B parameters are active per token. These models chat and write like large models but recall facts like small ones. The decode loop is dispatch-bound, not IO-bound, so there's clear headroom for kernel optimization. Current focus is on that speed.

Try It on Your Phone

The 35B runs on iPhone inside Priv AI (App Store). Open Settings → Experimental Models, download the model. It streams from storage and chats on-device with no server. The feature is in App Store review; build from source now if you want it immediately.

License and Availability

Apache 2.0. Model weights are downloaded separately and remain under their own terms (Qwen: Apache 2.0). See THIRD_PARTY_NOTICES.md.

Swiftlet is a significant step toward running frontier-scale models on consumer hardware. With its open-source codebase and validated correctness, it's a practical tool for developers building on-device AI applications. Clone the repo, try the 35B on your Mac, and watch for the iPhone update.