Cactus Hybrid: Gemma 4 Model Knows When It's Wrong, Routes to Bigger LLM

Cactus Hybrid: On-Device Confidence Scoring for Gemma 4

Cactus has released a post-trained version of Google's Gemma 4 E2B (the smallest Gemma model) that includes a confidence probe embedded directly in the checkpoint. The probe assigns a score between 0 and 1 to every generated answer, returned as structured data alongside the response — never parsed from text. This allows developers to implement a simple routing policy: if confidence < 0.85, hand off to a larger model.

Benchmarks: Matching Flash-Lite with 15-35% Handoff

According to the GitHub release, the Gemma 4 E2B Hybrid matches Gemini 3.1 Flash-Lite on most benchmarks while routing only 15-35% of queries to the larger model. The exact handoff percentage depends on the benchmark and quantization level:

Benchmark	Handoff to match Flash-Lite (FP16)	At 4-bit	At 3-bit
ChartQA	15-20%	25-30%	40-50%
MMBench	30-35%	40-45%	50-55%
LibriSpeech	25-30%	35-40%	55-65%
GigaSpeech	30-35%	40-45%	50-55%
MMAU	30-35%	35-40%	50-55%
MMLU-Pro	45-55%	~90%	n/a

Note: Quantization quality is measured on Cactus Quants, which performs well at uniform quantization. Developers are encouraged to benchmark independently for Unsloth, GGUF, and MLX quantization.

Confidence Probe Quality (AUROC)

The probe's ability to distinguish correct from incorrect answers is measured by Area Under the Receiver Operating Characteristic curve (AUROC). Higher is better; 0.5 is random, 1.0 is perfect. The Cactus Hybrid achieves a mean AUROC of 0.814 across text, vision, and audio benchmarks, compared to 0.549 for token entropy baselines:

Benchmark	Modality	Cactus Hybrid	Token Entropy
MMLU	text MCQ	0.770	0.697
MMLU-Pro	text MCQ	0.771	0.692
ARC-Easy	text MCQ	0.888	0.655
ARC-Challenge	text MCQ	0.834	0.646
GSM8K (3-shot)	text gen	0.782	0.731
MMBench-EN-Dev	vision MCQ	0.840	0.435
ChartQA	vision QA	0.779	0.615
DocVQA	vision QA	0.781	0.512
MMAU	audio MCQ	0.789	0.517
GigaSpeech	audio	0.876	0.343
Earnings-22	audio	0.839	0.323
LibriSpeech	audio	0.822	0.427
Mean		0.814	0.549

Notably, the probe was trained on zero audio data yet achieves 0.79-0.88 AUROC on four audio benchmarks (two transcription, one audio MCQ, one out-of-domain transcription). This suggests the probe reads a modality-independent correctness signal from the hidden state, not memorized patterns.

Code Examples

The model is available on Hugging Face as Cactus-Compute/gemma-4-E2B-it (Cactus API), Cactus-Compute/gemma-4-e2b-it-hybrid-mlx (MLX), Cactus-Compute/gemma-4-e2b-it-hybrid (Transformers), and Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4_K_M (llama.cpp).

Cactus API (Python):

# pip install cactus-compute
import json
from cactus.bindings.cactus import cactus_complete, cactus_init
from cactus.cli.download import download_bundle

lm = cactus_init(str(download_bundle("Cactus-Compute/gemma-4-E2B-it")))
result = cactus_complete(
    lm,
    [{"role": "user", "content": "What is the capital of France?"}],
    json.dumps({"max_tokens": 512, "auto_handoff": False}),
    None,
    lambda *_: None,
)
print(result["response"].strip())
print("confidence:", result["confidence"])

MLX (Apple Silicon):

# pip install mlx-lm
import re
from mlx_lm import load, generate

model, tokenizer = load(
    "Cactus-Compute/gemma-4-e2b-it-hybrid-mlx",
    tokenizer_config={"trust_remote_code": True},
)
messages = [{"role": "user", "content": "What is the capital of France?"}]
answer = generate(
    model,
    tokenizer,
    prompt=tokenizer.apply_chat_template(messages, add_generation_prompt=True),
    max_tokens=512,
)
answer = re.split(r"<\|?channel\|?>", answer)[-1]
answer = re.sub(r"^(thought|final)\b\s*", "", answer).strip()
print(answer)
print("confidence:", model.last_confidence)

Transformers:

# pip install "transformers>=5.5.4,<5.6" torch
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Cactus-Compute/gemma-4-e2b-it-hybrid"
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="auto").to(device)

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(device)
out = model.generate(**inputs, return_confidence=True, max_new_tokens=512)
print(tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
print("confidence:", out.confidence)

llama.cpp:

# Build patched server
git clone https://github.com/cactus-compute/cactus-hybrid && cd cactus-hybrid
./patches/llama.cpp/install.sh && rehash
# Serve
llama-server -hf Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4_K_M --jinja
# Query
curl -s http://localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":512}' \
  | jq '{answer: .choices[0].message.content, confidence}'

Important: Transformers Caveat

When using Transformers, load the model with an explicit .to(device), not device_map="auto". The probe scores generations outside the module forward() path, so weights that accelerate offloads (left on the meta device) crash the confidence read.

Licensing

The model is MIT-licensed. Gemma model use is subject to the Gemma terms.

Why This Matters

For developers deploying small models on-device (edge, mobile, privacy-sensitive apps), the trade-off between speed and accuracy is constant. Cactus Hybrid gives you a principled way to keep 65-85% of queries on-device while matching a much larger model's accuracy. The confidence probe is baked into the checkpoint, not an external classifier, so it's zero-latency and private.

Next Steps

1. Download the model from Hugging Face (Cactus-Compute/gemma-4-e2b-it-hybrid).
2. Integrate the confidence-based routing logic into your pipeline.
3. Benchmark on your own data to determine the optimal handoff threshold (the default 0.85 is a starting point).