📱 A 35B MoE model running on a phone at over 20 tokens/s.

Meet Lookahead Expert Scheduling (LES), the inference technology behind MiniCPM-V 4.7’s efficient on-device performance.

🧠 The challenge: 35B total parameters, 3B activated per token, and limited device memory. Keeping experts in flash storage makes the model fit, but loading their weights can leave computation waiting.

LES co-optimizes expert loading, compute scheduling, and weight access:

⚡ Expert prefetch and compute scheduling: Cross-layer routing prediction prefetches experts the next layer is likely to need. Experts selected by the actual router that are already loaded compute first, while the remaining required experts load in parallel. Prediction only guides prefetch and scheduling: every required expert is still loaded and used. No additional training is needed.

💾 Weight access optimization: Repack expert weights into contiguous storage to reduce scattered reads, and apply precise layer-wise mixed-precision quantization to reduce weight storage and transfer costs while preserving model capability.

We also adapted DSpark speculative sampling to work alongside LES in the on-device inference pipeline.

In our iPhone 18 Pro Max tests, the full system reaches 20.51 tokens/s, up from a 3.38 tokens/s on-demand loading baseline: 6.07× faster.
🔗 https://x.com/OpenBMB/status/2112529266537639962?s=20

The original model architecture and expert routing stay intact. Smarter scheduling makes larger models practical on-device.

👉 MiniCPM-V GitHub: http://github.com/OpenBMB/MiniCPM-V
🤗 Model info: http://huggingface.co/collections/openbmb/minicpm-v-47
💻 Technical Details: https://github.com/tc-mb/llama.cpp-omni/blob/master/docs/les/lookahead-expert-scheduling.md

#MiniCPM #OnDeviceAI #OpenSource