Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

Phi-4-Mini is a 3.8B decoder-only language model: 32 layers, hidden size 3,072, grouped-query attention with 24 query heads sharing 8 key-value heads, a 200,064-token vocabulary with tied input and output embeddings, and 128K context through LongRoPE. It was trained on 5 trillion tokens, with a better quality classifier than its predecessor, more instruction-based math and code data and Phi-4's synthetic data, then post-trained with more function calling, summarization and instruction-following data.

Phi-4-Multimodal (5.6B) adds vision and speech without touching the language model. The base stays frozen. Vision gets a SigLIP-400M encoder and projector (440M parameters) and a vision LoRA (370M); audio gets a conformer encoder and projector (460M) and an audio LoRA of rank 320 (460M). The authors call this a mixture of LoRAs: modality-specific adapters selected by input type, so text-only requests run the original model unchanged.

Phi-4-Mini averages 64.9 over 22 language benchmarks and 49.0 over 9 coding benchmarks, above every 3B model compared and some 8B models. Phi-4-Multimodal leads the OpenASR leaderboard at the time with 6.14 word error rate and scores 72.0 averaged over 15 vision benchmarks. An experimental reasoning version, trained on about 60 billion tokens of chain-of-thought from larger models and then fine-tuned and preference-trained, reaches 50.0 on AIME 2024; it was not released with the others.

Customers ask for models that run on their hardware: laptops for field staff, an air-gapped server, a phone app that cannot send audio to the cloud. Phi-4-Mini is a reference point for what fits in that envelope, about 7.6 GB of weights in bf16, and this report gives the numbers to plan around.

One model, three inputs, no text regression. Because the language model is frozen and modalities are LoRAs, a customer can deploy one checkpoint for chat, image questions and transcription. Text answers come from the same weights as Phi-4-Mini.

Speech is strong for transcription and translation, weaker for spoken questions. It leads OpenASR and beats Whisper V3 on all three ASR test sets reported. But when the question itself is spoken (spoken MMMLU) it scores 38.5 against 72.3 for Gemini 2.0 Flash; the report suggests it was tuned for speech understanding while Gemini and GPT-4o may be tuned for chat. Transcribe, then answer in text, if knowledge matters.

Long context is mostly a memory question. The KV cache is 128 KiB per token in bf16 even with grouped-query attention. A 32K-token session needs 4 GiB, and 128K needs 16 GiB, twice the weights. Audio arrives at 750 tokens a minute, so a 30-minute recording is 22,500 tokens before any prompt.