LFM2.5 Q4_0 Checkpoints from Quantization-Aware Distillation
LiquidAI has released new QAD Q4_0 GGUF checkpoints for its LFM2.5 models, enabling developers to run these models with significantly reduced memory and increased speed on edge devices without substantial quality degradation. The new checkpoints recover 97% of the accurac…
Intelligence analysis by Gemini 2.5 Flash
The company LiquidAI has introduced updated 4-bit quantized models for its LFM2.5 series, leveraging a technique called Quantization-Aware Distillation (QAD). This method allows the smaller, more efficient models to retain nearly all the performance of their larger, unquantized counterparts, making them highly suitable for deployment on resource-constrained hardware like mobile phones…
Imagine you have a super-smart robot brain (an AI model) that's really big and needs a huge computer to run. LiquidAI found a clever way to make these robot brains much smaller, like shrinking a big textbook into a tiny pocket guide, so they can run on your phone or a small computer. The trick is that they teach the smaller brain how to be almost as smart as the big one, so you don't lose much of its amazing abilities, making powerful AI available on everyday gadgets.
Analysis
LiquidAI's latest release of QAD Q4_0 GGUF checkpoints marks a significant step forward in making large language models (LLMs) more practical for real-world deployment on edge devices. The core innovation lies in Quantization-Aware Distillation (QAD), a technique that trains a smaller, quantized 'student' model by distilling knowledge from a high-precision 'teacher' model. This approach ensures that the student model, despite its reduced size and computational demands, maintains a high level of accuracy.
Quantization-Aware Distillation
Quantization is a process that reduces the precision of a model's weights, typically from 16-bit (BF16 or F16) to 4-bit (Q4_0), thereby shrinking its memory footprint and speeding up inference. However, this reduction in precision often comes at the cost of model quality. LiquidAI's QAD method directly tackles this by integrating the quantization process into the training loop, allowing the model to learn to perform well even with lower precision weights. This is distinct from post-training quantization (PTQ), which quantizes an already trained model and often leads to more significant quality drops.
97% Recovery
The effectiveness of the QAD approach is evident in the benchmark results. Across a suite of evaluations covering reasoning, instruction-following, tool use, and agentic capabilities—including GPQA Diamond, MMLU-Pro, IFEval, and BFCLv4—the QAD Q4_0 checkpoints consistently recovered approximately 97% of the BF16 baseline performance. Specifically, the four LFM2.5 models (230M, 350M, 1.2B-Instruct, and 2.6B) retained between 96.5% and 97.4% of their full-precision accuracy. This near-complete recovery is a substantial improvement over traditional quantization methods, which typically incur a more noticeable performance penalty.
Edge Hardware
Beyond accuracy, the QAD Q4_0 checkpoints deliver tangible benefits in terms of speed and size on diverse edge hardware. Measurements on devices like the MacBook Pro, NucBox EVO-X2, Samsung Galaxy S26 Ultra, and Raspberry Pi 5 demonstrated that these quantized models maintain the low memory footprint and high throughput characteristic of Q4_0 GGUFs. The 230M and 350M QAD Q4_0 checkpoints achieved Q5_K_M quality with 4-33% higher decode throughput, while the 1.2B and 2.6B models matched Q4_K_M quality with 3-14% higher throughput. This performance on both GPU and Arm CPU inference targets underscores their versatility for deployment across a wide range of consumer and embedded devices.
Key points
- LiquidAI released QAD Q4_0 GGUF checkpoints for LFM2.5 models (230M, 350M, 1.2B-Instruct, 2.6B).
- These checkpoints enable running LFM2.5 models at Q4_0 memory and speed without significant quality loss.
- The Quantization-Aware Distillation (QAD) method recovers approximately 97% of the BF16 average accuracy lost to quantization.
- Benchmarks show substantial improvement over post-training quantization (PTQ) GGUFs across various AI capabilities.
- The QAD Q4_0 checkpoints demonstrate improved decode throughput on diverse edge hardware, including mobile and single-board computers.
This breakthrough could significantly accelerate the deployment of advanced AI capabilities directly onto consumer devices, fostering innovation in areas like on-device AI assistants, real-time language processing, and intelligent automation without relying on cloud infrastructure. It promises to make powerful AI more accessible, private, and responsive for everyday users.
While QAD significantly mitigates quality loss, the remaining 3% gap in performance compared to BF16 baselines could still be a limiting factor for highly sensitive or mission-critical AI applications where absolute precision is paramount. Furthermore, the process of quantization-aware distillation itself introduces an additional step in the model training pipeline, potentially increasing development time and computational resources required to produce these optimized checkpoints.
