Janas: Run LLMs larger than RAM by streaming experts from SSD
Janas-LLM is an open-source C tool that allows large language models (LLMs) exceeding a computer's RAM to run efficiently by streaming model "experts" from an NVMe SSD, demonstrating significantly faster performance than llama.cpp.
Intelligence analysis by Gemini 2.5 Flash
This experimental C project, Janas-LLM, enables users to run massive LLMs, such as 80-billion parameter models, on standard laptops with limited RAM by intelligently managing model components on disk. It boasts superior speed compared to existing solutions like llama.cpp, making advanced AI accessible on consumer hardware.
Imagine your computer has a small desk (RAM) but a huge library (SSD). Janas is like a super-smart librarian who quickly fetches only the exact book pages (model parts) you need right now, so you can read a giant book (LLM) even if it's too big for your desk. This makes big AI brains work on your regular computer much faster.
Analysis
Janas-LLM represents a notable advancement in making large language models more accessible to a broader user base. By implementing a system that streams model "experts" from an NVMe SSD, it effectively bypasses the traditional limitation of requiring LLM weights to fit entirely within a computer's RAM. This approach allows models significantly larger than available memory, such as the Qwen3-Next-80B model, to run on consumer-grade hardware like a laptop with just 32 GB of RAM. The project's pure C implementation, with no dependencies beyond the standard C library, underscores its focus on efficiency and minimal overhead, which is crucial for performance-sensitive applications like LLM inference.
Janas-LLM
Janas-LLM is positioned as a family of tools for artificial intelligence, with its first component specifically designed to run large mixture-of-experts language models. The core innovation lies in its ability to keep model weights on disk and stream only the necessary "experts" for each token from an NVMe SSD. This intelligent memory management ensures that frequently accessed components remain in memory, optimizing performance. The project's early development stage means its public API and command line interface are subject to frequent changes, though the .jns model file format (version 2) is designed for forward compatibility.
The performance benchmarks provided are compelling, showing Janas-LLM outperforming llama.cpp across all 13 compared models on the same laptop hardware. For instance, the Qwen3-Next-80B model, which is 48 GB and exceeds the laptop's 32 GB RAM, achieved 24 tokens per second (tok/s) writing speed without drafts, compared to llama.cpp's 2.8 tok/s. This substantial speed difference highlights the effectiveness of Janas's disk-streaming approach in practical inference scenarios, making it a potentially transformative tool for local LLM deployment.
Qwen3-Next-80B
The Qwen3-Next-80B model serves as a prime example of the kind of large-scale LLM that Janas-LLM aims to make accessible. With a size of 48.4 GB, it significantly exceeds the 32 GB RAM of the test laptop, making it impossible to run efficiently with conventional methods. Janas-LLM's ability to run this model at 24 tok/s for writing and 62 tok/s for reading a 1,024-token prompt demonstrates its capability to handle models that would otherwise be out of reach for many users. This performance is achieved by intelligently managing the model's components, streaming them from the NVMe SSD as needed.
The project also tested other models like Qwen3-Coder-Next, Qwen3.6-35B, and various Gemma-4 models, consistently showing superior performance over llama.cpp. The detailed comparison table illustrates that Janas-LLM's advantage is particularly pronounced for larger models that strain system memory. This focus on optimizing for models larger than RAM is a key differentiator, addressing a significant bottleneck in local LLM deployment and experimentation.
Maurizio "camauri" Cammalleri
The Janas project is designed and written by Maurizio "camauri" Cammalleri, highlighting the individual effort behind this innovative open-source tool. Cammalleri's work on Janas-LLM demonstrates a deep understanding of system-level optimization and memory management, crucial for achieving the reported performance gains. The project's experimental nature, as explicitly stated by the developer, indicates an ongoing process of refinement and improvement, with a clear warning about potential system instability due to its aggressive resource utilization.
Cammalleri's decision to release Janas as an open-source project, written in pure C, aligns with the ethos of community-driven development and transparency. While the tool is currently developed and tested on a specific machine (Debian 13 Linux, Intel Core Ultra 9 185H, integrated Arc GPU, 32 GB RAM, NVMe SSD), the provision of janas-bench allows users to measure performance on their own hardware. This commitment to open development and user-testing is vital for the project's future growth and broader adoption within the AI and open-source communities.
Key points
- Janas-LLM enables running LLMs larger than a computer's RAM by streaming "experts" from an NVMe SSD.
- It demonstrates significantly faster performance than `llama.cpp` across 13 compared models, including Qwen3-Next-80B.
- The project is written in pure C with no dependencies beyond the C library, supporting both CPU and GPU.
- It is currently in early experimental development, with API and command-line changes expected.
- Developed by Maurizio "camauri" Cammalleri, it is available for Linux x86-64 and Windows 10/11.
This technology could significantly lower the barrier to entry for running and experimenting with state-of-the-art LLMs, fostering innovation in local AI applications. It promises to make powerful AI models accessible to a broader range of developers and researchers who lack access to high-end hardware.
As an early-stage experiment, Janas-LLM carries risks, including potential system instability, such as pushing the desktop into swap or causing GPU driver resets. Its performance and reliability are currently untested on a wide variety of hardware, which could limit its practical adoption.