oLLM: SSD Offloading for Large-Context Inference
oLLM is a lightweight, open-source Python library that changes the hardware math for running very large models. Instead of demanding large amounts of expensive VRAM, it offloads both the model weights and the attention KV cache to fast SSD storage. That lets a single consumer-grade NVIDIA GPU run 80B+ parameter models with contexts up to 100,000 tokens — at the cost of speed. The whole technique is a deliberate trade: throughput for accessibility.
The core idea: move the bottleneck to storage
Conventional inference keeps weights and the KV cache in VRAM, so VRAM capacity is the limit. oLLM streams weights from SSD layer by layer and writes the KV cache to disk, shifting the bottleneck from GPU memory to storage I/O.
| Dimension | Traditional inference (e.g. vLLM) | oLLM |
|---|---|---|
| Primary storage | VRAM / host RAM | NVMe SSD / host RAM |
| KV cache location | VRAM | NVMe SSD |
| Model weights | Fully loaded into VRAM | Streamed from SSD, layer by layer |
| Key advantage | High throughput (tokens/sec) | Low VRAM use, long context |
| Main bottleneck | VRAM capacity | SSD read/write speed |
| Best for | Real-time, interactive apps | Offline, batch processing of large data |
How it works
oLLM leans on a few high-performance components to make offloading viable:
- PyTorch and Hugging Face Transformers provide the foundation for model loading and execution.
- FlashAttention-2 optimises the attention mechanism, cutting memory use and speeding up the parts of the model actively running on the GPU.
- GPUDirect Storage (via KvikIO/cuFile) is the critical piece: it lets the GPU read data directly from the NVMe SSD without staging it through CPU RAM first, which sharply reduces the latency of streaming weights and cache.
- No quantization. oLLM deliberately keeps native FP16/BF16 precision rather than dropping to 4-bit or 8-bit, preserving output quality at the cost of larger storage and transfer volumes.
The trade-off in numbers
- Low VRAM footprint. Usage stays low — typically 8–10 GB — which keeps it compatible with common GPUs like the RTX 3060 or 4070.
- Low throughput. Because most of the model lives on the SSD, generation is slow: an 80B model may produce only around 0.5 tokens per second, which rules out chatbots and other real-time use.
- Very long context. Writing the KV cache to disk means context length is bounded by SSD space, not VRAM — well beyond what the same hardware could otherwise hold.
When to use it
oLLM is a specialised tool, not a general-purpose inference server. It fits when:
- The work is offline or asynchronous — batch-processing large documents, summarising whole repositories, or deep analysis of long texts you can start and collect later.
- You need a very large model without A100/H100-class hardware, democratising access to state-of-the-art open models for researchers and hobbyists.
- Privacy is paramount, enabling fully local processing of large, sensitive datasets on consumer hardware.
Do not use oLLM for any user-facing application that needs a fast, interactive response.
Hardware requirements
- GPU. An NVIDIA Ampere, Ada, or Hopper card (RTX 30xx/40xx or newer) with at least 8GB VRAM.
- SSD. A very fast NVMe SSD is non-negotiable — the whole system’s speed is bounded by storage read/write.
- Host RAM. Enough system memory for the OS and the model’s non-offloaded components.
