How RAMen works
Local AI gets frustrating on long work for two reasons. The model slows down as a conversation grows. And when you close the app, everything it had read is gone, so reopening a long chat means waiting while it reads the whole thing again. RAMen fixes both, on your own computer, in front of llama.cpp.
A replay of a real run at its real speed: Qwen3 32B on an RTX A6000, the same 87,099-token conversation, normal llama.cpp on the left (8.4 tokens per second) and RAMen on the right (20.5). Same answer.
1. Chats that come back in seconds
While a model reads a conversation it builds a working memory (the KV cache). RAMen saves each conversation's working memory to disk as you go and restores it the moment that conversation continues, even after a restart. Many chats and agents share a few live slots on the GPU; the rest wait on disk and come back on demand.
Real recording, real time, on a Mac mini M2 Pro with Qwen3 4B: a 70,082-token chat, restarted. Reading it from scratch took 12 minutes 47 seconds just before; RAMen restored it and wrote the first word after 3.7 seconds. A second take, with the Mac's file cache cleared first so the saved chat had to come from the SSD, wrote the first word after 3.0 seconds.
Saving the cache is not a new idea: llama.cpp has slot save and restore, and small tools automate it. How to do it by hand. RAMen does it for every session automatically, with checks that a saved cache matches the model and settings before it is reused, and it combines it with the speedup below.
2. Long chats that stay fast
Normally every new word the model writes reads the whole working memory, so a 100,000-token chat is several times slower than a short one. RAMen keeps a small summary of each block of the memory and, at each step, reads only the blocks that matter for the next word (about 8,000 tokens' worth), plus the start and the most recent part of the conversation. The idea comes from research on query-aware sparse attention (Quest, 2024). RAMen builds it into llama.cpp for NVIDIA and Apple GPUs. It switches on by itself only past the length where it is faster on your kind of machine (between 16K and about 43K tokens), so it is never slower.
Quality: when the selection covers the whole conversation, RAMen writes exactly the same text as normal attention. With selection on, it gave the same answers as full attention in 54 of 54 long-document tests (16K to 64K tokens), and a needle-in-a-haystack test passed at 32K and 64K.
Measured results
| Machine and model | Conversation | Cache | Normal llama.cpp | RAMen | Faster |
|---|---|---|---|---|---|
| RTX A6000, Qwen3 32B | 87K tokens | 8-bit | 8.4 tok/s | 20.4 tok/s | 2.4x |
| RTX A6000, Qwen3 32B | 56K tokens | 8-bit | 11.5 tok/s | 20.9 tok/s | 1.8x |
| RTX 4090, Qwen3 4B | 224K tokens | 8-bit | 31 tok/s | 88 tok/s | 2.9x |
| NVIDIA L4, Qwen3 4B | 128K tokens | 16-bit | 9.1 tok/s | 42.2 tok/s | 4.6x |
| NVIDIA L4, Qwen3 4B | 128K tokens | 8-bit | 8.5 tok/s | 35.1 tok/s | 4.1x |
| Mac mini M2 Pro 16 GB, Qwen3 4B | 81K tokens | 8-bit | 7.4 tok/s | 22.6 tok/s | 3.1x |
| Reopening a saved chat | Size | Reading it again | RAMen |
|---|---|---|---|
| Mac mini M2 Pro, Qwen3 4B | 69,864 tokens | 12.7 minutes | 3.7 seconds |
| NVIDIA L4, Qwen3 4B | 104,744 tokens | 220 seconds | 1.8 seconds |
| RTX 4090, Qwen3 4B | 197,860 tokens | 91 seconds | 5.6 seconds |
Writing speed (decode), same model, settings and cache type on both sides, November 2026. Below about 30K to 40K tokens a fast NVIDIA card gains little, and RAMen keeps normal attention there.
Limits, honestly
- Selective reading speeds up writing (decode), not the first read of a new long prompt.
- It does not stack with draft models (speculative decoding) yet; RAMen uses whichever is faster.
- Models that keep a small memory by design (for example sliding-window models) gain little speed. Session memory still helps them.
- RAMen is proprietary software with a free plan. It is built on llama.cpp (MIT license), whose license ships with it.
Try it
A fresh install on a Mac mini, recorded as it happened (the model download is shown 4x faster).
Mac and Linux: curl -fsSL https://getramenai.com/install.sh | bash
Windows (PowerShell): irm https://getramenai.com/install.ps1 | iex
Then ramen start. The free plan is free forever, and every install starts with 14 days of everything. See plans.