Red Lite dev61 — the 24 GiB Mac without a raised GPU limit: agents and the IQ3_XXS file from the SSD

Status: done. Measured on the Apple M4 Pro 24 GiB (macOS 26.7) at the default GPU limit, 2026-10-04.

Why#

A Red Lite user who never raises the GPU limit (it needs sudo) runs with the bounded expert cache.

• What stays in memory: the dense weights and a 4 GiB LRU of experts.
• What the SSD serves: every other expert a token needs is read from the file.

dev60 measured coding agents only with every expert resident. This milestone measures that default user, with pi installed on the same Mac. It also tries the 29.6 GiB IQ3_XXS file, which cannot be resident on 24 GiB at all.

pi on the 24 GiB Mac#

• Install and configure:
  • brew install node;
  • npm install -g @earendil-works/pi-coding-agent (pi 1.0.2);
  • redlite setup-pi --port 8091, which writes the redlite provider into ~/.pi/agent/models.json, keeps other providers and keeps the file private (0600; it may hold other providers' API keys).
• Server: redlite serve --native --context 32768 --port 8091. At the default GPU limit the planner picks the 4 GiB cache with cache-aware routing. The engine is ready in 0.1 s: no experts are preloaded.
• Run: scripts/dev/agent_eval.py --agent-dir ~/.pi/agent, with the five tasks of dev60, three runs per model, agent and server on the same Mac.

• The tasks pass as often from the SSD as resident: 30 of 30. With a 4 GiB cache the server uses about 5 GiB instead of 18–20 GiB, so the agent and other applications have room.
• They take longer: +20 % (F2) and +51 % (Qwen3-Coder-Next) over the five tasks. Most of it is in the multi-step ones: rename 29 → 43 s and 19 → 40 s.
• The cost of the resident runs: a raised GPU limit and 12–20 s of preloading at every server start.

IQ3_XXS from the SSD#

Bartowski's IQ3_XXS (31.7 GB, 29.6 GiB) is the file 48 GiB Macs keep resident: perplexity 14.29, against 15.38 for F2 and 16.47 for Bartowski's IQ2_XXS. On a 24 GiB Mac it cannot be resident, but the bounded cache streams it.

Decode. small_mac_ab.sh measured the six cache-trace prompts × 256 tokens, with cache-aware routing (λ 0.5, the product default with a bounded cache) and 30 s of cooling before each prompt.

Prompt ingestion (bench_m4.sh --only prefill4, 4 GiB cache, chunks of 2048, median of 2):

• The better file costs only 14 % of decode speed.
  • Reads: IQ3_XXS reads 2.6× more from the SSD per token (107 MiB), yet decodes at 29.3 tok/s against 34.0.
  • Why: as in dev51, the bounded-cache path is bound by its per-layer CPU-GPU round trip, not by the reads.
  • Prompts: long prompts amortize the loads (−9 % at 8,192 tokens); short ones cost more (−41 % at 1,100).
• A bigger cache does not help. 8 or 12 GiB cut the reads 2.6–3.8× but decode at 27.9–28.1 tok/s, slightly slower than 4 GiB. The 4 GiB default stays.
• Memory: swap did not move during the runs (751 MB in use throughout, from before).

What a 24 GiB user can choose (M4 Pro):

redlite chat keeps choosing F2 on 24 GiB; IQ3_XXS is a choice: redlite download 48gb, then redlite chat PATH/Qwen_Qwen3-Next-80B-A3B-Instruct-IQ3_XXS.gguf.

Scope boundary#

• Same five tasks as dev60, three runs each: a check that the loop works on the default configuration, not a coding benchmark.
• Agent runs: 759 MB of swap in use after both, not separated from earlier resident runs on the same boot.
• IQ3_XXS quality is the M4 Max measurement (exact routing). With a bounded cache the cache-aware routing changes some expert choices. dev46 found no perplexity change for IQ2_XXS; IQ3_XXS with the bias was not measured.
• IQ3_XXS with the coding agent was not run.
• Single machine: only the M4 Pro; single runs of six prompts per cache size.