1. 05 Mar, 2026 1 commit
    • Stefy Lanza (nextime / spora )'s avatar
      Add --max-gpu-percent parameter for fine-grained GPU memory control · d62bdffb
      Stefy Lanza (nextime / spora ) authored
      This new parameter allows users to specify the exact percentage of GPU VRAM
      to use, overriding the offload-strategy. When specified, the model will:
      1. Use up to max-gpu-percent of VRAM
      2. Offload remaining weights to CPU RAM (--ram)
      3. Overflow to disk (--offload-dir) if RAM exhausted
      4. Automatically fallback in 5% steps if OOM occurs
      
      Example usage for RTX 3090 with Qwen3.5-35B-A3B:
        coderai --model Qwen/Qwen3.5-35B-A3B --max-gpu-percent 50 --ram 64
      
      This ensures MoE models with high VRAM requirements during generation
      can run without OOM by using CPU RAM as the primary offload target.
      d62bdffb
  2. 01 Mar, 2026 35 commits
  3. 28 Feb, 2026 4 commits