• Stefy Lanza (nextime / spora )'s avatar
    Add --max-gpu-percent parameter for fine-grained GPU memory control · d62bdffb
    Stefy Lanza (nextime / spora ) authored
    This new parameter allows users to specify the exact percentage of GPU VRAM
    to use, overriding the offload-strategy. When specified, the model will:
    1. Use up to max-gpu-percent of VRAM
    2. Offload remaining weights to CPU RAM (--ram)
    3. Overflow to disk (--offload-dir) if RAM exhausted
    4. Automatically fallback in 5% steps if OOM occurs
    
    Example usage for RTX 3090 with Qwen3.5-35B-A3B:
      coderai --model Qwen/Qwen3.5-35B-A3B --max-gpu-percent 50 --ram 64
    
    This ensures MoE models with high VRAM requirements during generation
    can run without OOM by using CPU RAM as the primary offload target.
    d62bdffb
Name
Last commit
Last update
.gitignore Loading commit data...
LICENSE.md Loading commit data...
README.md Loading commit data...
build.sh Loading commit data...
coder Loading commit data...
coderai Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
requirements.txt~ Loading commit data...