• Stefy Lanza (nextime / spora )'s avatar
    text: LoRA and QLoRA on text models, locally and on RunPod · 34d08890
    Stefy Lanza (nextime / spora ) authored
    lora_path and lora_model_dir have been in the model config for a long
    time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and
    vllm.py contained zero LoRA references, so configuring a LoRA on a text
    model silently did nothing. Now all three apply them.
    
    codai/models/text_loras.py answers the one question they share — which
    adapters does this model want, and where are they — across every config
    shape the field has grown (lora_path, a list, lora_model_dir, a `loras`
    list of dicts), de-duplicated so the same adapter is not applied twice.
    Each runtime then does its own thing:
    
      transformers  PEFT load_adapter + set_adapters, several at once
      llama.cpp     lora_path + lora_scale, ONE adapter and it must be
                    GGUF-converted — a PEFT safetensors is reported and
                    skipped rather than failing deep inside the load
      vLLM          --enable-lora --lora-modules, with --max-lora-rank 64
                    because the default 16 refuses most trained adapters and
                    blames the flag rather than the adapter
    
    QLoRA needs nothing extra: the quantisation is how the BASE model loads
    (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes
    the model's entry to the backend as `model_config`, which is how any of
    this reaches them.
    
    On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent
    with the request, because that API carries adapters per request. A text
    LoRA is part of how the model LOADS, so it must resolve when the pod
    starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a
    local path is reported and skipped instead of being baked into a launch
    command that would fail minutes later inside a pod nobody can inspect. A
    coderai pod gets the adapters in its seed and applies them itself.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    34d08890
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...