• Stefy Lanza (nextime / spora )'s avatar
    text: LoRA and QLoRA on text models, locally and on RunPod · 34d08890
    Stefy Lanza (nextime / spora ) authored
    lora_path and lora_model_dir have been in the model config for a long
    time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and
    vllm.py contained zero LoRA references, so configuring a LoRA on a text
    model silently did nothing. Now all three apply them.
    
    codai/models/text_loras.py answers the one question they share — which
    adapters does this model want, and where are they — across every config
    shape the field has grown (lora_path, a list, lora_model_dir, a `loras`
    list of dicts), de-duplicated so the same adapter is not applied twice.
    Each runtime then does its own thing:
    
      transformers  PEFT load_adapter + set_adapters, several at once
      llama.cpp     lora_path + lora_scale, ONE adapter and it must be
                    GGUF-converted — a PEFT safetensors is reported and
                    skipped rather than failing deep inside the load
      vLLM          --enable-lora --lora-modules, with --max-lora-rank 64
                    because the default 16 refuses most trained adapters and
                    blames the flag rather than the adapter
    
    QLoRA needs nothing extra: the quantisation is how the BASE model loads
    (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes
    the model's entry to the backend as `model_config`, which is how any of
    this reaches them.
    
    On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent
    with the request, because that API carries adapters per request. A text
    LoRA is part of how the model LOADS, so it must resolve when the pod
    starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a
    local path is reported and skipped instead of being baked into a launch
    command that would fail minutes later inside a pod nobody can inspect. A
    coderai pod gets the adapters in its seed and applies them itself.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    34d08890
Name
Last commit
Last update
..
archive.html Loading commit data...
base.html Loading commit data...
change_password.html Loading commit data...
chat.html Loading commit data...
dashboard.html Loading commit data...
login.html Loading commit data...
models.html Loading commit data...
runpod.html Loading commit data...
settings.html Loading commit data...
tasks.html Loading commit data...
tokens.html Loading commit data...
users.html Loading commit data...