• Stefy Lanza (nextime / spora )'s avatar
    text: LoRA and QLoRA on text models, locally and on RunPod · 34d08890
    Stefy Lanza (nextime / spora ) authored
    lora_path and lora_model_dir have been in the model config for a long
    time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and
    vllm.py contained zero LoRA references, so configuring a LoRA on a text
    model silently did nothing. Now all three apply them.
    
    codai/models/text_loras.py answers the one question they share — which
    adapters does this model want, and where are they — across every config
    shape the field has grown (lora_path, a list, lora_model_dir, a `loras`
    list of dicts), de-duplicated so the same adapter is not applied twice.
    Each runtime then does its own thing:
    
      transformers  PEFT load_adapter + set_adapters, several at once
      llama.cpp     lora_path + lora_scale, ONE adapter and it must be
                    GGUF-converted — a PEFT safetensors is reported and
                    skipped rather than failing deep inside the load
      vLLM          --enable-lora --lora-modules, with --max-lora-rank 64
                    because the default 16 refuses most trained adapters and
                    blames the flag rather than the adapter
    
    QLoRA needs nothing extra: the quantisation is how the BASE model loads
    (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes
    the model's entry to the backend as `model_config`, which is how any of
    this reaches them.
    
    On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent
    with the request, because that API carries adapters per request. A text
    LoRA is part of how the model LOADS, so it must resolve when the pod
    starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a
    local path is reported and skipped instead of being baked into a launch
    command that would fail minutes later inside a pod nobody can inspect. A
    coderai pod gets the adapters in its seed and applies them itself.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    34d08890
remote-execution.md 21.5 KB