-
Stefy Lanza (nextime / spora ) authored
lora_path and lora_model_dir have been in the model config for a long time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and vllm.py contained zero LoRA references, so configuring a LoRA on a text model silently did nothing. Now all three apply them. codai/models/text_loras.py answers the one question they share — which adapters does this model want, and where are they — across every config shape the field has grown (lora_path, a list, lora_model_dir, a `loras` list of dicts), de-duplicated so the same adapter is not applied twice. Each runtime then does its own thing: transformers PEFT load_adapter + set_adapters, several at once llama.cpp lora_path + lora_scale, ONE adapter and it must be GGUF-converted — a PEFT safetensors is reported and skipped rather than failing deep inside the load vLLM --enable-lora --lora-modules, with --max-lora-rank 64 because the default 16 refuses most trained adapters and blames the flag rather than the adapter QLoRA needs nothing extra: the quantisation is how the BASE model loads (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes the model's entry to the backend as `model_config`, which is how any of this reaches them. On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent with the request, because that API carries adapters per request. A text LoRA is part of how the model LOADS, so it must resolve when the pod starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a local path is reported and skipped instead of being baked into a launch command that would fail minutes later inside a pod nobody can inspect. A coderai pod gets the adapters in its seed and applies them itself. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
34d08890