• Stefy Lanza (nextime / spora )'s avatar
    let a model page set the precision it runs in · be14830d
    Stefy Lanza (nextime / spora ) authored
    `precision` was read by the loader and listed in main.py's _LOAD_AFFECTING, but
    it was in neither the admin save allowlist nor the model page, so the one
    setting that decides how much VRAM a card actually needs could not be
    configured anywhere. The dtype then came from a default derived from GPU
    detection -- and a card's precision is a property of the card, not of the
    hardware it happens to land on.
    
    The select sits with the quantization boxes because that is where it is read
    wrongly: 4-bit and 8-bit quantize the WEIGHTS, while the embeddings, lm_head,
    the norms, the activations and the KV cache all stay in this dtype, so a 4-bit
    model left in fp32 still pays fp32 for every one of them. The hint says so, and
    says that fp32 also rules out Flash Attention.
    
    "" is stored as null, so "not configured" stays distinguishable from a value
    and the loader's own default still applies.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZnLhydX9QqUcfBbqLu
    be14830d
Name
Last commit
Last update
..
static Loading commit data...
templates Loading commit data...
__init__.py Loading commit data...
auth.py Loading commit data...
download_worker.py Loading commit data...
routes.py Loading commit data...