• Stefy Lanza (nextime / spora )'s avatar
    let a model page set the precision it runs in · be14830d
    Stefy Lanza (nextime / spora ) authored
    `precision` was read by the loader and listed in main.py's _LOAD_AFFECTING, but
    it was in neither the admin save allowlist nor the model page, so the one
    setting that decides how much VRAM a card actually needs could not be
    configured anywhere. The dtype then came from a default derived from GPU
    detection -- and a card's precision is a property of the card, not of the
    hardware it happens to land on.
    
    The select sits with the quantization boxes because that is where it is read
    wrongly: 4-bit and 8-bit quantize the WEIGHTS, while the embeddings, lm_head,
    the norms, the activations and the KV cache all stay in this dtype, so a 4-bit
    model left in fp32 still pays fp32 for every one of them. The hint says so, and
    says that fp32 also rules out Flash Attention.
    
    "" is stored as null, so "not configured" stays distinguishable from a value
    and the loader's own default still applies.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZnLhydX9QqUcfBbqLu
    be14830d
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-acestep.txt Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-latentsync.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-musetalk.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...