• Stefy Lanza (nextime / spora )'s avatar
    a learned layer split must not outlive the KV settings it was learned under · 051cc5b2
    Stefy Lanza (nextime / spora ) authored
    measured_n_gpu_layers is reused as a cap so a reload jumps straight to the split
    that fit instead of rediscovering it through OOM retries, and measured_quant_sig
    is what stops such a measurement outliving its configuration. It covered
    quantization only.
    
    Setting swa_full=False on a gemma card cuts its KV reservation by about 19 GB —
    its 25 sliding layers stop reserving the full 178k context and hold their
    1024-token window instead, at no quality cost. The learned split of 18 of 30
    layers was measured with the old full-SWA KV, and nothing in the signature
    changed, so it would have survived and pinned the model to CPU on a card that
    can now hold nearly all of it. That is the layer-ratchet bug again, arriving
    through a setting the signature was not watching.
    
    n_ctx, swa_full, cache_type_k/v and kv_offload now form part of it. They are not
    "how big the weights are", which is what the docstring used to claim the hash
    was for, but they decide how much of the card is left for the weights and
    therefore how many layers fit. offload_kqv is read as a spelling of kv_offload
    because the loader accepts either, and a card written one way must not keep a
    measurement taken the other.
    
    The field list is deliberately narrow: over-invalidating discards every
    measurement and restores the OOM ladder on every load, which is the whole cost
    this mechanism exists to avoid. A test pins that both ways.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    051cc5b2
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-acestep.txt Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-latentsync.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-musetalk.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-parler.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements-xtts.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...