• Stefy Lanza (nextime / spora )'s avatar
    a GGUF's weights total is the file; its placement is not ours to invent · ed02d1dd
    Stefy Lanza (nextime / spora ) authored
    Follow-up: the footprint fix earlier today addressed one of two sources of the
    same wrong attribution, and the live log showed the other one an hour later.
    
        footprint: 8.48 GB total (weights 7.71 GB (0.00 on GPU, 7.71 on host), …)
        Measured footprint for 'Gemma-4-14B-A4B-Heretic.i1-IQ4_XS': 8.48 GB total
          (… of which 7.71 GB was offloaded to host RAM this load; n_gpu_layers=-1)
    
    That model was loaded with `n_gpu_layers: all` and was holding 10.1 GB of VRAM.
    `measure()` has no torch object to walk for a GGUF, so it takes the weights
    total from the file size -- correctly, a GGUF is loaded as-is -- and banked it in
    `weights_cpu_gb`, which keeps the TOTAL right and asserts a placement nobody
    measured. "0.00 on GPU" is the exact opposite of the truth for a fully resident
    model, and that line is what an operator reads while chasing a CPU-spill
    problem. I read it that way myself this evening and called the earlier mismatch
    cosmetic.
    
    `weights_split_known` now records that the split is unknown, `describe()` says
    "placement not measured here" instead of inventing one, and the manager
    distinguishes the three different things that reach `measured_ram`: a per-module
    measurement (genuinely on the host), a GGUF's unsplit total, and the process's
    RSS growth. Only the first is reported as an offload, and only the first puts a
    size on the Tasks page -- which had been labelling fully-GPU-resident GGUFs
    "CPU offload 7.71 GB".
    
    The arithmetic is untouched in both paths. The total is what sizing and eviction
    consume, it was right, and under-reporting a footprint is how eviction stops
    freeing room and the forward OOMs.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    ed02d1dd
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pipelines Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
hf_cache.py Loading commit data...
main.py Loading commit data...
ocr_fleet_workers.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...