• Stefy Lanza (nextime / spora )'s avatar
    make an embedding request visible to the evictor · e227e7f5
    Stefy Lanza (nextime / spora ) authored
    An engine took SIGSEGV mid-sweep and every model resident on it died with it.
    _is_key_busy() is what stops eviction unloading a model under a live request —
    its docstring says so — and it reads two sources: the instance pool's ref counts
    and external_busy. An embedding request appeared in NEITHER. It takes no pool
    ref, and nothing on that path called external_busy, whose only caller in the
    whole codebase was the lipsync engine. To the evictor, an embedding model was
    permanently idle.
    
        12:27:41  Loading Qwen3.6-14B-A3B: need 9.3 GB, have 4.8 GB free — evicting
        12:27:41  [req] POST /v1/embeddings model=Qwen3-Embedding-4B
        12:27:42  Response status: 200
        12:27:42  On-request VRAM eviction: unloading ACTIVE model
                  'embedding:…Qwen3-Embedding-4B-Q4_K_M.gguf'  backend.model=None
        12:27:44  engine#1 exited (code -11)
    
    code -11 is a segfault, not the -9 OOM-kill everything else this week has been
    about: the native model was freed while a request was still dereferencing it.
    The 502s on /v1/embeddings that had been appearing for hours were this.
    
    The guard wraps the whole region from having the model to being done with it,
    not each inference call: there are three such functions across seven call sites
    and several early returns, and the window that must be protected is the whole of
    it. The key is request_model's `<model_type>:<resolved_name>`, which is exactly
    what the evictor logs and _is_key_busy looks up — marking busy under any other
    spelling would be a no-op that reads like a fix.
    
    A test asserts every inference call sits inside the guard. Its first version
    matched substrings and flagged `_may_embed_images` — a name-and-config
    capability check that runs before the model is even loaded — so it now matches
    exact names.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    e227e7f5
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pipelines Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
hf_cache.py Loading commit data...
main.py Loading commit data...
ocr_fleet_workers.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...