• Stefy Lanza (nextime / spora )'s avatar
    fix: GME vision embeddings 500 on large images (KV context overflow) · e362e56c
    Stefy Lanza (nextime / spora ) authored
    POST /v1/embeddings to the GME-Qwen2-VL GGUF embedder returned HTTP 500 for
    large photos. The qwen2-vl vision tower turns a big image into >n_ctx image
    tokens (observed batches of 1012–2048), and mtmd's decode then fails —
    "decode: failed to find a memory slot for batch of size N" / "failed to eval
    chunk 1" — because the image tokens plus the chat-prompt frame exceed the
    model's KV context. Small images (few hundred tokens) were unaffected.
    
    mtmd's own image_max_tokens budget is NOT honoured by this llama.cpp build
    (2048-token batches slipped through), so add a hard, Python-side backstop:
    _resize_image_to_token_budget() downscales any oversize PIL image (aspect
    preserved, 28px patches) to at most n_ctx-128 tokens BEFORE it reaches mtmd,
    so a request can never overflow the context regardless of input size. The
    budget derives from the live n_ctx, so it self-adjusts to any model config.
    
    Pairs with raising the GME model's n_ctx (instance models.json) so a
    full-size image (~2048 tokens) fits comfortably; n_batch/n_ubatch follow
    n_ctx in the loader.
    
    Verified on the radeon Vulkan embedder: 336² / 896² / 1400² / 1680² images
    all return 200 (the 1400²/1680² are logged resizing to ~2400 tokens to fit
    the 2432 budget); previously the ≥1024-token cases 500'd.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01LoSpEthysqmseCc6Geizty
    e362e56c
embeddings.py 53.7 KB