• Stefy Lanza (nextime / spora )'s avatar
    fix: GME vision embeddings 500 on large images (KV context overflow) · e362e56c
    Stefy Lanza (nextime / spora ) authored
    POST /v1/embeddings to the GME-Qwen2-VL GGUF embedder returned HTTP 500 for
    large photos. The qwen2-vl vision tower turns a big image into >n_ctx image
    tokens (observed batches of 1012–2048), and mtmd's decode then fails —
    "decode: failed to find a memory slot for batch of size N" / "failed to eval
    chunk 1" — because the image tokens plus the chat-prompt frame exceed the
    model's KV context. Small images (few hundred tokens) were unaffected.
    
    mtmd's own image_max_tokens budget is NOT honoured by this llama.cpp build
    (2048-token batches slipped through), so add a hard, Python-side backstop:
    _resize_image_to_token_budget() downscales any oversize PIL image (aspect
    preserved, 28px patches) to at most n_ctx-128 tokens BEFORE it reaches mtmd,
    so a request can never overflow the context regardless of input size. The
    budget derives from the live n_ctx, so it self-adjusts to any model config.
    
    Pairs with raising the GME model's n_ctx (instance models.json) so a
    full-size image (~2048 tokens) fits comfortably; n_batch/n_ubatch follow
    n_ctx in the loader.
    
    Verified on the radeon Vulkan embedder: 336² / 896² / 1400² / 1680² images
    all return 200 (the 1400²/1680² are logged resizing to ~2400 tokens to fit
    the 2432 budget); previously the ≥1024-token cases 500'd.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01LoSpEthysqmseCc6Geizty
    e362e56c
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
characters.py Loading commit data...
custom_pipelines.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
images.py Loading commit data...
log.py Loading commit data...
loras.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
prompt_cache.py Loading commit data...
ratelimit.py Loading commit data...
spatial.py Loading commit data...
state.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...