• Stefy Lanza (nextime / spora )'s avatar
    text: make auto-compaction actually fire — fix config lookup + max_tokens-aware layered trimming · 913e283a
    Stefy Lanza (nextime / spora ) authored
    Auto-compaction never triggered: multi_model_manager.config stores the
    whitelisted build_runtime_kwargs() dict, which drops the per-model
    auto_compact* keys (they survive only under _raw_cfg), so _resolve_compaction
    always read the global default (False) and returned None. Read the keys via a
    _raw_cfg fallback so per-model compaction config is honoured.
    
    Also rework the over-context handling to count the reply reservation, since the
    reply is generated into the same window (prompt + max_tokens <= n_ctx). Four
    layers, cheapest first:
      1. fits as-is              -> nothing
      2. overflow within tol     -> trim max_tokens to fit (lossless)
      3. beyond tol & big prompt -> compact history (drop/summarize)
      4. single message too big  -> slice it (summarize its middle, keep head/tail)
    
    The chars/4 estimate undercounts token-dense code/JSON, so trimming to the exact
    n_ctx edge could still overflow; inflate the estimate by a configurable
    estimate_safety (default 1.15) for all physical-fit decisions.
    
    New CompactionConfig knobs (per-model overridable): tolerance_pct (20),
    min_output (512), estimate_safety (1.15). Effective max_tokens is threaded back
    to both the streaming and non-streaming generation paths.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    913e283a
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
characters.py Loading commit data...
custom_pipelines.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
images.py Loading commit data...
log.py Loading commit data...
loras.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
prompt_cache.py Loading commit data...
ratelimit.py Loading commit data...
spatial.py Loading commit data...
state.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...