• Stefy Lanza (nextime / spora )'s avatar
    front/engine: responsive Tasks under load — front-side in-flight tracking + thermal cadence · afefee53
    Stefy Lanza (nextime / spora ) authored
    #1 Front-side in-flight task tracking: Engine.enter_request now records per-request
       metadata (model/kind/path/started_at) in engine.active; _merge_engine_tasks
       injects synthetic "running" task entries for in-flight requests not already
       reported by the engine (deduped by engine+model). Both the direct proxy and the
       broker route register/clear it. So the Tasks page shows work the front
       dispatched even when the engine is too GIL-busy generating to answer its own
       /admin/api/tasks. (Combines with the last-good cache.)
    
    #3 Thermal stopping-criteria cadence: the per-token callback held the GIL every
       token, starving admin handlers. Now it does real work every 50 tokens for text,
       scaling to every 100 when generation is fast (>50 tok/s) — far less GIL
       contention, negligible thermal-drift risk between checks.
    
    #2 (no change needed): generation already runs OFF the event loop via
       _aiter_blocking (asyncio.to_thread per token step) for streaming and
       asyncio.to_thread(manager.generate) for non-streaming.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    afefee53
registry.py 11.7 KB