• Stefy Lanza (nextime / spora )'s avatar
    front/engine: responsive Tasks under load — front-side in-flight tracking + thermal cadence · afefee53
    Stefy Lanza (nextime / spora ) authored
    #1 Front-side in-flight task tracking: Engine.enter_request now records per-request
       metadata (model/kind/path/started_at) in engine.active; _merge_engine_tasks
       injects synthetic "running" task entries for in-flight requests not already
       reported by the engine (deduped by engine+model). Both the direct proxy and the
       broker route register/clear it. So the Tasks page shows work the front
       dispatched even when the engine is too GIL-busy generating to answer its own
       /admin/api/tasks. (Combines with the last-good cache.)
    
    #3 Thermal stopping-criteria cadence: the per-token callback held the GIL every
       token, starving admin handlers. Now it does real work every 50 tokens for text,
       scaling to every 100 when generation is fast (>50 tok/s) — far less GIL
       contention, negligible thermal-drift risk between checks.
    
    #2 (no change needed): generation already runs OFF the event loop via
       _aiter_blocking (asyncio.to_thread per token step) for streaming and
       asyncio.to_thread(manager.generate) for non-streaming.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    afefee53
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...