• Stefy Lanza (nextime / spora )'s avatar
    front: keep an engine's models listed while it's mid-load · 1c57629b
    Stefy Lanza (nextime / spora ) authored
    collect_models unioned only registry.healthy() engines and re-fetched
    each engine's /v1/models live. An engine loading a model is GIL-blocked
    and misses the 2s health poll, so it goes "unhealthy" and ALL its models
    — including a freshly-added one — drop out of the aggregated /v1/models
    until the load finishes. A client (e.g. the kilo model script) polling
    during a load then sees models vanish. Cache each engine's last-good
    /v1/models and, when it's transiently unhealthy/unreachable, serve that
    cached list instead of dropping it. The models are still assigned to the
    engine and will serve once it's free.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    1c57629b
Name
Last commit
Last update
..
__init__.py Loading commit data...
app.py Loading commit data...
assignment.py Loading commit data...
engine_supervisor.py Loading commit data...
gpu_detect.py Loading commit data...
registry.py Loading commit data...
router.py Loading commit data...