• Stefy Lanza (nextime / spora )'s avatar
    front: keep an engine's models listed while it's mid-load · 1c57629b
    Stefy Lanza (nextime / spora ) authored
    collect_models unioned only registry.healthy() engines and re-fetched
    each engine's /v1/models live. An engine loading a model is GIL-blocked
    and misses the 2s health poll, so it goes "unhealthy" and ALL its models
    — including a freshly-added one — drop out of the aggregated /v1/models
    until the load finishes. A client (e.g. the kilo model script) polling
    during a load then sees models vanish. Cache each engine's last-good
    /v1/models and, when it's transiently unhealthy/unreachable, serve that
    cached list instead of dropping it. The models are still assigned to the
    engine and will serve once it's free.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    1c57629b
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...