-
Stefy Lanza (nextime / spora ) authored
collect_models unioned only registry.healthy() engines and re-fetched each engine's /v1/models live. An engine loading a model is GIL-blocked and misses the 2s health poll, so it goes "unhealthy" and ALL its models — including a freshly-added one — drop out of the aggregated /v1/models until the load finishes. A client (e.g. the kilo model script) polling during a load then sees models vanish. Cache each engine's last-good /v1/models and, when it's transiently unhealthy/unreachable, serve that cached list instead of dropping it. The models are still assigned to the engine and will serve once it's free. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
1c57629b
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| __init__.py | ||
| app.py | ||
| assignment.py | ||
| engine_supervisor.py | ||
| gpu_detect.py | ||
| registry.py | ||
| router.py |