rerank: resolve model via embedding category to pass manager type-validation

The reranker is registered in embedding_models, so request_model(model,"rerank")
was rejected ("is a embedding model and cannot be used for rerank generation").
Resolve with model_type="embedding" (its actual category) so validation passes;
rerank.py still loads it as a cross-encoder (AutoModelForSequenceClassification),
re-loading if a cached embedder object is found under the same key.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
parent 28d7711d
...@@ -102,8 +102,11 @@ async def create_rerank(request: RerankRequest, http_request: Request = None): ...@@ -102,8 +102,11 @@ async def create_rerank(request: RerankRequest, http_request: Request = None):
raise HTTPException(status_code=400, detail="'query' is required") raise HTTPException(status_code=400, detail="'query' is required")
# Resolve + reserve the model through the manager (thermal wait, caching, routing). # Resolve + reserve the model through the manager (thermal wait, caching, routing).
# Rerankers are registered in the embedding_models category (same transformers/nvidia
# infra), so resolve with model_type="embedding" to pass the manager's type
# validation; rerank.py then loads it as a cross-encoder (not an embedder).
model_info = await asyncio.to_thread( model_info = await asyncio.to_thread(
multi_model_manager.request_model, request.model, "rerank") multi_model_manager.request_model, request.model, "embedding")
model_name = model_info.get('model_name') model_name = model_info.get('model_name')
if not model_name: if not model_name:
raise HTTPException(status_code=404, raise HTTPException(status_code=404,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment