-
Stefy Lanza (nextime / spora ) authored
Every tool-using request to a pod-served model answered an EMPTY assistant message: content "", finish_reason "stop", no tool_calls. Asked directly the pod answered the call it had generated; through the orchestrator the same request answered nothing. Both the LoRA (diogene-alpha) and the base model were affected, so no adapter was involved. A proxy backend's generate_chat returns a STRING. The remote server owns the chat template, so it is the side that recognises a tool call and reports it structurally -- `{"content": null, "tool_calls": [...]}`. `message.content or ""` dropped that on the floor; the caller then had an empty string, found no call to parse in it, and answered the client with an empty message. The calls now travel beside the text, where usage and logprobs already do: get_last_tool_calls() on the backend, forwarded by the manager, preferred over re-parsing text in both the streamed and buffered chat paths. A backend that generates locally still keeps its call inside the text and answers None, so that path is untouched. Streaming needed the same: a streamed call arrives as deltas with no content at all, so merge_tool_call_deltas folds the fragments -- keyed by the call's index, never by arrival order, since concatenating the arguments of the wrong call yields valid JSON for a call nobody made. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
417328c7
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| __init__.py | ||
| base.py | ||
| colibri.py | ||
| colibri_families.py | ||
| cuda.py | ||
| ds4.py | ||
| ggml_rpc.py | ||
| gpu_probe.py | ||
| k3.py | ||
| ktransformers.py | ||
| overrides.py | ||
| remote_openai.py | ||
| runpod.py | ||
| vllm.py | ||
| vulkan.py |