Commit e17bc553 authored by Your Name's avatar Your Name

Add auto-detect support for --hf-chat-template

- Added 'auto' as a valid value for --hf-chat-template
- When --hf-chat-template auto is used, it auto-detects and applies HF template to all models
- Updated README with new syntax
parent 079bd8dc
......@@ -261,6 +261,9 @@ The `--hf-chat-template` option enables using HuggingFace's `apply_chat_template
**Usage:**
```bash
# Auto-detect and use HuggingFace chat template for all models
coderai --hf-chat-template auto --model llama-3.1-8b-instruct-q4_k_m.gguf
# Use HuggingFace chat template for ALL text models
coderai --hf-chat-template text --model llama-3.1-8b-instruct-q4_k_m.gguf
......@@ -271,20 +274,21 @@ coderai --hf-chat-template text:llama-3.1 --model llama-3.1-8b-instruct-q4_k_m.g
coderai --hf-chat-template text:llama-3.1 --hf-chat-template text:phi-3 --model llama-3.1-8b-instruct-q4_k_m.gguf
# Or with Vulkan backend
coderai --backend vulkan --hf-chat-template text --model llama-3.1-8b-instruct-q4_k_m.gguf
coderai --backend vulkan --hf-chat-template auto --model llama-3.1-8b-instruct-q4_k_m.gguf
```
**Syntax:**
| Syntax | Applies To |
|--------|------------|
| `--hf-chat-template auto` | Auto-detect and use HF template for all models |
| `--hf-chat-template text` | All text models |
| `--hf-chat-template image` | All image models |
| `--hf-chat-template text:model_name` | Specific text model |
| `--hf-chat-template image:model_name` | Specific image model |
**How it works:**
1. When `--hf-chat-template` is specified for a model, the server attempts to load a HuggingFace tokenizer
1. When `--hf-chat-template` is specified (with `auto` or a model spec), the server attempts to load a HuggingFace tokenizer
2. It first checks for a local `tokenizer_config.json` in the model directory
3. If not found locally, it tries to infer the model name from the GGUF filename and load from HuggingFace Hub
4. The tokenizer's `apply_chat_template` method is then used for formatting chat messages
......
......@@ -3132,6 +3132,9 @@ def check_hf_chat_template(model_type: str = "text", model_name: str = None) ->
True if HF chat template should be used, False otherwise.
Syntax:
# Auto-detect and apply to all text models
--hf-chat-template auto
# Apply to all text models
--hf-chat-template text
......@@ -3149,6 +3152,11 @@ def check_hf_chat_template(model_type: str = "text", model_name: str = None) ->
return False
for spec in hf_chat_template:
# Handle auto-detect - try to load HF tokenizer and auto-detect template
if spec == 'auto' or spec == '':
# Applies to all models when using 'auto'
return True
if ':' in spec:
# Format: text:model_name or image:model_name
parts = spec.split(':')
......@@ -5427,7 +5435,7 @@ def parse_args():
"--hf-chat-template",
action="append",
default=[],
help="Use HuggingFace transformers apply_chat_template for specific model(s). Format: --hf-chat-template text:model_name or --hf-chat-template text (all text models)",
help="Use HuggingFace transformers apply_chat_template. Use without value for auto-detect, or specify model: --hf-chat-template text:model_name. Use --hf-chat-template auto for global auto-detect.",
)
parser.add_argument(
"--system-prompt",
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment