Commit e17bc553 authored by Your Name's avatar Your Name

Add auto-detect support for --hf-chat-template

- Added 'auto' as a valid value for --hf-chat-template
- When --hf-chat-template auto is used, it auto-detects and applies HF template to all models
- Updated README with new syntax
parent 079bd8dc
...@@ -261,6 +261,9 @@ The `--hf-chat-template` option enables using HuggingFace's `apply_chat_template ...@@ -261,6 +261,9 @@ The `--hf-chat-template` option enables using HuggingFace's `apply_chat_template
**Usage:** **Usage:**
```bash ```bash
# Auto-detect and use HuggingFace chat template for all models
coderai --hf-chat-template auto --model llama-3.1-8b-instruct-q4_k_m.gguf
# Use HuggingFace chat template for ALL text models # Use HuggingFace chat template for ALL text models
coderai --hf-chat-template text --model llama-3.1-8b-instruct-q4_k_m.gguf coderai --hf-chat-template text --model llama-3.1-8b-instruct-q4_k_m.gguf
...@@ -271,20 +274,21 @@ coderai --hf-chat-template text:llama-3.1 --model llama-3.1-8b-instruct-q4_k_m.g ...@@ -271,20 +274,21 @@ coderai --hf-chat-template text:llama-3.1 --model llama-3.1-8b-instruct-q4_k_m.g
coderai --hf-chat-template text:llama-3.1 --hf-chat-template text:phi-3 --model llama-3.1-8b-instruct-q4_k_m.gguf coderai --hf-chat-template text:llama-3.1 --hf-chat-template text:phi-3 --model llama-3.1-8b-instruct-q4_k_m.gguf
# Or with Vulkan backend # Or with Vulkan backend
coderai --backend vulkan --hf-chat-template text --model llama-3.1-8b-instruct-q4_k_m.gguf coderai --backend vulkan --hf-chat-template auto --model llama-3.1-8b-instruct-q4_k_m.gguf
``` ```
**Syntax:** **Syntax:**
| Syntax | Applies To | | Syntax | Applies To |
|--------|------------| |--------|------------|
| `--hf-chat-template auto` | Auto-detect and use HF template for all models |
| `--hf-chat-template text` | All text models | | `--hf-chat-template text` | All text models |
| `--hf-chat-template image` | All image models | | `--hf-chat-template image` | All image models |
| `--hf-chat-template text:model_name` | Specific text model | | `--hf-chat-template text:model_name` | Specific text model |
| `--hf-chat-template image:model_name` | Specific image model | | `--hf-chat-template image:model_name` | Specific image model |
**How it works:** **How it works:**
1. When `--hf-chat-template` is specified for a model, the server attempts to load a HuggingFace tokenizer 1. When `--hf-chat-template` is specified (with `auto` or a model spec), the server attempts to load a HuggingFace tokenizer
2. It first checks for a local `tokenizer_config.json` in the model directory 2. It first checks for a local `tokenizer_config.json` in the model directory
3. If not found locally, it tries to infer the model name from the GGUF filename and load from HuggingFace Hub 3. If not found locally, it tries to infer the model name from the GGUF filename and load from HuggingFace Hub
4. The tokenizer's `apply_chat_template` method is then used for formatting chat messages 4. The tokenizer's `apply_chat_template` method is then used for formatting chat messages
......
...@@ -3132,6 +3132,9 @@ def check_hf_chat_template(model_type: str = "text", model_name: str = None) -> ...@@ -3132,6 +3132,9 @@ def check_hf_chat_template(model_type: str = "text", model_name: str = None) ->
True if HF chat template should be used, False otherwise. True if HF chat template should be used, False otherwise.
Syntax: Syntax:
# Auto-detect and apply to all text models
--hf-chat-template auto
# Apply to all text models # Apply to all text models
--hf-chat-template text --hf-chat-template text
...@@ -3149,6 +3152,11 @@ def check_hf_chat_template(model_type: str = "text", model_name: str = None) -> ...@@ -3149,6 +3152,11 @@ def check_hf_chat_template(model_type: str = "text", model_name: str = None) ->
return False return False
for spec in hf_chat_template: for spec in hf_chat_template:
# Handle auto-detect - try to load HF tokenizer and auto-detect template
if spec == 'auto' or spec == '':
# Applies to all models when using 'auto'
return True
if ':' in spec: if ':' in spec:
# Format: text:model_name or image:model_name # Format: text:model_name or image:model_name
parts = spec.split(':') parts = spec.split(':')
...@@ -5427,7 +5435,7 @@ def parse_args(): ...@@ -5427,7 +5435,7 @@ def parse_args():
"--hf-chat-template", "--hf-chat-template",
action="append", action="append",
default=[], default=[],
help="Use HuggingFace transformers apply_chat_template for specific model(s). Format: --hf-chat-template text:model_name or --hf-chat-template text (all text models)", help="Use HuggingFace transformers apply_chat_template. Use without value for auto-detect, or specify model: --hf-chat-template text:model_name. Use --hf-chat-template auto for global auto-detect.",
) )
parser.add_argument( parser.add_argument(
"--system-prompt", "--system-prompt",
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment