Skip to content

A
aisbf
Commit a96f358d authored by Your Name's avatar Your Name
Browse files
Options

Add KiloCode Claude provider comparison to review document 

- Add KiloCode implementation analysis (vendors/kilocode/packages/opencode/src/provider/)
- Compare KiloCode's AI SDK approach (@ai-sdk/anthropic) vs direct HTTP
- Document KiloCode's features: automatic prompt caching, thinking support,
  message validation, reasoning variants, model management
- Add comparison tables for architecture, message conversion, streaming,
  headers, model resolution, reasoning/thinking support, prompt caching
- Document KiloCode exclusive features: empty content filtering, tool call ID
  sanitization, duplicate reasoning fix, provider option remapping,
  Gemini schema sanitization, unsupported part handling
- Update summary with KiloCode strengths and additional improvement areas
parent ce2788a9
Show whitespace changes
Inline Side-by-side
Showing with 155 additions and 29 deletions
docs/claude_provider_comparison.md
View file @ a96f358d
# Claude Provider Comparison: AISBF vs Original Claude Code vs Kiro Gateway
# Claude Provider Comparison: AISBF vs Original Claude Code vs KiloCode vs Kiro Gateway
**Date:** 2026-03-31
**Reviewed by:** AI Assistant
......@@ -8,34 +8,38 @@
- [`aisbf/kiro_converters_openai.py`](aisbf/kiro_converters_openai.py) - Kiro OpenAI adapters
- [`aisbf/kiro_parsers.py`](aisbf/kiro_parsers.py) - Kiro response parser
**Original Source:** `vendors/claude/src/`
**KiloCode Source:** `vendors/kilocode/packages/opencode/src/provider/`
---
## Overview
This document compares three implementations:
This document compares four implementations:
1. **AISBF Claude Provider** - Direct HTTP implementation using OAuth2 tokens
2. **Original Claude Code** - TypeScript/React implementation from Anthropic
3. **Kiro Gateway** - Python implementation for Amazon Q Developer (Claude-based)
3. **KiloCode** - TypeScript implementation using AI SDK (`@ai-sdk/anthropic`)
4. **Kiro Gateway** - Python implementation for Amazon Q Developer (Claude-based)
---
## 1. Architecture & Approach
| Aspect | AISBF Claude | Original Claude Code | Kiro Gateway |
|--------|-------------|---------------------|--------------|
| **Language** | Python | TypeScript/React | Python |
| **API Method** | Direct HTTP via `httpx.AsyncClient` | Anthropic SDK + internal `callModel` | AWS Event Stream via `httpx` |
| **Authentication** | OAuth2 via `ClaudeAuth` class | Internal OAuth2 + session management | AWS SSO OIDC / Kiro Desktop |
| **Endpoint** | `https://api.anthropic.com/v1/messages` | Internal SDK routing | `https://q.{region}.amazonaws.com/generateAssistantResponse` |
| **Response Format** | Standard JSON / SSE | SDK streaming | AWS Event Stream binary |
| **Protocol** | Anthropic Messages API | Anthropic Messages API | AWS CodeWhisperer API |
| Aspect | AISBF Claude | Original Claude Code | KiloCode | Kiro Gateway |
|--------|-------------|---------------------|----------|--------------|
| **Language** | Python | TypeScript/React | TypeScript | Python |
| **API Method** | Direct HTTP via `httpx.AsyncClient` | Anthropic SDK + internal `callModel` | AI SDK (`@ai-sdk/anthropic`) | AWS Event Stream via `httpx` |
| **Authentication** | OAuth2 via `ClaudeAuth` class | Internal OAuth2 + session management | API key / OAuth via Auth system | AWS SSO OIDC / Kiro Desktop |
| **Endpoint** | `https://api.anthropic.com/v1/messages` | Internal SDK routing | Configurable (baseURL) | `https://q.{region}.amazonaws.com/generateAssistantResponse` |
| **Response Format** | Standard JSON / SSE | SDK streaming | AI SDK streaming | AWS Event Stream binary |
| **Protocol** | Anthropic Messages API | Anthropic Messages API | AI SDK (unified) | AWS CodeWhisperer API |
| **Beta Headers** | `claude-code-20250219,oauth-2025-04-20,interleaved-thinking-2025-05-14,...` | Internal | `claude-code-20250219,interleaved-thinking-2025-05-14,fine-grained-tool-streaming-2025-05-14` | N/A |
**Assessment:**
- AISBF uses the direct HTTP approach (kilocode method) appropriate for OAuth2 tokens
- KiloCode uses the AI SDK (`@ai-sdk/anthropic`) which provides a unified interface across providers
- Kiro uses a completely different API (AWS CodeWhisperer) with binary Event Stream responses
- Both AISBF and Kiro are Python, making them easier to compare directly
- KiloCode's custom loader for Anthropic ([`provider.ts:125`](vendors/kilocode/packages/opencode/src/provider/provider.ts:125)) sets beta headers similar to AISBF
---
......@@ -49,6 +53,15 @@ This document compares three implementations:
- Converts assistant `tool_calls` to Anthropic `tool_use` blocks
- Handles message role alternation requirements
### KiloCode: [`normalizeMessages()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:49) + [`applyCaching()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:177)
**What it does well:**
- **Empty content filtering**: Removes empty string messages and empty text/reasoning parts from array content
- **Tool call ID sanitization**: Sanitizes tool call IDs for Claude models (replaces non-alphanumeric chars with `_`)
- **Prompt caching**: Applies `cacheControl: { type: "ephemeral" }` to system messages and last 2 messages
- **Provider option remapping**: Remaps providerOptions keys from stored providerID to expected SDK key
- **Duplicate reasoning fix**: Removes duplicate reasoning_details from OpenRouter responses
### Kiro Gateway: [`convert_openai_messages_to_unified()`](aisbf/kiro_converters_openai.py:249) + [`build_kiro_history()`](aisbf/kiro_converters.py:1256)
**What it does well:**
......@@ -68,12 +81,13 @@ The original ([`normalizeMessagesForAPI()`](vendors/claude/src/utils/messages.ts
- Message UUID tracking for caching
### Key Architectural Difference:
| Feature | AISBF Claude | Kiro Gateway |
|---------|-------------|--------------|
| **Conversion Strategy** | Direct OpenAI → Anthropic | OpenAI → Unified → Kiro |
| **Image Support** | No | Yes (full multimodal) |
| **Message Validation** | Basic | Comprehensive (role normalization, merging, alternation) |
| **Synthetic Messages** | No | Yes (for role alternation, first message) |
| Feature | AISBF Claude | KiloCode | Kiro Gateway |
|---------|-------------|----------|--------------|
| **Conversion Strategy** | Direct OpenAI → Anthropic | AI SDK message normalization | OpenAI → Unified → Kiro |
| **Image Support** | No | Via AI SDK | Yes (full multimodal) |
| **Message Validation** | Basic | Empty content filtering, ID sanitization | Comprehensive (role normalization, merging, alternation) |
| **Synthetic Messages** | No | No | Yes (for role alternation, first message) |
| **Caching** | No | Yes (ephemeral cache on system/last 2 msgs) | N/A (different API) |
---
......@@ -87,6 +101,13 @@ The original ([`normalizeMessagesForAPI()`](vendors/claude/src/utils/messages.ts
- Removes `additionalProperties: false` (Anthropic doesn't need it)
- Recursively normalizes nested schemas
### KiloCode: AI SDK handles conversion internally
**What it does:**
- Uses `@ai-sdk/anthropic` which handles OpenAI → Anthropic tool conversion internally
- Tool schemas pass through [`schema()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:954) for Gemini/Google models (integer enum → string enum conversion)
- No explicit Anthropic-specific tool conversion needed (SDK handles it)
### Kiro Gateway: [`convert_tools_to_kiro_format()`](aisbf/kiro_converters.py:537) + [`sanitize_json_schema()`](aisbf/kiro_converters.py:374)
**What it does well:**
......@@ -113,6 +134,10 @@ The original has additional tool validation including:
- `"required"` → `{"type": "any"}`
- Specific function → `{"type": "tool", "name": "..."}`
### KiloCode: AI SDK handles internally
- Uses AI SDK's unified tool choice handling
- No explicit tool_choice conversion needed
### Kiro Gateway:
Kiro doesn't use tool_choice in the same way - tools are specified in `userInputMessageContext.tools` and the model decides which to use.
......@@ -128,6 +153,15 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
- Handles `message_stop` for final chunk
- Yields OpenAI-compatible chunks
### KiloCode: AI SDK streaming
**What it does:**
- Uses AI SDK's built-in streaming via `sdk.languageModel(modelId).doStream()` or `generateText()`
- Handles thinking blocks, tool calls, and text content through unified SDK interface
- Provider-specific streaming handled by `@ai-sdk/anthropic` package
- Supports `fine-grained-tool-streaming-2025-05-14` beta header for streaming tool calls
- Custom fetch wrapper with timeout handling ([`provider.ts:1138`](vendors/kilocode/packages/opencode/src/provider/provider.ts:1138))
### Kiro Gateway: [`AwsEventStreamParser`](aisbf/kiro_parsers.py:220)
**What it does:**
......@@ -139,13 +173,14 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
- **Tool call deduplication**: Removes duplicate tool calls by ID and name+args
### Key Differences:
| Feature | AISBF Claude | Kiro Gateway |
|---------|-------------|--------------|
| **Protocol** | SSE (text) | AWS Event Stream (binary) |
| **Tool Streaming** | No | Yes (tool_start/input/stop events) |
| **Usage Tracking** | No | Yes (usage + context_usage events) |
| **Error Recovery** | Basic | Truncation detection and diagnosis |
| **Content Dedup** | No | Yes |
| Feature | AISBF Claude | KiloCode | Kiro Gateway |
|---------|-------------|----------|--------------|
| **Protocol** | SSE (text) | AI SDK (abstracted) | AWS Event Stream (binary) |
| **Tool Streaming** | No | Yes (via fine-grained-tool-streaming beta) | Yes (tool_start/input/stop events) |
| **Thinking Blocks** | No | Yes (via SDK) | N/A |
| **Usage Tracking** | No | Yes (via SDK) | Yes (usage + context_usage events) |
| **Error Recovery** | Basic | SDK-level | Truncation detection and diagnosis |
| **Content Dedup** | No | No | Yes |
---
......@@ -159,6 +194,12 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
- Stop reason mapping (`end_turn` → `stop`, etc.)
- Usage metadata extraction
### KiloCode: AI SDK handles internally
- AI SDK provides unified response format across all providers
- No manual response conversion needed
- Usage metadata includes cache tokens via SDK
- Cost calculation handles provider-specific differences ([`session/index.ts:860`](vendors/kilocode/packages/opencode/src/session/index.ts:860))
### Kiro Gateway: [`AwsEventStreamParser.get_content()`](aisbf/kiro_parsers.py:557) + [`get_tool_calls()`](aisbf/kiro_parsers.py:561)
**What it does:**
......@@ -178,6 +219,14 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
- `Anthropic-Beta` with multiple beta features
- `X-App: cli` and other stainless headers
### KiloCode: [`CUSTOM_LOADERS.anthropic()`](vendors/kilocode/packages/opencode/src/provider/provider.ts:125)
**Includes:**
- API key via `options.apiKey` or auth system
- `anthropic-beta: claude-code-20250219,interleaved-thinking-2025-05-14,fine-grained-tool-streaming-2025-05-14`
- Custom fetch wrapper with timeout handling
- TODO comment for adaptive thinking headers: `adaptive-thinking-2026-01-28,effort-2025-11-24,max-effort-2026-01-24`
### Kiro Gateway: AWS SigV4 / OAuth2
**Uses:**
......@@ -193,6 +242,12 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
- Queries `https://api.anthropic.com/v1/models`
- Uses model names as returned by API
### KiloCode: Model ID passthrough
- Uses model IDs from models.dev database
- Model variants generated via [`variants()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:381) for reasoning efforts
- Model sorting via [`sort()`](vendors/kilocode/packages/opencode/src/provider/provider.ts:1348) with priority: `gpt-5`, `claude-sonnet-4`, `big-pickle`, `gemini-3-pro`
- Small model selection via [`getSmallModel()`](vendors/kilocode/packages/opencode/src/provider/provider.ts:1277) with priority list including `claude-haiku-4-5`
### Kiro Gateway: [`normalize_model_name()`](aisbf/kiro_converters_openai.py:52) + [`get_model_id_for_kiro()`](aisbf/kiro_converters_openai.py:130)
**Sophisticated normalization:**
......@@ -204,7 +259,64 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
---
## 9. Advanced Features
## 9. Reasoning/Thinking Support
### AISBF Claude: No explicit support
- No thinking block handling in current implementation
### KiloCode: Full thinking support via AI SDK
**Features ([`variants()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:381)):**
- **Adaptive thinking** for Opus 4.6 / Sonnet 4.6: `thinking: { type: "adaptive" }` with effort levels
- **Budget-based thinking** for other Claude models: `thinking: { type: "enabled", budgetTokens: N }`
- Effort levels: `low`, `medium`, `high`, `max`
- Temperature returns `undefined` for Claude models (let SDK decide)
### Kiro Gateway: Thinking mode injection
**Features:**
- Injects `<thinking_mode>enabled</thinking_mode>` tags into user messages
- Configurable max thinking tokens (default 32000)
- System prompt addition to legitimize thinking tags
---
## 10. Prompt Caching
### AISBF Claude: No explicit support
- No cache_control headers applied
### KiloCode: Automatic ephemeral caching
**Implementation ([`applyCaching()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:177)):**
- Applies `cacheControl: { type: "ephemeral" }` to:
- First 2 system messages
- Last 2 non-system messages
- Provider-specific cache options:
- Anthropic: `cacheControl: { type: "ephemeral" }`
- OpenRouter: `cacheControl: { type: "ephemeral" }`
- Bedrock: `cachePoint: { type: "default" }`
- OpenAI Compatible: `cache_control: { type: "ephemeral" }`
- GitHub Copilot: `copilot_cache_control: { type: "ephemeral" }`
### Kiro Gateway: N/A (different API)
---
## 11. Advanced Features
### KiloCode Exclusive Features:
| Feature | Description | Location |
|---------|-------------|----------|
| **Reasoning Variants** | Auto-generates low/medium/high/max variants for reasoning models | [`variants()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:381) |
| **Small Model Selection** | Automatic fallback to haiku/flash/nano models | [`getSmallModel()`](vendors/kilocode/packages/opencode/src/provider/provider.ts:1277) |
| **Empty Content Filtering** | Removes empty messages and text/reasoning parts | [`normalizeMessages()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:49) |
| **Tool Call ID Sanitization** | Replaces non-alphanumeric chars in tool call IDs for Claude | [`normalizeMessages()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:76) |
| **Duplicate Reasoning Fix** | Removes duplicate reasoning_details from OpenRouter | [`fixDuplicateReasoning()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:256) |
| **Provider Option Remapping** | Remaps providerOptions keys to match SDK expectations | [`message()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:318) |
| **Gemini Schema Sanitization** | Converts integer enums to string enums for Google models | [`schema()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:954) |
| **Unsupported Part Handling** | Converts unsupported media types to error text | [`unsupportedParts()`](vendors/kilocode/packages/opencode/src/provider/transform.ts:217) |
### Kiro Gateway Exclusive Features:
......@@ -218,7 +330,7 @@ Kiro doesn't use tool_choice in the same way - tools are specified in `userInput
---
## 10. Missing Features (Not in AISBF)
## 12. Missing Features (Not in AISBF)
The original Claude Code has many features that are out of scope for a provider handler but worth noting:
......@@ -247,6 +359,16 @@ The original Claude Code has many features that are out of scope for a provider
5. Robust fallback strategy for model discovery
6. Adaptive rate limiting with learning
### Strengths of KiloCode implementation:
1. **AI SDK abstraction**: Unified interface across all providers
2. **Automatic prompt caching**: Ephemeral caching on system/last 2 messages
3. **Full thinking support**: Adaptive thinking + budget-based thinking for Claude models
4. **Message validation**: Empty content filtering, tool call ID sanitization
5. **Reasoning variants**: Auto-generates effort level variants for reasoning models
6. **Provider option remapping**: Handles provider-specific SDK key differences
7. **Robust error handling**: Duplicate reasoning fix, unsupported part handling
8. **Model management**: Small model selection, priority sorting
### Strengths of Kiro Gateway implementation:
1. **Unified intermediate format** for API-agnostic processing
2. **Comprehensive message validation** (role normalization, merging, alternation)
......@@ -259,15 +381,19 @@ The original Claude Code has many features that are out of scope for a provider
### Areas for improvement (AISBF):
1. Add thinking block support for models that use it
2. Add tool call streaming
2. Add tool call streaming (fine-grained-tool-streaming beta)
3. Add more detailed usage metadata (cache tokens)
4. Consider adding model fallback support
5. Add tool result size validation
6. Consider adopting Kiro's unified message format approach
7. Add message role normalization and validation
8. Add image/multimodal support
9. Add prompt caching (ephemeral cache on system/last 2 messages)
10. Add tool call ID sanitization for Claude compatibility
### Overall assessment:
The AISBF Claude provider is a solid implementation that correctly handles the core API communication, message conversion, and tool handling. It appropriately focuses on the provider-level concerns (API translation) while leaving higher-level concerns (conversation management, compaction) to the rest of the framework.
The KiloCode implementation demonstrates the power of using the AI SDK (`@ai-sdk/anthropic`) for provider abstraction, with automatic handling of thinking, caching, and tool conversion. Its message validation pipeline (empty content filtering, ID sanitization, duplicate reasoning fix) provides robustness that AISBF could benefit from.
The Kiro Gateway implementation demonstrates more sophisticated message handling with its unified intermediate format, comprehensive validation, and multimodal support. Several of these patterns (particularly the message validation pipeline and unified format approach) could be valuable additions to the AISBF codebase.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment