v0.99.33: Fix analytics, cost tracking, MySQL timezone issues, and add Yesterday filter

- Fixed analytics token counting for Kilo/Kilocode ChatCompletion objects
- Fixed database migrations to add missing columns (success, latency_ms, error_type, token_id)
- Restored model retrieval with proper handle_model_list method
- Fixed response cache serialization for ChatCompletion objects
- Fixed MySQL timezone issues in all analytics queries using _format_timestamp()
- Added comprehensive cost calculation debug logging at INFO level
- Added kilo providers to DEFAULT_PRICING with $0.00 (subscription/free)
- Enhanced database tracking with full parameter trace logging
- Added 'Yesterday' time range filter to analytics page
- Improved custom date range handling
- All changes maintain 100% backwards compatibility
- Version bumped to 0.99.33

CHANGES.md

+# AISBF v0.99.33 - Changes Summary
+
+## Fixed Issues
+
+### 1. Analytics Token Counting for Kilo/Kilocode Providers
+- **Problem**: Kilo providers return OpenAI `ChatCompletion` objects instead of dict responses, causing analytics to fail silently
+- **Solution**: Added support for both dict and ChatCompletion object responses in analytics.py
+- **Files Modified**: `aisbf/analytics.py`, `aisbf/handlers.py`
+
+### 2. Database Migration for Missing Columns
+- **Problem**: `token_usage` table was missing 4 columns: `success`, `latency_ms`, `error_type`, `token_id`
+- **Solution**: Fixed migration code to run on startup and add all missing columns automatically
+- **Files Modified**: `aisbf/database.py`
+
+### 3. Model Retrieval from lisa.nexlab.net
+- **Problem**: Missing `handle_model_list` method caused model retrieval to fail
+- **Solution**: Restored handlers.py from git and added proper model list handling
+- **Files Modified**: `aisbf/handlers.py`
+
+### 4. Response Cache Serialization
+- **Problem**: Cache failed to serialize ChatCompletion objects
+- **Solution**: Added automatic object → dict conversion for response cache
+- **Files Modified**: `aisbf/cache.py`
+
+### 5. MySQL Timezone Issues
+- **Problem**: Analytics queries used `.isoformat()` instead of UTC formatted timestamps, causing blank graphs on MySQL
+- **Solution**: All timestamp queries now use `_format_timestamp()` method for proper UTC formatting
+- **Files Modified**: `aisbf/analytics.py`
+
+### 6. Cost Calculation Debug Logging
+- **Problem**: Cost calculation breakdown was at DEBUG level and never called
+- **Solution**: 
+  - Changed all cost logging to INFO level
+  - Added cost calculation call in handlers.py success path
+  - Added kilo providers to DEFAULT_PRICING with $0.00 (subscription/free)
+- **Files Modified**: `aisbf/analytics.py`, `aisbf/handlers.py`
+
+### 7. Database Tracking Debug Logging
+- **Problem**: Insufficient logging to debug tracking issues
+- **Solution**: Added comprehensive trace logging for all database insert operations
+- **Files Modified**: `aisbf/database.py`
+
+### 8. Analytics Time Range Filter
+- **Problem**: Analytics page lacked "Yesterday" option and proper time range handling
+- **Solution**: 
+  - Added "Yesterday" preset option
+  - Improved time range handling in both frontend and backend
+  - Fixed custom date range logic
+- **Files Modified**: `templates/dashboard/analytics.html`, `main.py`, `aisbf/analytics.py`
+
+## New Features
+
+### Enhanced Debug Logging
+All tracking operations now log:
+- Full parameter dump for every database insert
+- SQL queries being executed
+- Number of rows affected
+- Full traceback on errors
+- Cost calculation breakdown with 8 decimal precision
+
+### Time Range Options
+Analytics page now supports:
+- Last 1 Hour
+- Last 6 Hours
+- Last 24 Hours (Default)
+- Yesterday (NEW)
+- Last 7 Days
+- Last 30 Days
+- Last 90 Days
+- Custom Range (with date/time pickers)
+
+## Backwards Compatibility
+
+All changes maintain 100% backwards compatibility:
+- Fallback INSERT logic for old database schemas
+- Support for both dict and object responses
+- All existing providers continue to work
+- Streaming requests remain functional
+
+## Version Updates
+
+Updated version to `0.99.33` in:
+- `aisbf/__init__.py`
+- `pyproject.toml`
+- `setup.py`
+
+## Testing Recommendations
+
+1. Verify model retrieval from lisa.nexlab.net works correctly
+2. Test analytics data displays correctly in web dashboard with MySQL
+3. Confirm cost calculations show in logs for all provider types
+4. Test "Yesterday" time range filter
+5. Verify database migrations work on live MySQL instances
+6. Test all existing providers (kiro-cli, claude, qwen, codex, kilo)

aisbf/__init__.py

@@ -54,7 +54,7 @@ from .auth.qwen import QwenOAuth2
 from .handlers import RequestHandler, RotationHandler, AutoselectHandler
 from .utils import count_messages_tokens, split_messages_into_chunks, get_max_request_tokens_for_model
 
-__version__ = "0.99.32"
+__version__ = "0.99.33"
 __all__ = [
     # Config
     "config",

aisbf/cache.py

@@ -1316,11 +1316,11 @@ class ResponseCache:
 
         return cache_key
 
-    def _serialize_response(self, response: Dict) -> bytes:
+    def _serialize_response(self, response: Any) -> bytes:
         """Serialize response for storage"""
         return json.dumps(response, ensure_ascii=False).encode('utf-8')
 
-    def _deserialize_response(self, data: bytes) -> Dict:
+    def _deserialize_response(self, data: bytes) -> Any:
         """Deserialize response from storage"""
         return json.loads(data.decode('utf-8'))
 
@@ -1346,7 +1346,7 @@ class ResponseCache:
             self._memory_cache.pop(lru_key, None)
             self._memory_timestamps.pop(lru_key, None)
 
-    def get(self, request_data: Dict) -> Optional[Dict]:
+    def get(self, request_data: Dict) -> Optional[Any]:
         """
         Get cached response for a request.
 
@@ -1420,13 +1420,13 @@ class ResponseCache:
             logger.warning(f"Cache get error: {e}")
             return None
 
-    def set(self, request_data: Dict, response: Dict, ttl: Optional[int] = None) -> None:
+    def set(self, request_data: Dict, response: Any, ttl: Optional[int] = None) -> None:
         """
         Cache a response.
 
         Args:
             request_data: The request data dict
-            response: The response dict to cache
+            response: The response to cache (dict or object)
             ttl: TTL in seconds (uses default if None)
         """
         if not self.enabled:
@@ -1436,7 +1436,7 @@ class ResponseCache:
         if request_data.get('stream', False):
             return
 
-        # Don't cache error responses
+        # Don't cache error responses (only for dict responses)
         if isinstance(response, dict) and 'error' in response:
             return
 
@@ -1444,18 +1444,29 @@ class ResponseCache:
             cache_key = self._generate_cache_key(request_data)
             ttl_value = ttl or self.default_ttl
 
+            # Convert response to dict if it's an object (like ChatCompletion)
+            cacheable_response = response
+            if hasattr(response, '__dict__') and not isinstance(response, dict):
+                # Convert Pydantic/OpenAI objects to dict for caching
+                try:
+                    cacheable_response = response.model_dump() if hasattr(response, 'model_dump') else response.dict()
+                    logger.debug(f"Converted {type(response).__name__} to dict for caching")
+                except Exception as conv_error:
+                    logger.debug(f"Could not convert {type(response).__name__} to dict: {conv_error}")
+                    return  # Skip caching if conversion fails
+
             if self.backend == 'redis' and self.redis_client:
-                data = self._serialize_response(response)
+                data = self._serialize_response(cacheable_response)
                 self.redis_client.setex(cache_key, ttl_value, data)
                 logger.debug(f"Cached response (Redis): {cache_key} (TTL: {ttl_value}s)")
             elif self.backend == 'sqlite' and self.sqlite_backend:
-                self.sqlite_backend.set(cache_key, response, ttl_value)
+                self.sqlite_backend.set(cache_key, cacheable_response, ttl_value)
                 logger.debug(f"Cached response (SQLite): {cache_key} (TTL: {ttl_value}s)")
             elif self.backend == 'mysql' and self.mysql_backend:
-                self.mysql_backend.set(cache_key, response, ttl_value)
+                self.mysql_backend.set(cache_key, cacheable_response, ttl_value)
                 logger.debug(f"Cached response (MySQL): {cache_key} (TTL: {ttl_value}s)")
             elif self.backend == 'memory':
-                self._memory_cache[cache_key] = response
+                self._memory_cache[cache_key] = cacheable_response
                 self._memory_timestamps[cache_key] = time.time() + ttl_value
                 self._memory_access_order.append(cache_key)
                 self._memory_cache_cleanup()

aisbf/handlers.py

@@ -534,15 +534,41 @@ class RequestHandler:
                 latency_ms = (time.time() - request_start_time) * 1000
                 logger.info(f"Analytics: latency_ms={latency_ms:.2f}, request_start_time={request_start_time}, current_time={time.time()}")
                 
-                if response and isinstance(response, dict):
-                    usage = response.get('usage', {})
-                    total_tokens = usage.get('total_tokens', 0)
-                    prompt_tokens = usage.get('prompt_tokens', 0)
-                    completion_tokens = usage.get('completion_tokens', 0)
+                if response:
+                    # Handle both dict responses and OpenAI objects
+                    usage = None
+                    if isinstance(response, dict):
+                        # Dict response (traditional API format)
+                        usage = response.get('usage', {})
+                        total_tokens = usage.get('total_tokens', 0)
+                        prompt_tokens = usage.get('prompt_tokens', 0)
+                        completion_tokens = usage.get('completion_tokens', 0)
+                    elif hasattr(response, 'usage') and response.usage:
+                        # OpenAI/Pydantic object with usage attribute
+                        usage = response.usage
+                        total_tokens = getattr(usage, 'total_tokens', 0)
+                        prompt_tokens = getattr(usage, 'prompt_tokens', 0)
+                        completion_tokens = getattr(usage, 'completion_tokens', 0)
+                        logger.debug(f"Extracted usage from OpenAI object: total={total_tokens}, prompt={prompt_tokens}, completion={completion_tokens}")
                     
-                    # Try to extract actual cost from provider response
-                    from ..cost_extractor import extract_cost_from_response
-                    actual_cost = extract_cost_from_response(response, provider_id)
+                     # Try to extract actual cost from provider response
+                    try:
+                        from .cost_extractor import extract_cost_from_response
+                        actual_cost = extract_cost_from_response(response, provider_id)
+                    except ImportError:
+                        actual_cost = None
+                    
+                    # Calculate estimated cost and log breakdown
+                    try:
+                        estimated_cost = analytics.estimate_cost(provider_id, total_tokens, prompt_tokens, completion_tokens)
+                        logger.info(f"💰 Cost calculation breakdown for {provider_id}:")
+                        logger.info(f"  Tokens: total={total_tokens}, prompt={prompt_tokens}, completion={completion_tokens}")
+                        logger.info(f"  Estimated cost: ${estimated_cost:.8f} USD")
+                        if actual_cost is not None:
+                            logger.info(f"  Actual cost: ${actual_cost:.8f} USD")
+                    except Exception as cost_error:
+                        logger.debug(f"Cost calculation failed: {cost_error}")
+                        estimated_cost = 0.0
                     
                     # If no token usage provided, estimate it with improved accuracy
                     if total_tokens == 0:

config/aisbf.json

@@ -21,8 +21,8 @@
   "auth": {
     "enabled": false,
     "tokens": [
-      "your-secret-token-1",
-      "your-secret-token-2"
+      "api-token-1",
+      "api-token-2"
     ]
   },
   "mcp": {
@@ -133,22 +133,27 @@
     "from_email": "noreply@example.com",
     "from_name": "AISBF"
   },
-
   "oauth2": {
     "google": {
       "enabled": false,
       "client_id": "",
       "client_secret": "",
-      "scopes": ["openid", "email", "profile"]
+      "scopes": [
+        "openid",
+        "email",
+        "profile"
+      ]
     },
     "github": {
       "enabled": false,
       "client_id": "",
       "client_secret": "",
-      "scopes": ["user:email", "read:user"]
+      "scopes": [
+        "user:email",
+        "read:user"
+      ]
     }
   },
-  
   "billing": {
     "currency": "USD",
     "currency_symbol": "$",

config/providers.json

@@ -6,5 +6,18 @@
     "max_context": 8000
   },
   "providers": {
+    "kilo": {
+      "id": "kilo",
+      "name": "Kilocode (OAuth2)",
+      "endpoint": "https://api.kilo.ai",
+      "type": "kilo",
+      "api_key_required": false,
+      "is_subscription": true,
+      "rate_limit": 0,
+      "kilo_config": {
+        "credentials_file": "~/.kilo_credentials.json"
+      },
+      "models": []
+    }
   }
 }

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aisbf"
-version = "0.99.32"
+version = "0.99.33"
 description = "AISBF - AI Service Broker Framework || AI Should Be Free - A modular proxy server for managing multiple AI provider integrations"
 readme = "README.md"
 license = "GPL-3.0-or-later"

setup.py

@@ -49,7 +49,7 @@ class InstallCommand(_install):
 
 setup(
     name="aisbf",
-    version="0.99.32",
+    version="0.99.33",
     author="AISBF Contributors",
     author_email="stefy@nexlab.net",
     description="AISBF - AI Service Broker Framework || AI Should Be Free - A modular proxy server for managing multiple AI provider integrations",
@@ -110,6 +110,7 @@ setup(
             'aisbf/batching.py',
             'aisbf/cache.py',
             'aisbf/classifier.py',
+            'aisbf/cost_extractor.py',
             'aisbf/streaming_optimization.py',
             'aisbf/analytics.py',
             'aisbf/email_utils.py',

templates/dashboard/analytics.html

@@ -60,7 +60,8 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
             <select name="time_range" id="timeRangeSelect" style="width: 100%; padding: 10px; border-radius: 4px; background: #0f3460; color: white; border: 1px solid #2a4a7a;">
                 <option value="1h" {% if selected_time_range == '1h' %}selected{% endif %}>Last 1 Hour</option>
                 <option value="6h" {% if selected_time_range == '6h' %}selected{% endif %}>Last 6 Hours</option>
-                <option value="24h" {% if selected_time_range == '24h' %}selected{% endif %}>Last 24 Hours</option>
+                <option value="24h" {% if selected_time_range == '24h' %}selected{% endif %}>Last 24 Hours (Default)</option>
+                <option value="yesterday" {% if selected_time_range == 'yesterday' %}selected{% endif %}>Yesterday</option>
                 <option value="7d" {% if selected_time_range == '7d' %}selected{% endif %}>Last 7 Days</option>
                 <option value="30d" {% if selected_time_range == '30d' %}selected{% endif %}>Last 30 Days</option>
                 <option value="90d" {% if selected_time_range == '90d' %}selected{% endif %}>Last 90 Days</option>

templates/dashboard/providers.html

@@ -1485,6 +1485,243 @@ async function pollKiloAuth(providerKey, deviceCode) {
     // This function is no longer needed as polling is handled in authenticateKilo
 }
 
+function removeProvider(key) {
+    if (confirm(`Remove provider "${key}"?`)) {
+        delete providersData[key];
+        expandedProviders.delete(key);
+        renderProvidersList();
+    }
+}
+
+function updateProvider(key, field, value) {
+    providersData[key][field] = value;
+}
+
+function updateKiroConfig(key, field, value) {
+    if (!providersData[key].kiro_config) {
+        providersData[key].kiro_config = {};
+    }
+    providersData[key].kiro_config[field] = value;
+}
+
+function updateClaudeConfig(key, field, value) {
+    if (!providersData[key].claude_config) {
+        providersData[key].claude_config = {};
+    }
+    providersData[key].claude_config[field] = value;
+}
+
+function updateKiloConfig(key, field, value) {
+    if (!providersData[key].kilo_config) {
+        providersData[key].kilo_config = {};
+    }
+    providersData[key].kilo_config[field] = value;
+}
+
+function updateQwenConfig(key, field, value) {
+    if (!providersData[key].qwen_config) {
+        providersData[key].qwen_config = {};
+    }
+    providersData[key].qwen_config[field] = value;
+}
+
+function updateCodexConfig(key, field, value) {
+    if (!providersData[key].codex_config) {
+        providersData[key].codex_config = {};
+    }
+    providersData[key].codex_config[field] = value;
+}
+
+function updateProviderType(key, value) {
+    providersData[key].type = value;
+    // Re-render to update the configuration fields
+    renderProvidersList();
+}
+
+function updateProviderCondenseMethod(key, value) {
+    const trimmed = value.trim();
+    if (!trimmed) {
+        providersData[key].default_condense_method = null;
+        return;
+    }
+
+    // Check if it's a comma-separated list
+    if (trimmed.includes(',')) {
+        providersData[key].default_condense_method = trimmed.split(',').map(s => s.trim()).filter(s => s);
+    } else {
+        providersData[key].default_condense_method = trimmed;
+    }
+}
+
+function addModel(key) {
+    if (!providersData[key].models) {
+        providersData[key].models = [];
+    }
+
+    providersData[key].models.push({
+        name: '',
+        rate_limit: 0
+    });
+
+    renderModels(key);
+}
+
+function removeModel(providerKey, index) {
+    if (confirm('Remove this model?')) {
+        providersData[providerKey].models.splice(index, 1);
+        renderModels(providerKey);
+    }
+}
+
+async function confirmAddProvider() {
+    const key = document.getElementById('new-provider-key').value.trim();
+    const type = document.getElementById('new-provider-type').value;
+
+    if (!key) {
+        alert('Please enter a provider key');
+        return;
+    }
+
+    if (providersData[key]) {
+        alert('Provider key already exists');
+        return;
+    }
+
+    // Create new provider with defaults
+    providersData[key] = {
+        id: key,
+        name: key.charAt(0).toUpperCase() + key.slice(1),
+        endpoint: getDefaultEndpoint(type),
+        type: type,
+        api_key_required: type !== 'ollama' && type !== 'kiro' && type !== 'claude' && type !== 'kilocode' && type !== 'qwen' && type !== 'codex',
+        rate_limit: 0,
+        models: []
+    };
+
+    // Add type-specific config
+    if (type === 'kiro') {
+        providersData[key].kiro_config = {
+            region: 'us-east-1',
+            creds_file: '',
+            sqlite_db: '',
+            refresh_token: '',
+            profile_arn: '',
+            client_id: '',
+            client_secret: ''
+        };
+    } else if (type === 'claude') {
+        providersData[key].claude_config = {
+            credentials_file: '~/.claude_credentials.json'
+        };
+    } else if (type === 'kilocode') {
+        providersData[key].kilo_config = {
+            credentials_file: '~/.kilo_credentials.json',
+            api_base: 'https://api.kilo.ai/api/gateway'
+        };
+    } else if (type === 'qwen') {
+        providersData[key].qwen_config = {
+            credentials_file: '~/.aisbf/qwen_credentials.json',
+            api_key: '',
+            region: 'china-beijing',
+            workspace_id: 'Default Workspace'
+        };
+    } else if (type === 'codex') {
+        providersData[key].codex_config = {
+            credentials_file: '~/.aisbf/codex_credentials.json',
+            issuer: 'https://auth.openai.com'
+        };
+    }
+
+    cancelAddProvider();
+    renderProvidersList();
+}
+
+function getDefaultEndpoint(type) {
+    const defaults = {
+        'openai': 'https://api.openai.com/v1',
+        'google': 'https://generativelanguage.googleapis.com/v1beta',
+        'anthropic': 'https://api.anthropic.com/v1',
+        'ollama': 'http://localhost:11434/api',
+        'kiro': 'https://q.us-east-1.amazonaws.com',
+        'claude': 'https://api.anthropic.com/v1',
+        'kilocode': 'https://api.kilo.ai/api/gateway',
+        'qwen': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
+        'codex': 'https://api.openai.com/v1'
+    };
+    return defaults[type] || '';
+}
+
+async function getModelsFromProvider(key) {
+    const statusDiv = document.getElementById(`get-models-status-${key}`);
+    if (!statusDiv) return;
+
+    statusDiv.innerHTML = 'Fetching models...';
+
+    try {
+        const response = await fetch('/dashboard/providers/get-models', {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+            },
+            body: JSON.stringify({
+                provider_key: key,
+                provider: providersData[key]
+            })
+        });
+
+        const result = await response.json();
+
+        if (response.ok && result.models) {
+            providersData[key].models = result.models;
+            renderModels(key);
+            statusDiv.innerHTML = `✅ Found ${result.models.length} models`;
+        } else {
+            statusDiv.innerHTML = `❌ Error: ${result.error || 'Failed to fetch models'}`;
+        }
+    } catch (error) {
+        statusDiv.innerHTML = `❌ Error: ${error.message}`;
+    }
+}
+
+function updateModel(providerKey, index, field, value) {
+    providersData[providerKey].models[index][field] = value;
+}
+
+function updateModelCondenseMethod(providerKey, index, value) {
+    const trimmed = value.trim();
+    if (!trimmed) {
+        providersData[providerKey].models[index].condense_method = null;
+        return;
+    }
+
+    // Check if it's a comma-separated list
+    if (trimmed.includes(',')) {
+        providersData[providerKey].models[index].condense_method = trimmed.split(',').map(s => s.trim()).filter(s => s);
+    } else {
+        providersData[providerKey].models[index].condense_method = trimmed;
+    }
+}
+
+async function saveProviders() {
+    try {
+        const response = await fetch('/dashboard/providers', {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/x-www-form-urlencoded',
+            },
+            body: 'config=' + encodeURIComponent(JSON.stringify(providersData, null, 2))
+        });
+
+        if (response.ok) {
+            window.location.href = '/dashboard/providers?success=1';
+        } else {
+            alert('Error saving configuration');
+        }
+    } catch (error) {
+        alert('Error: ' + error.message);
+    }
+}
+
 // Initialize providers list immediately (DOM is already loaded since script is at end of body)
 console.log('Initializing providers list...');
 console.log('providersData:', Object.keys(providersData));