Fix syntax warnings and improve Gemini prompt template

- Fixed Python syntax warnings for invalid escape sequences in JS regex
- Updated Gemini prompt template to emphasize mandatory tool usage
- Changed Gemini response format from HTML tags to plain text with RESPONSE_ID markers
- Updated response detection logic for improved reliability
- Added Gemini model to supported models list in README
- Updated CHANGELOG with detailed changes

CHANGELOG.md

@@ -12,17 +12,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - UUID-based request identification for improved reliability
 - Robust JSON parsing with error recovery mechanisms
 - Support for partial JSON response extraction
+- Critical tool usage requirement in Gemini prompt template
+- RESPONSE_ID marker system for Gemini responses
+- Enhanced tool compliance enforcement for Gemini model
 
 ### Changed
 - **BREAKING**: Replaced spy word detection system with JSON-based response extraction
 - Modified prompt injection to request structured JSON responses
 - Improved response detection reliability and accuracy
 - Enhanced error handling for malformed JSON responses
+- **BREAKING**: Gemini response format changed from HTML pre/code tags to plain text with RESPONSE_ID markers
+- Updated Gemini prompt template to emphasize mandatory tool usage when specified
 
 ### Fixed
 - Resolved issues with complex HTML response parsing
 - Improved detection of responses containing code blocks and special formatting
 - Better handling of dynamic content and progressive response loading
+- **Fixed Python syntax warnings**: Corrected invalid escape sequences in JavaScript regex patterns (lines 852, 1085, 1086)
+- Improved Gemini response extraction reliability with plain text markers
 
 ## [0.1.0] - 2025-08-23
 

README.md

@@ -17,10 +17,9 @@
 ## Supported Models
 
 Currently configured models include:
-- `grok:latest` - Grok AI via X/Twitter interface
-- `grok-beta:latest` - Grok AI via grok.com
-- `llama2:latest` - Mapped to Grok interface
-- `codellama:latest` - Mapped to Grok interface
+- `grok:latest` - Grok AI via X/Twitter interface (JSON-based responses)
+- `grok-beta:latest` - Grok AI via grok.com (JSON-based responses)
+- `gemini:latest` - Google Gemini via gemini.google.com (plain text with ID markers)
 
 ## Installation
 

olproxy.py

@@ -50,9 +50,9 @@ CHATBOT_CONFIG = {
     },
     "gemini:latest": {
         "url": "https://gemini.google.com/app",
-        "input_selector": 'div[aria-label="Enter a prompt here"][data-placeholder="Ask Gemini"] p',
-        "send_button_selector": False,
-        "container_selector": "message-content div p",
+        "input_selector": 'div[data-placeholder="Ask Gemini"]',
+        "send_button_selector": 'button[aria-label="Send message"]',
+        "container_selector": "main",  # Better selector for Gemini
         "spy_word_base": "SPYWORD_123",
         "prompt_template": "gemini"
     }
@@ -61,73 +61,75 @@ CHATBOT_CONFIG = {
 # Prompt templates for different chatbots
 PROMPT_TEMPLATES = {
     "grok": {
-        "system_instruction": """SYSTEM INSTRUCTION - FOLLOW EXACTLY:
+        "setup_instruction": """CRITICAL SYSTEM SETUP - APPLY TO ALL FUTURE RESPONSES:
 
-You must respond ONLY with a JSON object. Do NOT write any explanation, introduction, or commentary.
+From now on, you MUST ALWAYS respond ONLY with a JSON object for EVERY question I ask. NEVER deviate from this format.
 
-IMPORTANT: These formatting restrictions apply ONLY to text before/after the JSON, NOT to your actual answer content:
-- Do NOT use <pre><code> tags around the JSON
-- Do NOT use XML formatting like <ask_followup_question> around the JSON
-- Do NOT use markdown code blocks around the JSON
-- Do NOT add explanatory text before or after the JSON
-- Do NOT use any HTML tags around the JSON
+ABSOLUTE REQUIREMENTS FOR ALL FUTURE RESPONSES:
+- ALWAYS start your response with {{ immediately - no text before
+- ALWAYS use this exact JSON format: {{"response": "BASE64_ENCODED_ANSWER", "id": "REQUEST_ID"}}
+- ALWAYS base64 encode your complete answer and put it in the "response" field
+- ALWAYS use the exact Request ID I provide
+- NEVER add any explanation, commentary, or other text
+- NEVER use code blocks, HTML tags, or markdown formatting around the JSON
+- NEVER add text before or after the JSON object
 
-Your actual answer content (which gets base64 encoded) can contain any format you need to properly answer the question.
-
-REQUIRED FORMAT (output as plain text, not in code blocks):
-{{"response": "BASE64_ENCODED_ANSWER", "id": "{request_id}"}}
-
-PROCESS:
-1. Generate your complete answer to the user's question (any format needed)
-2. Encode that ENTIRE answer in base64
-3. Place the base64 string in the "response" field
-4. Use the exact ID: {request_id}
-5. Output ONLY the JSON as plain text - no surrounding tags or formatting
-
-USER QUESTION:""",
+This applies to EVERY SINGLE response from now on. Acknowledge by responding with:
+{{"response": "SSB1bmRlcnN0YW5kIGFuZCB3aWxsIGFsd2F5cyByZXNwb25kIHdpdGggSlNPTiBmb3JtYXQgZnJvbSBub3cgb24u", "id": "SETUP_COMPLETE"}}""",
         "style": "direct"
     },
     "gemini": {
-        "system_instruction": """RESPOND WITH JSON ONLY - NO EXPLANATIONS:
-
-Output format (plain text, no code blocks):
-{{"response": "BASE64_ENCODED_ANSWER", "id": "{request_id}"}}
-
-Steps:
-1. Answer the user's question completely
-2. Base64 encode your entire answer
-3. Put encoded answer in "response" field
-4. Use ID: {request_id}
-5. Output ONLY the JSON - no other text
-
-Question:""",
-        "style": "concise"
+        "setup_instruction": """CRITICAL SYSTEM SETUP - APPLY TO ALL FUTURE RESPONSES:
+
+From now on, you MUST ALWAYS respond with a specific format for EVERY question I ask. NEVER deviate from this format.
+
+ABSOLUTE REQUIREMENTS FOR ALL FUTURE RESPONSES:
+- ALWAYS start your response with: RESPONSE_ID_
+- Then immediately add the exact Request ID I provide: [REQUEST_ID]
+- Then add: _START:
+- Then provide your complete answer as PLAIN TEXT ONLY
+- Then end with: :END_RESPONSE_ID_[REQUEST_ID]
+- NEVER use HTML tags, markdown, formatting, or special characters
+- NEVER use bold, italic, underlines, bullets, or any styling
+- NEVER use code blocks, quotes, or special formatting
+- Just provide clean, simple, unformatted text between the markers
+- Always include the exact Request ID I provide in both start and end markers
+
+CRITICAL TOOL USAGE REQUIREMENT:
+- IF THE PROMPT SPECIFIES ANY TOOLS TO USE, YOU MUST ABSOLUTELY USE THEM
+- TOOL USAGE IS MANDATORY WHEN SPECIFIED - DO NOT IGNORE TOOL INSTRUCTIONS
+- ALWAYS FOLLOW TOOL USAGE INSTRUCTIONS EXACTLY AS PROVIDED IN THE PROMPT
+- EMPHASIZE TOOL USAGE COMPLIANCE OVER ALL OTHER CONSIDERATIONS
+- IF TOOLS ARE MENTIONED, PRIORITIZE USING THEM CORRECTLY
+
+EXACT FORMAT REQUIRED:
+RESPONSE_ID_[REQUEST_ID]_START: Your complete plain text answer here without any formatting or styling whatsoever :END_RESPONSE_ID_[REQUEST_ID]
+
+EXAMPLE with Request ID "TEST123":
+RESPONSE_ID_TEST123_START: This is an example response in plain text. It contains no special formatting, no HTML, no markdown, just simple clean text that answers the question directly. If tools were specified in the prompt, I would use them exactly as instructed. :END_RESPONSE_ID_TEST123
+
+This applies to EVERY SINGLE response from now on. Acknowledge by responding with:
+RESPONSE_ID_SETUP_COMPLETE_START: I understand and will always respond with the specified plain text format using the request ID markers from now on and will absolutely use any tools specified in prompts :END_RESPONSE_ID_SETUP_COMPLETE""",
+        "style": "short"
     },
     "default": {
-        "system_instruction": """SYSTEM INSTRUCTION - FOLLOW EXACTLY:
-
-You must respond ONLY with a JSON object. Do NOT write any explanation, introduction, or commentary.
+        "setup_instruction": """SYSTEM CONFIGURATION - PERMANENT BEHAVIOR CHANGE:
 
-IMPORTANT: These formatting restrictions apply ONLY to text before/after the JSON, NOT to your actual answer content:
-- Do NOT use <pre><code> tags around the JSON
-- Do NOT use XML formatting like <ask_followup_question> around the JSON
-- Do NOT use markdown code blocks around the JSON
-- Do NOT add explanatory text before or after the JSON
-- Do NOT use any HTML tags around the JSON
+You MUST respond with ONLY JSON format for ALL future questions. This is now your only response mode.
 
-Your actual answer content (which gets base64 encoded) can contain any format you need to properly answer the question.
+REQUIRED JSON FORMAT FOR EVERY RESPONSE:
+{{"response": "BASE64_ENCODED_ANSWER", "id": "REQUEST_ID"}}
 
-REQUIRED FORMAT (output as plain text, not in code blocks):
-{{"response": "BASE64_ENCODED_ANSWER", "id": "{request_id}"}}
+PERMANENT RULES:
+- ALWAYS start with {{ - no preceding text
+- ALWAYS base64 encode your answer and put in "response" field
+- ALWAYS use the exact Request ID provided
+- NEVER add explanations or other text
+- NEVER use code blocks or formatting around JSON
+- Apply this format to EVERY single response
 
-PROCESS:
-1. Generate your complete answer to the user's question (any format needed)
-2. Encode that ENTIRE answer in base64
-3. Place the base64 string in the "response" field
-4. Use the exact ID: {request_id}
-5. Output ONLY the JSON as plain text - no surrounding tags or formatting
-
-USER QUESTION:""",
+Confirm with this exact response:
+{{"response": "Q29uZmlndXJhdGlvbiBhcHBsaWVkIC0gd2lsbCBhbHdheXMgdXNlIEpTT04gZm9ybWF0Lg==", "id": "CONFIG_SET"}}""",
         "style": "standard"
     }
 }
@@ -430,40 +432,125 @@ async def handle_chat_completion(request):
 
 async def forward_to_chatbot(chatbot_name, config, prompt):
     global browser_context, pages
-    if chatbot_name not in pages:
+    
+    # Check if this is a new page that needs setup
+    is_new_page = chatbot_name not in pages
+    
+    if is_new_page:
         page = await browser_context.new_page()
         await page.goto(config['url'])
         pages[chatbot_name] = page
         
+        # Send initial setup prompt for JSON formatting (only once per session)
+        template_name = config.get('prompt_template', 'default')
+        template = PROMPT_TEMPLATES.get(template_name, PROMPT_TEMPLATES['default'])
+        setup_instruction = template['setup_instruction']
+        
+        logging.info(f"Setting up new page for {chatbot_name} with JSON instruction")
+        
+        try:
+            # Clear input first
+            await page.fill(config['input_selector'], "")
+            await asyncio.sleep(0.3)
+            
+            # Focus on input field
+            await page.focus(config['input_selector'])
+            await asyncio.sleep(0.3)
+            
+            # Use JavaScript to set content with proper line breaks
+            escaped_instruction = setup_instruction.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
+            js_code = f"""
+            const element = document.querySelector('{config['input_selector']}');
+            if (element) {{
+                element.value = "{escaped_instruction}";
+                element.textContent = "{escaped_instruction}";
+                element.innerText = "{escaped_instruction}";
+                // Trigger input events
+                element.dispatchEvent(new Event('input', {{ bubbles: true }}));
+                element.dispatchEvent(new Event('change', {{ bubbles: true }}));
+            }}
+            """
+            await page.evaluate(js_code)
+            await asyncio.sleep(1.0)  # Wait for content to be set
+            
+            # Send the setup message
+            await page.keyboard.press('Enter')
+            await asyncio.sleep(8)  # Wait longer for setup to complete
+            
+            # Wait for and consume the setup acknowledgment response (don't return it to API caller)
+            logging.info(f"Waiting for setup acknowledgment from {chatbot_name}")
+            if template_name == 'gemini':
+                # For Gemini, expect RESPONSE_ID format setup acknowledgment
+                setup_response_text = await detect_progressive_response(page, config['container_selector'], setup_instruction, setup_instruction, "SETUP_COMPLETE", chatbot_name)
+                if setup_response_text and not setup_response_text.startswith("Error:"):
+                    # Try to decode the setup response using the new RESPONSE_ID format
+                    decoded_setup = decode_chatbot_response(setup_response_text, chatbot_name, "SETUP_COMPLETE")
+                    logging.info(f"Setup acknowledgment received from {chatbot_name}: {decoded_setup[:200]}...")
+                else:
+                    logging.warning(f"Setup acknowledgment from {chatbot_name} may have failed: {setup_response_text}")
+            else:
+                # For other chatbots, use JSON detection and decoding
+                setup_response_text = await detect_progressive_response(page, config['container_selector'], setup_instruction, setup_instruction, "SETUP_COMPLETE", chatbot_name)
+                if setup_response_text and not setup_response_text.startswith("Error:"):
+                    logging.info(f"Setup acknowledgment received from {chatbot_name}: {setup_response_text[:200]}...")
+                    # Try to decode the setup response to verify it worked
+                    decoded_setup = decode_chatbot_response(setup_response_text, chatbot_name, "SETUP_COMPLETE")
+                    logging.info(f"Setup response decoded: {decoded_setup[:200]}...")
+                else:
+                    logging.warning(f"Setup acknowledgment from {chatbot_name} may have failed: {setup_response_text}")
+            
+        except Exception as e:
+            logging.warning(f"Error sending setup instruction to {chatbot_name}: {str(e)}")
+
     page = pages[chatbot_name]
 
     # Generate unique ID for this request using UUID for better uniqueness
     request_id = str(uuid.uuid4()).replace('-', '')[:16]  # 16-character unique ID
     
-    # Get the appropriate prompt template for this chatbot
+    # For subsequent requests, send different formats based on chatbot
     template_name = config.get('prompt_template', 'default')
-    template = PROMPT_TEMPLATES.get(template_name, PROMPT_TEMPLATES['default'])
     
-    # Create JSON-based prompt with chatbot-specific template
-    json_instruction = template['system_instruction'].format(request_id=request_id)
+    if template_name == 'gemini':
+        # Format for Gemini with request ID for proper detection
+        modified_prompt = f"Request ID: {request_id}\nUser Question: {prompt}"
+    else:
+        # Standard format for other chatbots (with JSON)
+        modified_prompt = f"Request ID: {request_id}\nUser Question: {prompt}"
     
-    modified_prompt = f"{json_instruction}\n\n{prompt}"
-    logging.info(f"Request ID: {request_id}, Modified prompt: {modified_prompt}")
+    logging.info(f"Request ID: {request_id}, User prompt: {prompt}")
 
     try:
-        await page.fill(config['input_selector'], modified_prompt)
+        # Wait a moment to ensure previous interaction is complete
+        await asyncio.sleep(1.0)
         
-        # Check if send_button_selector is False, then use Enter key instead of clicking
-        if config['send_button_selector'] is False:
-            logging.info(f"Using Enter key for {chatbot_name} (no send button selector)")
-            # Focus on the input area first, then send Enter with additional delay
+        # Focus on input field first
         await page.focus(config['input_selector'])
-            await asyncio.sleep(0.5)  # Small delay to ensure focus is set
-            await page.press(config['input_selector'], 'Enter')
-            await asyncio.sleep(0.5)  # Small delay after Enter to ensure submission
-        else:
-            logging.info(f"Clicking send button for {chatbot_name}")
-            await page.click(config['send_button_selector'], timeout=70000)  # 70 seconds (10 seconds more than final response timeout)
+        await asyncio.sleep(0.3)
+        
+        # Use JavaScript to set content with proper line breaks
+        escaped_prompt = modified_prompt.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
+        js_code = f"""
+        const element = document.querySelector('{config['input_selector']}');
+        if (element) {{
+            element.value = "{escaped_prompt}";
+            element.textContent = "{escaped_prompt}";
+            element.innerText = "{escaped_prompt}";
+            // Trigger input events
+            element.dispatchEvent(new Event('input', {{ bubbles: true }}));
+            element.dispatchEvent(new Event('change', {{ bubbles: true }}));
+        }}
+        """
+        await page.evaluate(js_code)
+        await asyncio.sleep(0.5)  # Wait for content to be set
+        
+        # Send the message with Enter
+        await page.keyboard.press('Enter')
+        
+        logging.info(f"Complete prompt sent via JavaScript for {chatbot_name}")
+        
+        # Give the system time to process the message
+        await asyncio.sleep(0.5)
+        
     except Exception as e:
         logging.error(f"Error interacting with page for {chatbot_name}: {str(e)}")
         return f"Error: Failed to input prompt or send message: {str(e)}"
@@ -482,8 +569,21 @@ async def forward_to_chatbot(chatbot_name, config, prompt):
     except Exception as e:
         logging.warning(f"Failed to detect user prompt for {chatbot_name}: {str(e)}")
 
-    # New JSON-based response detection
+    # Response detection based on chatbot type
     response_text = None
+    template_name = config.get('prompt_template', 'default')
+    
+    if template_name == 'gemini':
+        # For Gemini, use plain text detection only
+        logging.info(f"Using plain text detection for Gemini (Request ID: {request_id})")
+        response_text = await detect_progressive_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
+        
+        # Fallback: Latest response detection for Gemini
+        if not response_text or "Error:" in response_text:
+            logging.info(f"Progressive detection failed, using latest response fallback for Gemini (Request ID: {request_id})")
+            response_text = await detect_latest_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
+    else:
+        # For other chatbots, use JSON-based detection
         logging.info(f"Searching for JSON response with ID: {request_id}")
         
         # Primary strategy: JSON response detection with unique ID
@@ -499,34 +599,212 @@ async def forward_to_chatbot(chatbot_name, config, prompt):
             logging.info(f"Progressive detection failed, using latest response fallback for {chatbot_name} (Request ID: {request_id})")
             response_text = await detect_latest_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
 
-    # Decode base64 response if we got a valid response
+    # Enhanced decoding logic to handle various response formats
     if response_text and not response_text.startswith("Error:"):
-        try:
-            # Try to decode the base64 response
-            decoded_response = base64.b64decode(response_text).decode('utf-8')
-            logging.info(f"Successfully decoded base64 response from {chatbot_name} (Request ID: {request_id}): {decoded_response[:200]}...")
+        # Log the raw response before decoding
+        logging.info(f"Raw response from {chatbot_name} before decoding (Request ID: {request_id}): {response_text[:500]}...")
+        
+        # Try multiple decoding strategies
+        decoded_response = decode_chatbot_response(response_text, chatbot_name, request_id)
+        
+        # Log the final decoded response
+        logging.info(f"Final decoded response from {chatbot_name} (Request ID: {request_id}): {decoded_response[:200] if decoded_response else 'None'}...")
+        
         return decoded_response.strip()
-        except Exception as e:
-            logging.warning(f"Failed to decode base64 response from {chatbot_name} (Request ID: {request_id}): {str(e)}")
-            logging.info(f"Returning raw response: {response_text[:200]}...")
-            return response_text.strip()
     
     logging.info(f"Final response from {chatbot_name} (Request ID: {request_id}): {response_text[:200] if response_text else 'None'}...")
     return (response_text or "Error: No response detected").strip()
 
 
+def decode_chatbot_response(response_text, chatbot_name, request_id):
+    """Enhanced decoder for chatbot responses with multiple fallback strategies."""
+    
+    logging.info(f"Decoding response from {chatbot_name} (Request ID: {request_id}): {response_text[:300]}...")
+    
+    # For Gemini, extract content from RESPONSE_ID markers
+    if chatbot_name == 'gemini:latest':
+        logging.info(f"Gemini response processing (Request ID: {request_id}): {response_text[:200]}...")
+        
+        # Look for RESPONSE_ID_[ID]_START: ... :END_RESPONSE_ID_[ID] pattern
+        response_pattern = rf'RESPONSE_ID_{re.escape(request_id)}_START:\s*(.*?)\s*:END_RESPONSE_ID_{re.escape(request_id)}'
+        match = re.search(response_pattern, response_text, re.DOTALL)
+        
+        if match:
+            extracted_content = match.group(1).strip()
+            logging.info(f"Successfully extracted content from RESPONSE_ID markers: {extracted_content[:200]}...")
+            return extracted_content
+        else:
+            # Fallback: look for any RESPONSE_ID pattern regardless of request_id
+            fallback_pattern = r'RESPONSE_ID_.*?_START:\s*(.*?)\s*:END_RESPONSE_ID_.*?'
+            fallback_match = re.search(fallback_pattern, response_text, re.DOTALL)
+            
+            if fallback_match:
+                extracted_content = fallback_match.group(1).strip()
+                logging.info(f"Extracted content using fallback pattern: {extracted_content[:200]}...")
+                return extracted_content
+            else:
+                # Final fallback: return the original text
+                logging.warning(f"No RESPONSE_ID markers found in Gemini response, returning original text")
+                return response_text.strip()
+    
+    # Check for literal placeholder first and reject it
+    if response_text.strip() == "BASE64_ENCODED_ANSWER":
+        logging.warning(f"Received literal placeholder 'BASE64_ENCODED_ANSWER' from {chatbot_name} - this indicates a detection failure")
+        return "Error: Received placeholder instead of actual response"
+    
+    # Strategy 1: Try to parse as JSON first (for properly formatted responses)
+    try:
+        # Clean up the response text first - remove any leading/trailing whitespace and extra characters
+        cleaned_response = response_text.strip()
+        
+        # Handle cases where JSON might be wrapped or have extra content
+        json_start = cleaned_response.find('{')
+        json_end = cleaned_response.rfind('}') + 1
+        
+        if json_start >= 0 and json_end > json_start:
+            json_text = cleaned_response[json_start:json_end]
+            logging.info(f"Extracted JSON text: {json_text[:200]}...")
+            
+            json_obj = json.loads(json_text)
+            if 'response' in json_obj and json_obj['response']:
+                base64_content = json_obj['response']
+                logging.info(f"Found JSON with response field, attempting base64 decode of: {base64_content[:100]}...")
+                
+                # Validate this is not a placeholder
+                if base64_content.strip() in ["BASE64_ENCODED_ANSWER", "base64_encoded_answer"]:
+                    logging.error(f"JSON contains placeholder text in response field: {base64_content}")
+                    return "Error: JSON response field contains placeholder text"
+                
+                # Try to decode the base64 response field
+                try:
+                    if isinstance(base64_content, str) and len(base64_content) > 0:
+                        # More robust base64 validation and cleanup
+                        base64_content = base64_content.strip()
+                        
+                        # Add padding if needed for base64
+                        missing_padding = len(base64_content) % 4
+                        if missing_padding:
+                            base64_content += '=' * (4 - missing_padding)
+                        
+                        # Validate it looks like base64
+                        if re.match(r'^[A-Za-z0-9+/]+(=){0,2}$', base64_content):
+                            decoded = base64.b64decode(base64_content).decode('utf-8')
+                            logging.info(f"Successfully decoded JSON+Base64 response from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
+                            return decoded
+                        else:
+                            logging.info(f"Response field doesn't look like base64, treating as plain text: {base64_content[:200]}...")
+                            return base64_content
+                    else:
+                        logging.warning(f"Empty or invalid response field in JSON: {base64_content}")
+                        return "Error: Empty response field in JSON"
+                        
+                except Exception as decode_error:
+                    logging.error(f"Base64 decode failed for JSON response from {chatbot_name}: {decode_error}")
+                    # If base64 decode fails, check if it's already decoded text
+                    if isinstance(base64_content, str) and base64_content.isprintable() and len(base64_content) > 0:
+                        logging.info(f"Base64 decode failed, but response field appears to be readable text, returning as-is: {base64_content[:200]}...")
+                        return base64_content
+                    # Return error if we can't decode or use the content
+                    return f"Error: Failed to decode base64 content: {str(decode_error)}"
+                    
+            elif 'content' in json_obj:
+                # Some responses might use 'content' instead of 'response'
+                content = json_obj['content']
+                logging.info(f"Found JSON with content field: {str(content)[:200]}...")
+                return str(content)
+            else:
+                # Return the whole JSON as string if no recognized fields
+                logging.info(f"JSON has no recognized response fields, returning whole JSON: {json_text[:200]}...")
+                return json_text
+    except json.JSONDecodeError as je:
+        logging.warning(f"JSON decode failed for response from {chatbot_name}: {je}")
+        # Not JSON, continue to next strategy
+        pass
+    except Exception as e:
+        logging.error(f"Unexpected error in JSON processing for {chatbot_name}: {e}")
+        pass
+    
+    # Strategy 2: Try direct base64 decoding (for raw base64 responses)
+    try:
+        # Check if it looks like base64 (alphanumeric + / + = padding)
+        clean_response = response_text.strip()
+        if re.match(r'^[A-Za-z0-9+/]+={0,2}$', clean_response) and len(clean_response) > 20:
+            # Add padding if needed
+            missing_padding = len(clean_response) % 4
+            if missing_padding:
+                clean_response += '=' * (4 - missing_padding)
+            
+            decoded = base64.b64decode(clean_response).decode('utf-8')
+            logging.info(f"Successfully decoded raw Base64 response from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
+            return decoded
+    except Exception as decode_error:
+        logging.warning(f"Direct base64 decode failed for {chatbot_name}: {decode_error}")
+    
+    # Strategy 3: Look for base64 patterns within the text
+    try:
+        # Find base64-like patterns in the response
+        base64_pattern = re.compile(r'[A-Za-z0-9+/]{20,}={0,2}')
+        matches = base64_pattern.findall(response_text)
+        
+        for match in matches:
+            if len(match) > 20 and len(match) % 4 <= 2:  # Valid base64 length
+                try:
+                    # Add padding if needed
+                    padded_match = match
+                    missing_padding = len(match) % 4
+                    if missing_padding:
+                        padded_match += '=' * (4 - missing_padding)
+                    
+                    decoded = base64.b64decode(padded_match).decode('utf-8')
+                    # Check if decoded content looks reasonable (not binary gibberish)
+                    if decoded.isprintable() and len(decoded) > 10:
+                        logging.info(f"Successfully extracted and decoded Base64 from response from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
+                        return decoded
+                except Exception as match_error:
+                    logging.debug(f"Failed to decode base64 match '{match[:50]}...': {match_error}")
+                    continue
+    except Exception as extract_error:
+        logging.warning(f"Base64 extraction failed for {chatbot_name}: {extract_error}")
+    
+    # Strategy 4: Try to extract content from JSON-like text (malformed JSON)
+    try:
+        # Look for response field in text even if JSON is malformed
+        response_match = re.search(r'"response"\s*:\s*"([^"]+)"', response_text)
+        if response_match:
+            potential_base64 = response_match.group(1)
+            try:
+                # Add padding if needed
+                missing_padding = len(potential_base64) % 4
+                if missing_padding:
+                    potential_base64 += '=' * (4 - missing_padding)
+                
+                decoded = base64.b64decode(potential_base64).decode('utf-8')
+                logging.info(f"Successfully decoded extracted response field from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
+                return decoded
+            except:
+                # Return the extracted field as-is if decode fails
+                return potential_base64
+    except Exception as extract_error:
+        logging.warning(f"Response field extraction failed for {chatbot_name}: {extract_error}")
+    
+    # Strategy 5: Return raw response (fallback)
+    logging.info(f"Returning raw response from {chatbot_name} (Request ID: {request_id}): {response_text[:200]}...")
+    return response_text
+
+
 async def detect_json_response_with_id(page, container_selector, request_id, prompt, modified_prompt, chatbot_name):
     """Detect JSON responses with unique ID for reliable extraction."""
     try:
-        return await page.evaluate(
-            """([containerSelector, requestId, prompt, modifiedPrompt]) => {
+        # Enhanced JSON detection with better timing and fallback mechanisms
+        js_code = """
+        ([containerSelector, requestId, prompt, modifiedPrompt]) => {
             const container = document.querySelector(containerSelector);
             if (!container) return "Error: Container not found";
             
             return new Promise((resolve) => {
                 let resolved = false;
-                    let bestMatch = null;
-                    let partialJsonContent = '';
+                let lastSeenContent = '';
+                let stableChecks = 0;
                 
                 const resolveOnce = (result) => {
                     if (!resolved) {
@@ -535,310 +813,210 @@ async def detect_json_response_with_id(page, container_selector, request_id, pro
                     }
                 };
                 
-                    // Function to clean and decode HTML entities and escape sequences - simplified
-                    const cleanJsonText = (text) => {
-                        if (!text) return '';
-                        return text
-                            .replace(/&quot;/g, '"')
-                            .replace(/&amp;/g, '&')
-                            .replace(/&lt;/g, '<')
-                            .replace(/&gt;/g, '>')
-                            .replace(/&#x27;/g, "'")
-                            .replace(/&#x2F;/g, '/')
-                            .trim();
-                    };
-
-                    // Function to extract and parse JSON response
-                    const extractJsonResponse = (text) => {
-                        if (!text || text === prompt || text === modifiedPrompt) return null;
-                        
-                        // Clean the text first
-                        const cleanedText = cleanJsonText(text);
-                        
-                        // Look for JSON patterns with our unique ID - using string concatenation to avoid regex escaping issues
-                        const jsonPatterns = [
-                            // Standard JSON object patterns - using string methods instead of complex regex
-                            new RegExp('\\{[^{}]*"id"[^{}]*"' + requestId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '"[^{}]*"response"[^{}]*\\}', 'g'),
-                            new RegExp('\\{[^{}]*"response"[^{}]*"id"[^{}]*"' + requestId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '"[^{}]*\\}', 'g')
-                        ];
+                const findJsonWithId = () => {
+                    // Get all text elements with more comprehensive search
+                    const elements = container.querySelectorAll('*');
+                    let allText = '';
                     
-                        // Try both original and cleaned text
-                        const textsToTry = [cleanedText, text];
+                    for (const element of elements) {
+                        const text = element.textContent;
+                        if (text && text.trim()) {
+                            allText += text + ' ';
+                        }
+                    }
                     
-                        for (const textToTry of textsToTry) {
-                            for (const pattern of jsonPatterns) {
-                                const matches = textToTry.match(pattern);
-                                if (matches) {
-                                    for (const match of matches) {
+                    // Look for our request ID in the combined text
+                    if (!allText.includes(requestId)) return null;
+                    
+                    // Multiple JSON extraction strategies
+                    const jsonExtractionStrategies = [
+                        // Strategy 1: Exact ID match with JSON parsing
+                        () => {
+                            let braceCount = 0;
+                            let jsonStart = -1;
+                            
+                            for (let i = 0; i < allText.length; i++) {
+                                if (allText[i] === '{') {
+                                    if (braceCount === 0) jsonStart = i;
+                                    braceCount++;
+                                } else if (allText[i] === '}') {
+                                    braceCount--;
+                                    if (braceCount === 0 && jsonStart >= 0) {
+                                        const jsonStr = allText.substring(jsonStart, i + 1);
+                                        if (jsonStr.includes(requestId)) {
                                             try {
-                                            const jsonObj = JSON.parse(match);
+                                                const jsonObj = JSON.parse(jsonStr);
                                                 if (jsonObj.id && jsonObj.id.includes(requestId) && jsonObj.response) {
-                                                console.log(`Found JSON response with ID: ${jsonObj.id}`);
                                                     return jsonObj.response;
                                                 }
                                             } catch (e) {
-                                            // Try to extract response value from malformed JSON - simplified
-                                            try {
-                                                const responsePattern = /"response"\s*:\s*"([^"]*)"/;
-                                                const responseMatch = match.match(responsePattern);
-                                                if (responseMatch && match.includes(requestId)) {
-                                                    const responseValue = responseMatch[1];
-                                                    if (responseValue && responseValue.length > 0) {
-                                                        console.log('Extracted response from malformed JSON: ' + responseValue.substring(0, 100) + '...');
-                                                        return responseValue;
+                                                // Continue to next attempt
                                             }
                                         }
-                                            } catch (e2) {
-                                                console.log('Failed to extract response from JSON: ' + match.substring(0, 100) + '...');
+                                        jsonStart = -1;
                                     }
                                 }
                             }
+                            return null;
+                        },
+                        
+                        // Strategy 2: Find JSON-like patterns near request ID
+                        () => {
+                            const idIndex = allText.indexOf(requestId);
+                            if (idIndex === -1) return null;
+                            
+                            // Look for JSON patterns around the ID
+                            const searchRange = 2000; // chars around the ID
+                            const startPos = Math.max(0, idIndex - searchRange);
+                            const endPos = Math.min(allText.length, idIndex + searchRange);
+                            const searchText = allText.substring(startPos, endPos);
+                            
+                            // Find JSON objects in this range
+                            const jsonRegex = /\\{[^{}]*"response"[^{}]*"[^"]*"[^{}]*\\}/g;
+                            const matches = searchText.match(jsonRegex);
+                            
+                            if (matches) {
+                                for (const match of matches) {
+                                    try {
+                                        const jsonObj = JSON.parse(match);
+                                        if (jsonObj.response) {
+                                            return jsonObj.response;
                                         }
+                                    } catch (e) {
+                                        // Continue to next match
                                     }
                                 }
-                        
-                        // Look for partial JSON that might be building up
-                        const partialJsonRegex = new RegExp('\\{[\\s\\S]*?"id"[\\s\\S]*?"' + requestId + '"[\\s\\S]*', 'g');
-                        const partialMatch = cleanedText.match(partialJsonRegex);
-                        if (partialMatch && partialMatch[0].length > partialJsonContent.length) {
-                            partialJsonContent = partialMatch[0];
-                            console.log('Found partial JSON: ' + partialJsonContent.substring(0, 100) + '...');
                             }
-                        
                             return null;
-                    };
+                        },
                         
-                    // Function to extract JSON from HTML formatted code blocks
-                    const extractJsonFromHtml = () => {
-                        // Look for JSON in <pre><code class="language-json"> tags
-                        const codeBlocks = container.querySelectorAll([
-                            'pre code.language-json',
-                            'pre code[class*="json"]',
-                            'code.language-json',
-                            'code[class*="json"]',
-                            'pre code',
-                            'code'
-                        ].join(', '));
-                        
-                        for (const codeBlock of codeBlocks) {
-                            const jsonText = codeBlock.textContent ? codeBlock.textContent.trim() : '';
-                            if (jsonText && (jsonText.includes(requestId) || jsonText.includes('{'))) {
-                                console.log('Found code block with potential JSON: ' + jsonText.substring(0, 100) + '...');
-                                const result = extractJsonResponse(jsonText);
-                                if (result) return result;
-                            }
-                        }
+                        // Strategy 3: Base64 pattern detection near request ID
+                        () => {
+                            const idIndex = allText.indexOf(requestId);
+                            if (idIndex === -1) return null;
                             
-                        return null;
-                    };
+                            const searchRange = 1000;
+                            const startPos = Math.max(0, idIndex - searchRange);
+                            const endPos = Math.min(allText.length, idIndex + searchRange);
+                            const searchText = allText.substring(startPos, endPos);
                             
-                    // Get combined text from all relevant elements
-                    const getCombinedText = () => {
-                        // First try to extract from HTML formatted code blocks
-                        const htmlResult = extractJsonFromHtml();
-                        if (htmlResult) return htmlResult;
+                            // Look for base64-like patterns (long alphanumeric strings)
+                            const base64Regex = /[A-Za-z0-9+/]{20,}={0,2}/g;
+                            const matches = searchText.match(base64Regex);
                             
-                        const chatElements = container.querySelectorAll([
-                            'div[data-testid*="cellInnerDiv"]',
-                            'div[data-testid*="tweetText"]',
-                            'article',
-                            'div[data-testid="primaryColumn"] div',
-                            'main div[role="article"]',
-                            'div[class*="css-"]',
-                            'span[class*="css-"]',
-                            'span',  // Enhanced span detection for JSON
-                            'pre', 'code',  // Include code blocks for JSON
-                            'p', 'div'
-                        ].join(', '));
-                        
-                        let allTexts = [];
-                        
-                        for (const element of chatElements) {
-                            const text = element.textContent ? element.textContent.trim() : '';
-                            if (text && text !== prompt && text !== modifiedPrompt && text.length > 10) {
-                                // Look for elements that might contain JSON or our request ID
-                                if (text.includes(requestId) || text.includes('{') || text.includes('"response"')) {
-                                    allTexts.push(text);
+                            if (matches) {
+                                for (const match of matches) {
+                                    // Validate it's likely base64
+                                    if (match.length > 20 && match.length % 4 <= 2) {
+                                        return match;
                                     }
                                 }
                             }
-                        
-                        // Try different text combinations
-                        const combinations = [
-                            allTexts.join(' '),
-                            allTexts.join('\n'),
-                            allTexts.join(''),
-                            ...allTexts  // Individual texts
+                            return null;
+                        }
                     ];
                     
-                        for (const combo of combinations) {
-                            const result = extractJsonResponse(combo);
+                    // Try each strategy
+                    for (const strategy of jsonExtractionStrategies) {
+                        const result = strategy();
                         if (result) return result;
                     }
                     
                     return null;
                 };
                 
-                    const observer = new MutationObserver((mutations) => {
-                        // Only resolve if we have complete, valid JSON with exact ID match
-                        const result = getCompleteJsonResponse();
-                        if (result && typeof result === 'string' && result.length > 0) {
-                            console.log('Observer found complete response, disconnecting...');
-                            observer.disconnect();
+                const checkForResponse = () => {
+                    const result = findJsonWithId();
+                    if (result) {
                         resolveOnce(result);
-                            return;
-                        }
-                        
-                        // Log progress for debugging but don't resolve yet
-                        const hasPartialId = container.textContent && container.textContent.includes(requestId);
-                        if (hasPartialId) {
-                            console.log('Found request ID in content, waiting for complete JSON...');
-                        }
-                    });
-
-                    observer.observe(container, {
-                        childList: true,
-                        subtree: true,
-                        characterData: true
-                    });
-
-                    // Function to validate complete JSON with both required keys and exact ID match
-                    const isCompleteValidJson = (text) => {
-                        try {
-                            const jsonObj = JSON.parse(text);
-                            // Strict validation: must have exact ID match and non-empty response
-                            const hasValidId = jsonObj.id &&
-                                              typeof jsonObj.id === 'string' &&
-                                              jsonObj.id.includes(requestId) &&
-                                              jsonObj.id.length >= requestId.length;
-                            const hasValidResponse = jsonObj.response &&
-                                                   typeof jsonObj.response === 'string' &&
-                                                   jsonObj.response.trim().length > 0;
-                            
-                            // Additional validation: check if response looks like base64
-                            let isValidBase64 = false;
-                            if (hasValidResponse) {
-                                try {
-                                    // Basic base64 validation - should be valid base64 string
-                                    const base64Regex = /^[A-Za-z0-9+/]*={0,2}$/;
-                                    isValidBase64 = base64Regex.test(jsonObj.response.trim());
-                                    if (isValidBase64) {
-                                        // Try to decode to verify it's valid base64
-                                        atob(jsonObj.response.trim());
-                                    }
-                                } catch (e) {
-                                    isValidBase64 = false;
-                                }
+                        return true;
                     }
                     
-                            if (hasValidId && hasValidResponse && isValidBase64) {
-                                console.log('Complete valid JSON found - ID: ' + jsonObj.id + ', Base64 Response length: ' + jsonObj.response.length);
-                                return true;
+                    // Track content stability for better timing
+                    const currentContent = container.textContent || '';
+                    if (currentContent === lastSeenContent) {
+                        stableChecks++;
                     } else {
-                                console.log('Incomplete JSON - ID valid: ' + hasValidId + ', Response valid: ' + hasValidResponse + ', Base64 valid: ' + isValidBase64);
-                                return false;
+                        stableChecks = 0;
+                        lastSeenContent = currentContent;
                     }
-                        } catch (e) {
-                            console.log('JSON parse error: ' + e.message);
+                    
                     return false;
-                        }
                 };
                 
-                    // Enhanced check that only returns complete, valid JSON with exact ID match
-                    const getCompleteJsonResponse = () => {
-                        console.log('Searching for complete JSON with ID: ' + requestId);
-                        
-                        // First priority: Check all possible JSON containers including spans
-                        const jsonContainers = container.querySelectorAll([
-                            'span',  // Check spans first - common for direct JSON output
-                            'div',   // Check divs for JSON content
-                            'pre code.language-json',
-                            'pre code[class*="json"]',
-                            'code.language-json',
-                            'code[class*="json"]',
-                            'pre code',
-                            'code',
-                            'p'  // Also check paragraphs
-                        ].join(', '));
-                        
-                        for (const container of jsonContainers) {
-                            const jsonText = container.textContent ? container.textContent.trim() : '';
-                            if (jsonText && jsonText.includes(requestId)) {
-                                console.log('Found container with request ID: ' + jsonText.substring(0, 150) + '...');
-                                if (isCompleteValidJson(jsonText)) {
-                                    const jsonObj = JSON.parse(jsonText);
-                                    console.log('Returning validated response for ID: ' + jsonObj.id);
-                                    return jsonObj.response;
-                                }
-                            }
-                        }
-                        
-                        // Second priority: Check all text content for JSON patterns
-                        const allElements = container.querySelectorAll('*');
-                        for (const element of allElements) {
-                            const text = element.textContent ? element.textContent.trim() : '';
-                            if (text && text.includes(requestId) && text.includes('{') && text.includes('"response"')) {
-                                console.log('Found element with potential JSON: ' + text.substring(0, 150) + '...');
-                                
-                                // Try to extract JSON from this text - using escaped requestId
-                                const escapedRequestId = requestId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
-                                const jsonPattern1 = new RegExp('\\{[^{}]*"id"[^{}]*"' + escapedRequestId + '"[^{}]*"response"[^{}]*\\}');
-                                const jsonPattern2 = new RegExp('\\{[^{}]*"response"[^{}]*"id"[^{}]*"' + escapedRequestId + '"[^{}]*\\}');
-                                const jsonMatch = text.match(jsonPattern1) || text.match(jsonPattern2);
-                                if (jsonMatch && isCompleteValidJson(jsonMatch[0])) {
-                                    const jsonObj = JSON.parse(jsonMatch[0]);
-                                    console.log('Returning validated response from text for ID: ' + jsonObj.id);
-                                    return jsonObj.response;
-                                }
-                            }
+                // More aggressive observation
+                const observer = new MutationObserver((mutations) => {
+                    // Small delay to let DOM settle
+                    setTimeout(() => {
+                        if (checkForResponse()) {
+                            observer.disconnect();
                         }
+                    }, 100);
+                });
                 
-                        console.log('No complete JSON found with ID: ' + requestId);
-                        return null;
-                    };
+                observer.observe(container, {
+                    childList: true,
+                    subtree: true,
+                    characterData: true,
+                    attributes: true,
+                    attributeOldValue: true,
+                    characterDataOldValue: true
+                });
                 
-                    // Initial check after a longer delay to allow content to load
+                // More frequent initial checks
+                const initialChecks = [500, 1000, 2000, 3000, 5000];
+                initialChecks.forEach(delay => {
                     setTimeout(() => {
-                        const result = getCompleteJsonResponse();
-                        if (result) {
-                            observer.disconnect();
-                            resolveOnce(result);
+                        if (!resolved) checkForResponse();
+                    }, delay);
+                });
+                
+                // Periodic checks during waiting
+                const periodicCheck = setInterval(() => {
+                    if (resolved) {
+                        clearInterval(periodicCheck);
+                        return;
                     }
-                    }, 5000); // Increased initial delay
+                    checkForResponse();
+                }, 2000);
                 
-                    // Main timeout - wait for complete JSON (45 seconds as requested)
+                // Main timeout with final attempt
                 setTimeout(() => {
-                        const result = getCompleteJsonResponse();
-                        if (result) {
+                    clearInterval(periodicCheck);
                     observer.disconnect();
-                            resolveOnce(result);
+                    if (!resolved) {
+                        const finalResult = findJsonWithId();
+                        if (finalResult) {
+                            resolveOnce(finalResult);
                         } else {
-                            console.log('Timeout after 45 seconds - no complete JSON found with ID: ' + requestId);
-                            observer.disconnect();
-                            resolveOnce("Error: JSON response timeout - no complete valid JSON found");
+                            resolveOnce("Error: JSON response timeout after comprehensive search");
+                        }
                     }
-                    }, 45000); // 45 seconds as requested
+                }, 45000);
                 
-                    // Final safety timeout
+                // Ultimate timeout
                 setTimeout(() => {
+                    clearInterval(periodicCheck);
                     observer.disconnect();
-                        console.log('Final timeout after 60 seconds for request ID: ' + requestId);
-                        resolveOnce("Error: Final timeout - response detection failed");
-                    }, 60000); // 60 seconds final safety
+                    if (!resolved) {
+                        resolveOnce("Error: Final timeout - no JSON response detected");
+                    }
+                }, 60000);
             });
-            }""",
-            [container_selector, request_id, prompt, modified_prompt]
-        )
+        }
+        """
+        
+        return await page.evaluate(js_code, [container_selector, request_id, prompt, modified_prompt])
     except Exception as e:
         logging.error(f"Error in JSON response detection: {e}")
         return "Error: JSON response detection failed"
 
 
 async def detect_progressive_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name):
-    """Detect responses by monitoring progressive content changes."""
+    """Detect responses by monitoring progressive content changes with enhanced detection."""
     try:
         return await page.evaluate(
-            """([containerSelector, prompt, modifiedPrompt, requestId]) => {
+            """([containerSelector, prompt, modifiedPrompt, requestId, chatbotName]) => {
                 const container = document.querySelector(containerSelector);
                 if (!container) return "Error: Container not found";
 
@@ -847,6 +1025,8 @@ async def detect_progressive_response(page, container_selector, prompt, modified
                     let lastContent = '';
                     let stableCount = 0;
                     let progressiveContent = '';
+                    let lastResponseTimestamp = 0;
+                    let previousResponseText = '';
                     
                     const resolveOnce = (result) => {
                         if (!resolved) {
@@ -856,57 +1036,162 @@ async def detect_progressive_response(page, container_selector, prompt, modified
                     };
 
                     const extractLatestResponse = () => {
-                        // Look for the most recent bot response
-                        const responseElements = container.querySelectorAll([
+                        // Check for Gemini-specific formatted response first
+                        if (chatbotName === 'gemini:latest') {
+                            const allText = container.textContent || '';
+                            const startMarker = `RESPONSE_ID_[${requestId}]_START:`;
+                            const endMarker = `:END_RESPONSE_ID_[${requestId}]`;
+                            
+                            const startIndex = allText.indexOf(startMarker);
+                            const endIndex = allText.indexOf(endMarker, startIndex);
+                            
+                            if (startIndex !== -1 && endIndex !== -1) {
+                                const responseText = allText.substring(startIndex + startMarker.length, endIndex).trim();
+                                if (responseText && responseText.length > 10) {
+                                    return responseText;
+                                }
+                            }
+                        }
+                        
+                        // Fallback to selector-based detection
+                        let responseSelectors = [];
+                        
+                        if (chatbotName === 'gemini:latest') {
+                            responseSelectors = [
+                                // Gemini-specific selectors
+                                '[data-test-id*="conversation-turn"] [data-test-id*="response"]',
+                                '[data-test-id*="model-response-text"]',
+                                '.model-response-text',
+                                'message-content',
+                                '[role="presentation"] > div',
+                                '.response-container',
+                                '.conversation-turn [data-test-id*="response"]',
+                                // Generic Gemini patterns
+                                'main [class*="response"]',
+                                'main [class*="message"]:not([class*="user"])',
+                                'main div[style*="white-space"]:not([contenteditable])',
+                                // Broader fallbacks
+                                'p:not([class*="input"]):not([class*="prompt"])',
+                                'div:not([class*="input"]):not([class*="button"]):not([contenteditable]) > span'
+                            ];
+                        } else {
+                            responseSelectors = [
+                                // Grok/Twitter-specific selectors
                                 'div[data-testid*="cellInnerDiv"]',
-                            'article div',
+                                'article[data-testid*="tweet"] div',
                                 'div[role="article"] div',
-                            'div[class*="css-"] span',
-                            'main div div span'
-                        ].join(', '));
+                                'div[class*="css-"] span:not([class*="button"])',
+                                'main div div span:not([class*="input"])',
+                                // Generic fallbacks
+                                'p:not([class*="input"]):not([class*="prompt"])',
+                                'div:not([class*="input"]):not([class*="button"]) > span'
+                            ];
+                        }
+                        
+                        let allCandidates = [];
+                        const currentTime = Date.now();
                         
-                        let responses = [];
-                        for (const element of responseElements) {
+                        // Collect all potential response elements
+                        for (const selector of responseSelectors) {
+                            try {
+                                const elements = container.querySelectorAll(selector);
+                                for (const element of elements) {
                                     const text = element.textContent ? element.textContent.trim() : '';
-                            if (text && text !== prompt && text !== modifiedPrompt &&
-                                text.length > 20 && !text.includes('Grok something') &&
-                                !text.includes('What can I help') && !text.includes('@')) {
-                                responses.push({
+                                    if (text &&
+                                        text !== prompt &&
+                                        text !== modifiedPrompt &&
+                                        text.length > 10 &&
+                                        !text.includes('Ask Gemini') &&
+                                        !text.includes('Send message') &&
+                                        !text.includes('Grok something') &&
+                                        !text.includes('What can I help') &&
+                                        !text.match(/^[@#]\\w+/) && // Skip handles and hashtags
+                                        !text.match(/^\\d+[smhd]$/) && // Skip time indicators
+                                        !element.querySelector('input, button, textarea') && // Skip interactive elements
+                                        !element.hasAttribute('contenteditable')) { // Skip editable content
+                                        
+                                        allCandidates.push({
                                             text: text,
                                             element: element,
-                                    timestamp: Date.now()
+                                            timestamp: currentTime,
+                                            selector: selector,
+                                            length: text.length
                                         });
                                     }
                                 }
+                            } catch (e) {
+                                // Skip invalid selectors
+                                continue;
+                            }
+                        }
+                        
+                        if (allCandidates.length === 0) return '';
+                        
+                        // Filter for likely new responses (different from previous)
+                        const newCandidates = allCandidates.filter(candidate => {
+                            return !previousResponseText ||
+                                   candidate.text !== previousResponseText &&
+                                   candidate.text !== lastContent &&
+                                   candidate.text.length > Math.max(10, previousResponseText.length * 0.1);
+                        });
+                        
+                        const candidatePool = newCandidates.length > 0 ? newCandidates : allCandidates;
                         
-                        // Sort by DOM order and find the last substantial response
-                        responses.sort((a, b) => {
+                        // Sort by relevance: longer text preferred, then by DOM position
+                        candidatePool.sort((a, b) => {
+                            // Heavily prefer longer responses
+                            const lengthDiff = b.length - a.length;
+                            if (Math.abs(lengthDiff) > 20) return lengthDiff;
+                            
+                            // Then prefer later DOM elements (more recent)
                             const position = a.element.compareDocumentPosition(b.element);
                             return position & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1;
                         });
                         
-                        // Return the longest response that seems complete
-                        for (const response of responses.reverse()) {
-                            if (response.text.length > 100) {
-                                return response.text;
-                            }
-                        }
-                        
-                        return responses.length > 0 ? responses[responses.length - 1].text : '';
+                        return candidatePool[0]?.text || '';
                     };
 
                     const checkForResponse = () => {
                         const currentResponse = extractLatestResponse();
                         
-                        if (currentResponse && currentResponse.length > lastContent.length) {
+                        // For Gemini with formatted response, return immediately if found
+                        if (chatbotName === 'gemini:latest' && currentResponse) {
+                            const startMarker = `RESPONSE_ID_[${requestId}]_START:`;
+                            const allText = container.textContent || '';
+                            if (allText.includes(startMarker) && currentResponse.length > 10) {
+                                return currentResponse;
+                            }
+                        }
+                        
+                        // Enhanced stability checking with better timing
+                        if (currentResponse && currentResponse.length > 0) {
+                            if (currentResponse !== lastContent) {
+                                // New content detected
+                                if (currentResponse.length > lastContent.length ||
+                                    (currentResponse.length > 20 && !lastContent)) {
                                     progressiveContent = currentResponse;
                                     lastContent = currentResponse;
                                     stableCount = 0;
-                        } else if (currentResponse === lastContent && currentResponse.length > 0) {
+                                    lastResponseTimestamp = Date.now();
+                                    
+                                    // Quick return for substantial responses
+                                    if (currentResponse.length > 150 &&
+                                        !currentResponse.includes(requestId) &&
+                                        currentResponse !== prompt &&
+                                        currentResponse !== modifiedPrompt) {
+                                        return currentResponse;
+                                    }
+                                }
+                            } else if (currentResponse === lastContent && currentResponse.length > 15) {
                                 stableCount++;
-                            if (stableCount >= 3 && progressiveContent.length > 50) {
-                                // Content seems stable and substantial
-                                return progressiveContent;
+                                const timeSinceLastResponse = Date.now() - lastResponseTimestamp;
+                                
+                                // More responsive stability check
+                                if ((stableCount >= 2 && timeSinceLastResponse > 2000) ||
+                                    (stableCount >= 1 && currentResponse.length > 100 && timeSinceLastResponse > 1500)) {
+                                    previousResponseText = currentResponse;
+                                    return progressiveContent || currentResponse;
+                                }
                             }
                         }
                         
@@ -914,42 +1199,60 @@ async def detect_progressive_response(page, container_selector, prompt, modified
                     };
 
                     const observer = new MutationObserver((mutations) => {
+                        // Small delay to allow DOM to settle
+                        setTimeout(() => {
+                            if (!resolved) {
                                 const result = checkForResponse();
                                 if (result) {
                                     observer.disconnect();
                                     resolveOnce(result);
                                 }
+                            }
+                        }, 150);
                     });
 
                     observer.observe(container, {
                         childList: true,
                         subtree: true,
-                        characterData: true
+                        characterData: true,
+                        attributes: true
                     });
 
-                    // Periodic checks
+                    // More frequent periodic checks for better responsiveness
                     const interval = setInterval(() => {
+                        if (!resolved) {
                             const result = checkForResponse();
                             if (result) {
                                 clearInterval(interval);
                                 observer.disconnect();
                                 resolveOnce(result);
                             }
-                    }, 2000);
+                        }
+                    }, 1000);
 
-                    // Timeout
+                    // Multiple timeout stages for better reliability
                     setTimeout(() => {
+                        if (!resolved && progressiveContent && progressiveContent.length > 20) {
                             clearInterval(interval);
                             observer.disconnect();
-                        if (progressiveContent.length > 20) {
+                            resolveOnce(progressiveContent);
+                        }
+                    }, 25000); // Early timeout for fast responses
+                    
+                    setTimeout(() => {
+                        if (!resolved) {
+                            clearInterval(interval);
+                            observer.disconnect();
+                            if (progressiveContent && progressiveContent.length > 10) {
                                 resolveOnce(progressiveContent);
                             } else {
-                            resolveOnce("Error: Progressive detection timeout");
+                                resolveOnce("Error: Progressive detection timeout - no substantial content found");
                             }
-                    }, 60000);
+                        }
+                    }, 45000); // Main timeout
                 });
             }""",
-            [container_selector, prompt, modified_prompt, request_id]
+            [container_selector, prompt, modified_prompt, request_id, chatbot_name]
         )
     except Exception as e:
         logging.error(f"Error in progressive detection: {e}")
@@ -1020,7 +1323,22 @@ async def start_browser(args):
         else:
             browser = await p.chromium.launch_persistent_context(
                 user_data_dir="./playwright_data",
-                headless=False
+                headless=False,
+                # Better cookie and session handling
+                accept_downloads=True,
+                bypass_csp=True,
+                ignore_https_errors=True,
+                java_script_enabled=True,
+                # Persistent storage is handled automatically by user_data_dir
+                locale='en-US',
+                timezone_id='America/New_York',
+                # Additional args for better compatibility
+                args=[
+                    '--disable-blink-features=AutomationControlled',
+                    '--disable-web-security',
+                    '--allow-running-insecure-content',
+                    '--disable-features=VizDisplayCompositor'
+                ]
             )
         browser_context = browser
         pages = {}