Skip to content

olproxy
olproxy
Commit 62fc2340 authored by nextime's avatar nextime
Browse files
Options

Fix syntax warnings and improve Gemini prompt template 

- Fixed Python syntax warnings for invalid escape sequences in JS regex
- Updated Gemini prompt template to emphasize mandatory tool usage
- Changed Gemini response format from HTML tags to plain text with RESPONSE_ID markers
- Updated response detection logic for improved reliability
- Added Gemini model to supported models list in README
- Updated CHANGELOG with detailed changes
parent 437b2d53
Hide whitespace changes
Inline Side-by-side
Showing with 777 additions and 453 deletions
CHANGELOG.md
View file @ 62fc2340
......@@ -12,17 +12,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- UUID-based request identification for improved reliability
- Robust JSON parsing with error recovery mechanisms
- Support for partial JSON response extraction
- Critical tool usage requirement in Gemini prompt template
- RESPONSE_ID marker system for Gemini responses
- Enhanced tool compliance enforcement for Gemini model
### Changed
- **BREAKING**: Replaced spy word detection system with JSON-based response extraction
- Modified prompt injection to request structured JSON responses
- Improved response detection reliability and accuracy
- Enhanced error handling for malformed JSON responses
- **BREAKING**: Gemini response format changed from HTML pre/code tags to plain text with RESPONSE_ID markers
- Updated Gemini prompt template to emphasize mandatory tool usage when specified
### Fixed
- Resolved issues with complex HTML response parsing
- Improved detection of responses containing code blocks and special formatting
- Better handling of dynamic content and progressive response loading
- **Fixed Python syntax warnings**: Corrected invalid escape sequences in JavaScript regex patterns (lines 852, 1085, 1086)
- Improved Gemini response extraction reliability with plain text markers
## [0.1.0] - 2025-08-23
......
README.md
View file @ 62fc2340
......@@ -17,10 +17,9 @@
## Supported Models
Currently configured models include:
- `grok:latest` - Grok AI via X/Twitter interface
- `grok-beta:latest` - Grok AI via grok.com
- `llama2:latest` - Mapped to Grok interface
- `codellama:latest` - Mapped to Grok interface
- `grok:latest` - Grok AI via X/Twitter interface (JSON-based responses)
- `grok-beta:latest` - Grok AI via grok.com (JSON-based responses)
- `gemini:latest` - Google Gemini via gemini.google.com (plain text with ID markers)
## Installation
......
olproxy.py
View file @ 62fc2340
......@@ -50,9 +50,9 @@ CHATBOT_CONFIG = {
},
"gemini:latest": {
"url": "https://gemini.google.com/app",
"input_selector": 'div[aria-label="Enter a prompt here"][data-placeholder="Ask Gemini"] p',
"send_button_selector": False,
"container_selector": "message-content div p",
"input_selector": 'div[data-placeholder="Ask Gemini"]',
"send_button_selector": 'button[aria-label="Send message"]',
"container_selector": "main", # Better selector for Gemini
"spy_word_base": "SPYWORD_123",
"prompt_template": "gemini"
}
......@@ -61,73 +61,75 @@ CHATBOT_CONFIG = {
# Prompt templates for different chatbots
PROMPT_TEMPLATES = {
"grok": {
"system_instruction": """SYSTEM INSTRUCTION - FOLLOW EXACTLY:
"setup_instruction": """CRITICAL SYSTEM SETUP - APPLY TO ALL FUTURE RESPONSES:
You must respond ONLY with a JSON object. Do NOT write any explanation, introduction, or commentary.
From now on, you MUST ALWAYS respond ONLY with a JSON object for EVERY question I ask. NEVER deviate from this format.
IMPORTANT: These formatting restrictions apply ONLY to text before/after the JSON, NOT to your actual answer content:
- Do NOT use <pre><code> tags around the JSON
- Do NOT use XML formatting like <ask_followup_question> around the JSON
- Do NOT use markdown code blocks around the JSON
- Do NOT add explanatory text before or after the JSON
- Do NOT use any HTML tags around the JSON
ABSOLUTE REQUIREMENTS FOR ALL FUTURE RESPONSES:
- ALWAYS start your response with {{ immediately - no text before
- ALWAYS use this exact JSON format: {{"response": "BASE64_ENCODED_ANSWER", "id": "REQUEST_ID"}}
- ALWAYS base64 encode your complete answer and put it in the "response" field
- ALWAYS use the exact Request ID I provide
- NEVER add any explanation, commentary, or other text
- NEVER use code blocks, HTML tags, or markdown formatting around the JSON
- NEVER add text before or after the JSON object
Your actual answer content (which gets base64 encoded) can contain any format you need to properly answer the question.
REQUIRED FORMAT (output as plain text, not in code blocks):
{{"response": "BASE64_ENCODED_ANSWER", "id": "{request_id}"}}
PROCESS:
1. Generate your complete answer to the user's question (any format needed)
2. Encode that ENTIRE answer in base64
3. Place the base64 string in the "response" field
4. Use the exact ID: {request_id}
5. Output ONLY the JSON as plain text - no surrounding tags or formatting
USER QUESTION:""",
This applies to EVERY SINGLE response from now on. Acknowledge by responding with:
{{"response": "SSB1bmRlcnN0YW5kIGFuZCB3aWxsIGFsd2F5cyByZXNwb25kIHdpdGggSlNPTiBmb3JtYXQgZnJvbSBub3cgb24u", "id": "SETUP_COMPLETE"}}""",
"style": "direct"
},
"gemini": {
"system_instruction": """RESPOND WITH JSON ONLY - NO EXPLANATIONS:
Output format (plain text, no code blocks):
{{"response": "BASE64_ENCODED_ANSWER", "id": "{request_id}"}}
Steps:
1. Answer the user's question completely
2. Base64 encode your entire answer
3. Put encoded answer in "response" field
4. Use ID: {request_id}
5. Output ONLY the JSON - no other text
Question:""",
"style": "concise"
"setup_instruction": """CRITICAL SYSTEM SETUP - APPLY TO ALL FUTURE RESPONSES:
From now on, you MUST ALWAYS respond with a specific format for EVERY question I ask. NEVER deviate from this format.
ABSOLUTE REQUIREMENTS FOR ALL FUTURE RESPONSES:
- ALWAYS start your response with: RESPONSE_ID_
- Then immediately add the exact Request ID I provide: [REQUEST_ID]
- Then add: _START:
- Then provide your complete answer as PLAIN TEXT ONLY
- Then end with: :END_RESPONSE_ID_[REQUEST_ID]
- NEVER use HTML tags, markdown, formatting, or special characters
- NEVER use bold, italic, underlines, bullets, or any styling
- NEVER use code blocks, quotes, or special formatting
- Just provide clean, simple, unformatted text between the markers
- Always include the exact Request ID I provide in both start and end markers
CRITICAL TOOL USAGE REQUIREMENT:
- IF THE PROMPT SPECIFIES ANY TOOLS TO USE, YOU MUST ABSOLUTELY USE THEM
- TOOL USAGE IS MANDATORY WHEN SPECIFIED - DO NOT IGNORE TOOL INSTRUCTIONS
- ALWAYS FOLLOW TOOL USAGE INSTRUCTIONS EXACTLY AS PROVIDED IN THE PROMPT
- EMPHASIZE TOOL USAGE COMPLIANCE OVER ALL OTHER CONSIDERATIONS
- IF TOOLS ARE MENTIONED, PRIORITIZE USING THEM CORRECTLY
EXACT FORMAT REQUIRED:
RESPONSE_ID_[REQUEST_ID]_START: Your complete plain text answer here without any formatting or styling whatsoever :END_RESPONSE_ID_[REQUEST_ID]
EXAMPLE with Request ID "TEST123":
RESPONSE_ID_TEST123_START: This is an example response in plain text. It contains no special formatting, no HTML, no markdown, just simple clean text that answers the question directly. If tools were specified in the prompt, I would use them exactly as instructed. :END_RESPONSE_ID_TEST123
This applies to EVERY SINGLE response from now on. Acknowledge by responding with:
RESPONSE_ID_SETUP_COMPLETE_START: I understand and will always respond with the specified plain text format using the request ID markers from now on and will absolutely use any tools specified in prompts :END_RESPONSE_ID_SETUP_COMPLETE""",
"style": "short"
},
"default": {
"system_instruction": """SYSTEM INSTRUCTION - FOLLOW EXACTLY:
You must respond ONLY with a JSON object. Do NOT write any explanation, introduction, or commentary.
IMPORTANT: These formatting restrictions apply ONLY to text before/after the JSON, NOT to your actual answer content:
- Do NOT use <pre><code> tags around the JSON
- Do NOT use XML formatting like <ask_followup_question> around the JSON
- Do NOT use markdown code blocks around the JSON
- Do NOT add explanatory text before or after the JSON
- Do NOT use any HTML tags around the JSON
"setup_instruction": """SYSTEM CONFIGURATION - PERMANENT BEHAVIOR CHANGE:
Your actual answer content (which gets base64 encoded) can contain any format you need to properly answer the question.
You MUST respond with ONLY JSON format for ALL future questions. This is now your only response mode.
REQUIRED FORMAT (output as plain text, not in code blocks):
{{"response": "BASE64_ENCODED_ANSWER", "id": "{request_id}"}}
REQUIRED JSON FORMAT FOR EVERY RESPONSE:
{{"response": "BASE64_ENCODED_ANSWER", "id": "REQUEST_ID"}}
PROCESS:
1. Generate your complete answer to the user's question (any format needed)
2. Encode that ENTIRE answer in base64
3. Place the base64 string in the "response" field
4. Use the exact ID: {request_id}
5. Output ONLY the JSON as plain text - no surrounding tags or formatting
PERMANENT RULES:
- ALWAYS start with {{ - no preceding text
- ALWAYS base64 encode your answer and put in "response" field
- ALWAYS use the exact Request ID provided
- NEVER add explanations or other text
- NEVER use code blocks or formatting around JSON
- Apply this format to EVERY single response
USER QUESTION:""",
Confirm with this exact response:
{{"response": "Q29uZmlndXJhdGlvbiBhcHBsaWVkIC0gd2lsbCBhbHdheXMgdXNlIEpTT04gZm9ybWF0Lg==", "id": "CONFIG_SET"}}""",
"style": "standard"
}
}
......@@ -430,40 +432,125 @@ async def handle_chat_completion(request):
async def forward_to_chatbot(chatbot_name, config, prompt):
global browser_context, pages
if chatbot_name not in pages:
# Check if this is a new page that needs setup
is_new_page = chatbot_name not in pages
if is_new_page:
page = await browser_context.new_page()
await page.goto(config['url'])
pages[chatbot_name] = page
# Send initial setup prompt for JSON formatting (only once per session)
template_name = config.get('prompt_template', 'default')
template = PROMPT_TEMPLATES.get(template_name, PROMPT_TEMPLATES['default'])
setup_instruction = template['setup_instruction']
logging.info(f"Setting up new page for {chatbot_name} with JSON instruction")
try:
# Clear input first
await page.fill(config['input_selector'], "")
await asyncio.sleep(0.3)
# Focus on input field
await page.focus(config['input_selector'])
await asyncio.sleep(0.3)
# Use JavaScript to set content with proper line breaks
escaped_instruction = setup_instruction.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
js_code = f"""
const element = document.querySelector('{config['input_selector']}');
if (element) {{
element.value = "{escaped_instruction}";
element.textContent = "{escaped_instruction}";
element.innerText = "{escaped_instruction}";
// Trigger input events
element.dispatchEvent(new Event('input', {{ bubbles: true }}));
element.dispatchEvent(new Event('change', {{ bubbles: true }}));
}}
"""
await page.evaluate(js_code)
await asyncio.sleep(1.0) # Wait for content to be set
# Send the setup message
await page.keyboard.press('Enter')
await asyncio.sleep(8) # Wait longer for setup to complete
# Wait for and consume the setup acknowledgment response (don't return it to API caller)
logging.info(f"Waiting for setup acknowledgment from {chatbot_name}")
if template_name == 'gemini':
# For Gemini, expect RESPONSE_ID format setup acknowledgment
setup_response_text = await detect_progressive_response(page, config['container_selector'], setup_instruction, setup_instruction, "SETUP_COMPLETE", chatbot_name)
if setup_response_text and not setup_response_text.startswith("Error:"):
# Try to decode the setup response using the new RESPONSE_ID format
decoded_setup = decode_chatbot_response(setup_response_text, chatbot_name, "SETUP_COMPLETE")
logging.info(f"Setup acknowledgment received from {chatbot_name}: {decoded_setup[:200]}...")
else:
logging.warning(f"Setup acknowledgment from {chatbot_name} may have failed: {setup_response_text}")
else:
# For other chatbots, use JSON detection and decoding
setup_response_text = await detect_progressive_response(page, config['container_selector'], setup_instruction, setup_instruction, "SETUP_COMPLETE", chatbot_name)
if setup_response_text and not setup_response_text.startswith("Error:"):
logging.info(f"Setup acknowledgment received from {chatbot_name}: {setup_response_text[:200]}...")
# Try to decode the setup response to verify it worked
decoded_setup = decode_chatbot_response(setup_response_text, chatbot_name, "SETUP_COMPLETE")
logging.info(f"Setup response decoded: {decoded_setup[:200]}...")
else:
logging.warning(f"Setup acknowledgment from {chatbot_name} may have failed: {setup_response_text}")
except Exception as e:
logging.warning(f"Error sending setup instruction to {chatbot_name}: {str(e)}")
page = pages[chatbot_name]
# Generate unique ID for this request using UUID for better uniqueness
request_id = str(uuid.uuid4()).replace('-', '')[:16] # 16-character unique ID
# Get the appropriate prompt template for this chatbot
# For subsequent requests, send different formats based on chatbot
template_name = config.get('prompt_template', 'default')
template = PROMPT_TEMPLATES.get(template_name, PROMPT_TEMPLATES['default'])
# Create JSON-based prompt with chatbot-specific template
json_instruction = template['system_instruction'].format(request_id=request_id)
if template_name == 'gemini':
# Format for Gemini with request ID for proper detection
modified_prompt = f"Request ID: {request_id}\nUser Question: {prompt}"
else:
# Standard format for other chatbots (with JSON)
modified_prompt = f"Request ID: {request_id}\nUser Question: {prompt}"
modified_prompt = f"{json_instruction}\n\n{prompt}"
logging.info(f"Request ID: {request_id}, Modified prompt: {modified_prompt}")
logging.info(f"Request ID: {request_id}, User prompt: {prompt}")
try:
await page.fill(config['input_selector'], modified_prompt)
# Wait a moment to ensure previous interaction is complete
await asyncio.sleep(1.0)
# Focus on input field first
await page.focus(config['input_selector'])
await asyncio.sleep(0.3)
# Use JavaScript to set content with proper line breaks
escaped_prompt = modified_prompt.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
js_code = f"""
const element = document.querySelector('{config['input_selector']}');
if (element) {{
element.value = "{escaped_prompt}";
element.textContent = "{escaped_prompt}";
element.innerText = "{escaped_prompt}";
// Trigger input events
element.dispatchEvent(new Event('input', {{ bubbles: true }}));
element.dispatchEvent(new Event('change', {{ bubbles: true }}));
}}
"""
await page.evaluate(js_code)
await asyncio.sleep(0.5) # Wait for content to be set
# Send the message with Enter
await page.keyboard.press('Enter')
logging.info(f"Complete prompt sent via JavaScript for {chatbot_name}")
# Give the system time to process the message
await asyncio.sleep(0.5)
# Check if send_button_selector is False, then use Enter key instead of clicking
if config['send_button_selector'] is False:
logging.info(f"Using Enter key for {chatbot_name} (no send button selector)")
# Focus on the input area first, then send Enter with additional delay
await page.focus(config['input_selector'])
await asyncio.sleep(0.5) # Small delay to ensure focus is set
await page.press(config['input_selector'], 'Enter')
await asyncio.sleep(0.5) # Small delay after Enter to ensure submission
else:
logging.info(f"Clicking send button for {chatbot_name}")
await page.click(config['send_button_selector'], timeout=70000) # 70 seconds (10 seconds more than final response timeout)
except Exception as e:
logging.error(f"Error interacting with page for {chatbot_name}: {str(e)}")
return f"Error: Failed to input prompt or send message: {str(e)}"
......@@ -482,363 +569,454 @@ async def forward_to_chatbot(chatbot_name, config, prompt):
except Exception as e:
logging.warning(f"Failed to detect user prompt for {chatbot_name}: {str(e)}")
# New JSON-based response detection
# Response detection based on chatbot type
response_text = None
logging.info(f"Searching for JSON response with ID: {request_id}")
# Primary strategy: JSON response detection with unique ID
response_text = await detect_json_response_with_id(page, container_selector, request_id, prompt, modified_prompt, chatbot_name)
template_name = config.get('prompt_template', 'default')
# Fallback strategy: If JSON detection fails, try progressive content detection
if not response_text or "Error:" in response_text:
logging.info(f"JSON detection failed, trying progressive content detection for {chatbot_name} (Request ID: {request_id})")
if template_name == 'gemini':
# For Gemini, use plain text detection only
logging.info(f"Using plain text detection for Gemini (Request ID: {request_id})")
response_text = await detect_progressive_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
# Final fallback: Latest response detection
if not response_text or "Error:" in response_text:
logging.info(f"Progressive detection failed, using latest response fallback for {chatbot_name} (Request ID: {request_id})")
response_text = await detect_latest_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
# Fallback: Latest response detection for Gemini
if not response_text or "Error:" in response_text:
logging.info(f"Progressive detection failed, using latest response fallback for Gemini (Request ID: {request_id})")
response_text = await detect_latest_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
else:
# For other chatbots, use JSON-based detection
logging.info(f"Searching for JSON response with ID: {request_id}")
# Primary strategy: JSON response detection with unique ID
response_text = await detect_json_response_with_id(page, container_selector, request_id, prompt, modified_prompt, chatbot_name)
# Fallback strategy: If JSON detection fails, try progressive content detection
if not response_text or "Error:" in response_text:
logging.info(f"JSON detection failed, trying progressive content detection for {chatbot_name} (Request ID: {request_id})")
response_text = await detect_progressive_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
# Final fallback: Latest response detection
if not response_text or "Error:" in response_text:
logging.info(f"Progressive detection failed, using latest response fallback for {chatbot_name} (Request ID: {request_id})")
response_text = await detect_latest_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name)
# Decode base64 response if we got a valid response
# Enhanced decoding logic to handle various response formats
if response_text and not response_text.startswith("Error:"):
try:
# Try to decode the base64 response
decoded_response = base64.b64decode(response_text).decode('utf-8')
logging.info(f"Successfully decoded base64 response from {chatbot_name} (Request ID: {request_id}): {decoded_response[:200]}...")
return decoded_response.strip()
except Exception as e:
logging.warning(f"Failed to decode base64 response from {chatbot_name} (Request ID: {request_id}): {str(e)}")
logging.info(f"Returning raw response: {response_text[:200]}...")
return response_text.strip()
# Log the raw response before decoding
logging.info(f"Raw response from {chatbot_name} before decoding (Request ID: {request_id}): {response_text[:500]}...")
# Try multiple decoding strategies
decoded_response = decode_chatbot_response(response_text, chatbot_name, request_id)
# Log the final decoded response
logging.info(f"Final decoded response from {chatbot_name} (Request ID: {request_id}): {decoded_response[:200] if decoded_response else 'None'}...")
return decoded_response.strip()
logging.info(f"Final response from {chatbot_name} (Request ID: {request_id}): {response_text[:200] if response_text else 'None'}...")
return (response_text or "Error: No response detected").strip()
def decode_chatbot_response(response_text, chatbot_name, request_id):
"""Enhanced decoder for chatbot responses with multiple fallback strategies."""
logging.info(f"Decoding response from {chatbot_name} (Request ID: {request_id}): {response_text[:300]}...")
# For Gemini, extract content from RESPONSE_ID markers
if chatbot_name == 'gemini:latest':
logging.info(f"Gemini response processing (Request ID: {request_id}): {response_text[:200]}...")
# Look for RESPONSE_ID_[ID]_START: ... :END_RESPONSE_ID_[ID] pattern
response_pattern = rf'RESPONSE_ID_{re.escape(request_id)}_START:\s*(.*?)\s*:END_RESPONSE_ID_{re.escape(request_id)}'
match = re.search(response_pattern, response_text, re.DOTALL)
if match:
extracted_content = match.group(1).strip()
logging.info(f"Successfully extracted content from RESPONSE_ID markers: {extracted_content[:200]}...")
return extracted_content
else:
# Fallback: look for any RESPONSE_ID pattern regardless of request_id
fallback_pattern = r'RESPONSE_ID_.*?_START:\s*(.*?)\s*:END_RESPONSE_ID_.*?'
fallback_match = re.search(fallback_pattern, response_text, re.DOTALL)
if fallback_match:
extracted_content = fallback_match.group(1).strip()
logging.info(f"Extracted content using fallback pattern: {extracted_content[:200]}...")
return extracted_content
else:
# Final fallback: return the original text
logging.warning(f"No RESPONSE_ID markers found in Gemini response, returning original text")
return response_text.strip()
# Check for literal placeholder first and reject it
if response_text.strip() == "BASE64_ENCODED_ANSWER":
logging.warning(f"Received literal placeholder 'BASE64_ENCODED_ANSWER' from {chatbot_name} - this indicates a detection failure")
return "Error: Received placeholder instead of actual response"
# Strategy 1: Try to parse as JSON first (for properly formatted responses)
try:
# Clean up the response text first - remove any leading/trailing whitespace and extra characters
cleaned_response = response_text.strip()
# Handle cases where JSON might be wrapped or have extra content
json_start = cleaned_response.find('{')
json_end = cleaned_response.rfind('}') + 1
if json_start >= 0 and json_end > json_start:
json_text = cleaned_response[json_start:json_end]
logging.info(f"Extracted JSON text: {json_text[:200]}...")
json_obj = json.loads(json_text)
if 'response' in json_obj and json_obj['response']:
base64_content = json_obj['response']
logging.info(f"Found JSON with response field, attempting base64 decode of: {base64_content[:100]}...")
# Validate this is not a placeholder
if base64_content.strip() in ["BASE64_ENCODED_ANSWER", "base64_encoded_answer"]:
logging.error(f"JSON contains placeholder text in response field: {base64_content}")
return "Error: JSON response field contains placeholder text"
# Try to decode the base64 response field
try:
if isinstance(base64_content, str) and len(base64_content) > 0:
# More robust base64 validation and cleanup
base64_content = base64_content.strip()
# Add padding if needed for base64
missing_padding = len(base64_content) % 4
if missing_padding:
base64_content += '=' * (4 - missing_padding)
# Validate it looks like base64
if re.match(r'^[A-Za-z0-9+/]+(=){0,2}$', base64_content):
decoded = base64.b64decode(base64_content).decode('utf-8')
logging.info(f"Successfully decoded JSON+Base64 response from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
return decoded
else:
logging.info(f"Response field doesn't look like base64, treating as plain text: {base64_content[:200]}...")
return base64_content
else:
logging.warning(f"Empty or invalid response field in JSON: {base64_content}")
return "Error: Empty response field in JSON"
except Exception as decode_error:
logging.error(f"Base64 decode failed for JSON response from {chatbot_name}: {decode_error}")
# If base64 decode fails, check if it's already decoded text
if isinstance(base64_content, str) and base64_content.isprintable() and len(base64_content) > 0:
logging.info(f"Base64 decode failed, but response field appears to be readable text, returning as-is: {base64_content[:200]}...")
return base64_content
# Return error if we can't decode or use the content
return f"Error: Failed to decode base64 content: {str(decode_error)}"
elif 'content' in json_obj:
# Some responses might use 'content' instead of 'response'
content = json_obj['content']
logging.info(f"Found JSON with content field: {str(content)[:200]}...")
return str(content)
else:
# Return the whole JSON as string if no recognized fields
logging.info(f"JSON has no recognized response fields, returning whole JSON: {json_text[:200]}...")
return json_text
except json.JSONDecodeError as je:
logging.warning(f"JSON decode failed for response from {chatbot_name}: {je}")
# Not JSON, continue to next strategy
pass
except Exception as e:
logging.error(f"Unexpected error in JSON processing for {chatbot_name}: {e}")
pass
# Strategy 2: Try direct base64 decoding (for raw base64 responses)
try:
# Check if it looks like base64 (alphanumeric + / + = padding)
clean_response = response_text.strip()
if re.match(r'^[A-Za-z0-9+/]+={0,2}$', clean_response) and len(clean_response) > 20:
# Add padding if needed
missing_padding = len(clean_response) % 4
if missing_padding:
clean_response += '=' * (4 - missing_padding)
decoded = base64.b64decode(clean_response).decode('utf-8')
logging.info(f"Successfully decoded raw Base64 response from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
return decoded
except Exception as decode_error:
logging.warning(f"Direct base64 decode failed for {chatbot_name}: {decode_error}")
# Strategy 3: Look for base64 patterns within the text
try:
# Find base64-like patterns in the response
base64_pattern = re.compile(r'[A-Za-z0-9+/]{20,}={0,2}')
matches = base64_pattern.findall(response_text)
for match in matches:
if len(match) > 20 and len(match) % 4 <= 2: # Valid base64 length
try:
# Add padding if needed
padded_match = match
missing_padding = len(match) % 4
if missing_padding:
padded_match += '=' * (4 - missing_padding)
decoded = base64.b64decode(padded_match).decode('utf-8')
# Check if decoded content looks reasonable (not binary gibberish)
if decoded.isprintable() and len(decoded) > 10:
logging.info(f"Successfully extracted and decoded Base64 from response from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
return decoded
except Exception as match_error:
logging.debug(f"Failed to decode base64 match '{match[:50]}...': {match_error}")
continue
except Exception as extract_error:
logging.warning(f"Base64 extraction failed for {chatbot_name}: {extract_error}")
# Strategy 4: Try to extract content from JSON-like text (malformed JSON)
try:
# Look for response field in text even if JSON is malformed
response_match = re.search(r'"response"\s*:\s*"([^"]+)"', response_text)
if response_match:
potential_base64 = response_match.group(1)
try:
# Add padding if needed
missing_padding = len(potential_base64) % 4
if missing_padding:
potential_base64 += '=' * (4 - missing_padding)
decoded = base64.b64decode(potential_base64).decode('utf-8')
logging.info(f"Successfully decoded extracted response field from {chatbot_name} (Request ID: {request_id}): {decoded[:200]}...")
return decoded
except:
# Return the extracted field as-is if decode fails
return potential_base64
except Exception as extract_error:
logging.warning(f"Response field extraction failed for {chatbot_name}: {extract_error}")
# Strategy 5: Return raw response (fallback)
logging.info(f"Returning raw response from {chatbot_name} (Request ID: {request_id}): {response_text[:200]}...")
return response_text
async def detect_json_response_with_id(page, container_selector, request_id, prompt, modified_prompt, chatbot_name):
"""Detect JSON responses with unique ID for reliable extraction."""
try:
return await page.evaluate(
"""([containerSelector, requestId, prompt, modifiedPrompt]) => {
const container = document.querySelector(containerSelector);
if (!container) return "Error: Container not found";
return new Promise((resolve) => {
let resolved = false;
let bestMatch = null;
let partialJsonContent = '';
# Enhanced JSON detection with better timing and fallback mechanisms
js_code = """
([containerSelector, requestId, prompt, modifiedPrompt]) => {
const container = document.querySelector(containerSelector);
if (!container) return "Error: Container not found";
return new Promise((resolve) => {
let resolved = false;
let lastSeenContent = '';
let stableChecks = 0;
const resolveOnce = (result) => {
if (!resolved) {
resolved = true;
resolve(result);
}
};
const findJsonWithId = () => {
// Get all text elements with more comprehensive search
const elements = container.querySelectorAll('*');
let allText = '';
const resolveOnce = (result) => {
if (!resolved) {
resolved = true;
resolve(result);
for (const element of elements) {
const text = element.textContent;
if (text && text.trim()) {
allText += text + ' ';
}
};
// Function to clean and decode HTML entities and escape sequences - simplified
const cleanJsonText = (text) => {
if (!text) return '';
return text
.replace(/&quot;/g, '"')
.replace(/&amp;/g, '&')
.replace(/&lt;/g, '<')
.replace(/&gt;/g, '>')
.replace(/&#x27;/g, "'")
.replace(/&#x2F;/g, '/')
.trim();
};
// Function to extract and parse JSON response
const extractJsonResponse = (text) => {
if (!text || text === prompt || text === modifiedPrompt) return null;
// Clean the text first
const cleanedText = cleanJsonText(text);
// Look for JSON patterns with our unique ID - using string concatenation to avoid regex escaping issues
const jsonPatterns = [
// Standard JSON object patterns - using string methods instead of complex regex
new RegExp('\\{[^{}]*"id"[^{}]*"' + requestId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '"[^{}]*"response"[^{}]*\\}', 'g'),
new RegExp('\\{[^{}]*"response"[^{}]*"id"[^{}]*"' + requestId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '"[^{}]*\\}', 'g')
];
// Try both original and cleaned text
const textsToTry = [cleanedText, text];
for (const textToTry of textsToTry) {
for (const pattern of jsonPatterns) {
const matches = textToTry.match(pattern);
if (matches) {
for (const match of matches) {
try {
const jsonObj = JSON.parse(match);
if (jsonObj.id && jsonObj.id.includes(requestId) && jsonObj.response) {
console.log(`Found JSON response with ID: ${jsonObj.id}`);
return jsonObj.response;
}
} catch (e) {
// Try to extract response value from malformed JSON - simplified
}
// Look for our request ID in the combined text
if (!allText.includes(requestId)) return null;
// Multiple JSON extraction strategies
const jsonExtractionStrategies = [
// Strategy 1: Exact ID match with JSON parsing
() => {
let braceCount = 0;
let jsonStart = -1;
for (let i = 0; i < allText.length; i++) {
if (allText[i] === '{') {
if (braceCount === 0) jsonStart = i;
braceCount++;
} else if (allText[i] === '}') {
braceCount--;
if (braceCount === 0 && jsonStart >= 0) {
const jsonStr = allText.substring(jsonStart, i + 1);
if (jsonStr.includes(requestId)) {
try {
const responsePattern = /"response"\s*:\s*"([^"]*)"/;
const responseMatch = match.match(responsePattern);
if (responseMatch && match.includes(requestId)) {
const responseValue = responseMatch[1];
if (responseValue && responseValue.length > 0) {
console.log('Extracted response from malformed JSON: ' + responseValue.substring(0, 100) + '...');
return responseValue;
}
const jsonObj = JSON.parse(jsonStr);
if (jsonObj.id && jsonObj.id.includes(requestId) && jsonObj.response) {
return jsonObj.response;
}
} catch (e2) {
console.log('Failed to extract response from JSON: ' + match.substring(0, 100) + '...');
} catch (e) {
// Continue to next attempt
}
}
jsonStart = -1;
}
}
}
}
// Look for partial JSON that might be building up
const partialJsonRegex = new RegExp('\\{[\\s\\S]*?"id"[\\s\\S]*?"' + requestId + '"[\\s\\S]*', 'g');
const partialMatch = cleanedText.match(partialJsonRegex);
if (partialMatch && partialMatch[0].length > partialJsonContent.length) {
partialJsonContent = partialMatch[0];
console.log('Found partial JSON: ' + partialJsonContent.substring(0, 100) + '...');
}
return null;
};
// Function to extract JSON from HTML formatted code blocks
const extractJsonFromHtml = () => {
// Look for JSON in <pre><code class="language-json"> tags
const codeBlocks = container.querySelectorAll([
'pre code.language-json',
'pre code[class*="json"]',
'code.language-json',
'code[class*="json"]',
'pre code',
'code'
].join(', '));
for (const codeBlock of codeBlocks) {
const jsonText = codeBlock.textContent ? codeBlock.textContent.trim() : '';
if (jsonText && (jsonText.includes(requestId) || jsonText.includes('{'))) {
console.log('Found code block with potential JSON: ' + jsonText.substring(0, 100) + '...');
const result = extractJsonResponse(jsonText);
if (result) return result;
}
}
return null;
};
// Get combined text from all relevant elements
const getCombinedText = () => {
// First try to extract from HTML formatted code blocks
const htmlResult = extractJsonFromHtml();
if (htmlResult) return htmlResult;
const chatElements = container.querySelectorAll([
'div[data-testid*="cellInnerDiv"]',
'div[data-testid*="tweetText"]',
'article',
'div[data-testid="primaryColumn"] div',
'main div[role="article"]',
'div[class*="css-"]',
'span[class*="css-"]',
'span', // Enhanced span detection for JSON
'pre', 'code', // Include code blocks for JSON
'p', 'div'
].join(', '));
let allTexts = [];
for (const element of chatElements) {
const text = element.textContent ? element.textContent.trim() : '';
if (text && text !== prompt && text !== modifiedPrompt && text.length > 10) {
// Look for elements that might contain JSON or our request ID
if (text.includes(requestId) || text.includes('{') || text.includes('"response"')) {
allTexts.push(text);
}
}
}
// Try different text combinations
const combinations = [
allTexts.join(' '),
allTexts.join('\n'),
allTexts.join(''),
...allTexts // Individual texts
];
return null;
},
for (const combo of combinations) {
const result = extractJsonResponse(combo);
if (result) return result;
}
return null;
};
const observer = new MutationObserver((mutations) => {
// Only resolve if we have complete, valid JSON with exact ID match
const result = getCompleteJsonResponse();
if (result && typeof result === 'string' && result.length > 0) {
console.log('Observer found complete response, disconnecting...');
observer.disconnect();
resolveOnce(result);
return;
}
// Log progress for debugging but don't resolve yet
const hasPartialId = container.textContent && container.textContent.includes(requestId);
if (hasPartialId) {
console.log('Found request ID in content, waiting for complete JSON...');
}
});
observer.observe(container, {
childList: true,
subtree: true,
characterData: true
});
// Function to validate complete JSON with both required keys and exact ID match
const isCompleteValidJson = (text) => {
try {
const jsonObj = JSON.parse(text);
// Strict validation: must have exact ID match and non-empty response
const hasValidId = jsonObj.id &&
typeof jsonObj.id === 'string' &&
jsonObj.id.includes(requestId) &&
jsonObj.id.length >= requestId.length;
const hasValidResponse = jsonObj.response &&
typeof jsonObj.response === 'string' &&
jsonObj.response.trim().length > 0;
// Strategy 2: Find JSON-like patterns near request ID
() => {
const idIndex = allText.indexOf(requestId);
if (idIndex === -1) return null;
// Additional validation: check if response looks like base64
let isValidBase64 = false;
if (hasValidResponse) {
try {
// Basic base64 validation - should be valid base64 string
const base64Regex = /^[A-Za-z0-9+/]*={0,2}$/;
isValidBase64 = base64Regex.test(jsonObj.response.trim());
if (isValidBase64) {
// Try to decode to verify it's valid base64
atob(jsonObj.response.trim());
}
} catch (e) {
isValidBase64 = false;
}
}
// Look for JSON patterns around the ID
const searchRange = 2000; // chars around the ID
const startPos = Math.max(0, idIndex - searchRange);
const endPos = Math.min(allText.length, idIndex + searchRange);
const searchText = allText.substring(startPos, endPos);
if (hasValidId && hasValidResponse && isValidBase64) {
console.log('Complete valid JSON found - ID: ' + jsonObj.id + ', Base64 Response length: ' + jsonObj.response.length);
return true;
} else {
console.log('Incomplete JSON - ID valid: ' + hasValidId + ', Response valid: ' + hasValidResponse + ', Base64 valid: ' + isValidBase64);
return false;
}
} catch (e) {
console.log('JSON parse error: ' + e.message);
return false;
}
};
// Enhanced check that only returns complete, valid JSON with exact ID match
const getCompleteJsonResponse = () => {
console.log('Searching for complete JSON with ID: ' + requestId);
// First priority: Check all possible JSON containers including spans
const jsonContainers = container.querySelectorAll([
'span', // Check spans first - common for direct JSON output
'div', // Check divs for JSON content
'pre code.language-json',
'pre code[class*="json"]',
'code.language-json',
'code[class*="json"]',
'pre code',
'code',
'p' // Also check paragraphs
].join(', '));
for (const container of jsonContainers) {
const jsonText = container.textContent ? container.textContent.trim() : '';
if (jsonText && jsonText.includes(requestId)) {
console.log('Found container with request ID: ' + jsonText.substring(0, 150) + '...');
if (isCompleteValidJson(jsonText)) {
const jsonObj = JSON.parse(jsonText);
console.log('Returning validated response for ID: ' + jsonObj.id);
return jsonObj.response;
// Find JSON objects in this range
const jsonRegex = /\\{[^{}]*"response"[^{}]*"[^"]*"[^{}]*\\}/g;
const matches = searchText.match(jsonRegex);
if (matches) {
for (const match of matches) {
try {
const jsonObj = JSON.parse(match);
if (jsonObj.response) {
return jsonObj.response;
}
} catch (e) {
// Continue to next match
}
}
}
}
return null;
},
// Second priority: Check all text content for JSON patterns
const allElements = container.querySelectorAll('*');
for (const element of allElements) {
const text = element.textContent ? element.textContent.trim() : '';
if (text && text.includes(requestId) && text.includes('{') && text.includes('"response"')) {
console.log('Found element with potential JSON: ' + text.substring(0, 150) + '...');
// Try to extract JSON from this text - using escaped requestId
const escapedRequestId = requestId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const jsonPattern1 = new RegExp('\\{[^{}]*"id"[^{}]*"' + escapedRequestId + '"[^{}]*"response"[^{}]*\\}');
const jsonPattern2 = new RegExp('\\{[^{}]*"response"[^{}]*"id"[^{}]*"' + escapedRequestId + '"[^{}]*\\}');
const jsonMatch = text.match(jsonPattern1) || text.match(jsonPattern2);
if (jsonMatch && isCompleteValidJson(jsonMatch[0])) {
const jsonObj = JSON.parse(jsonMatch[0]);
console.log('Returning validated response from text for ID: ' + jsonObj.id);
return jsonObj.response;
// Strategy 3: Base64 pattern detection near request ID
() => {
const idIndex = allText.indexOf(requestId);
if (idIndex === -1) return null;
const searchRange = 1000;
const startPos = Math.max(0, idIndex - searchRange);
const endPos = Math.min(allText.length, idIndex + searchRange);
const searchText = allText.substring(startPos, endPos);
// Look for base64-like patterns (long alphanumeric strings)
const base64Regex = /[A-Za-z0-9+/]{20,}={0,2}/g;
const matches = searchText.match(base64Regex);
if (matches) {
for (const match of matches) {
// Validate it's likely base64
if (match.length > 20 && match.length % 4 <= 2) {
return match;
}
}
}
return null;
}
console.log('No complete JSON found with ID: ' + requestId);
return null;
};
// Initial check after a longer delay to allow content to load
];
// Try each strategy
for (const strategy of jsonExtractionStrategies) {
const result = strategy();
if (result) return result;
}
return null;
};
const checkForResponse = () => {
const result = findJsonWithId();
if (result) {
resolveOnce(result);
return true;
}
// Track content stability for better timing
const currentContent = container.textContent || '';
if (currentContent === lastSeenContent) {
stableChecks++;
} else {
stableChecks = 0;
lastSeenContent = currentContent;
}
return false;
};
// More aggressive observation
const observer = new MutationObserver((mutations) => {
// Small delay to let DOM settle
setTimeout(() => {
const result = getCompleteJsonResponse();
if (result) {
if (checkForResponse()) {
observer.disconnect();
resolveOnce(result);
}
}, 5000); // Increased initial delay
// Main timeout - wait for complete JSON (45 seconds as requested)
}, 100);
});
observer.observe(container, {
childList: true,
subtree: true,
characterData: true,
attributes: true,
attributeOldValue: true,
characterDataOldValue: true
});
// More frequent initial checks
const initialChecks = [500, 1000, 2000, 3000, 5000];
initialChecks.forEach(delay => {
setTimeout(() => {
const result = getCompleteJsonResponse();
if (result) {
observer.disconnect();
resolveOnce(result);
if (!resolved) checkForResponse();
}, delay);
});
// Periodic checks during waiting
const periodicCheck = setInterval(() => {
if (resolved) {
clearInterval(periodicCheck);
return;
}
checkForResponse();
}, 2000);
// Main timeout with final attempt
setTimeout(() => {
clearInterval(periodicCheck);
observer.disconnect();
if (!resolved) {
const finalResult = findJsonWithId();
if (finalResult) {
resolveOnce(finalResult);
} else {
console.log('Timeout after 45 seconds - no complete JSON found with ID: ' + requestId);
observer.disconnect();
resolveOnce("Error: JSON response timeout - no complete valid JSON found");
resolveOnce("Error: JSON response timeout after comprehensive search");
}
}, 45000); // 45 seconds as requested
// Final safety timeout
setTimeout(() => {
observer.disconnect();
console.log('Final timeout after 60 seconds for request ID: ' + requestId);
resolveOnce("Error: Final timeout - response detection failed");
}, 60000); // 60 seconds final safety
});
}""",
[container_selector, request_id, prompt, modified_prompt]
)
}
}, 45000);
// Ultimate timeout
setTimeout(() => {
clearInterval(periodicCheck);
observer.disconnect();
if (!resolved) {
resolveOnce("Error: Final timeout - no JSON response detected");
}
}, 60000);
});
}
"""
return await page.evaluate(js_code, [container_selector, request_id, prompt, modified_prompt])
except Exception as e:
logging.error(f"Error in JSON response detection: {e}")
return "Error: JSON response detection failed"
async def detect_progressive_response(page, container_selector, prompt, modified_prompt, request_id, chatbot_name):
"""Detect responses by monitoring progressive content changes."""
"""Detect responses by monitoring progressive content changes with enhanced detection."""
try:
return await page.evaluate(
"""([containerSelector, prompt, modifiedPrompt, requestId]) => {
"""([containerSelector, prompt, modifiedPrompt, requestId, chatbotName]) => {
const container = document.querySelector(containerSelector);
if (!container) return "Error: Container not found";
......@@ -847,6 +1025,8 @@ async def detect_progressive_response(page, container_selector, prompt, modified
let lastContent = '';
let stableCount = 0;
let progressiveContent = '';
let lastResponseTimestamp = 0;
let previousResponseText = '';
const resolveOnce = (result) => {
if (!resolved) {
......@@ -856,57 +1036,162 @@ async def detect_progressive_response(page, container_selector, prompt, modified
};
const extractLatestResponse = () => {
// Look for the most recent bot response
const responseElements = container.querySelectorAll([
'div[data-testid*="cellInnerDiv"]',
'article div',
'div[role="article"] div',
'div[class*="css-"] span',
'main div div span'
].join(', '));
// Check for Gemini-specific formatted response first
if (chatbotName === 'gemini:latest') {
const allText = container.textContent || '';
const startMarker = `RESPONSE_ID_[${requestId}]_START:`;
const endMarker = `:END_RESPONSE_ID_[${requestId}]`;
const startIndex = allText.indexOf(startMarker);
const endIndex = allText.indexOf(endMarker, startIndex);
if (startIndex !== -1 && endIndex !== -1) {
const responseText = allText.substring(startIndex + startMarker.length, endIndex).trim();
if (responseText && responseText.length > 10) {
return responseText;
}
}
}
// Fallback to selector-based detection
let responseSelectors = [];
let responses = [];
for (const element of responseElements) {
const text = element.textContent ? element.textContent.trim() : '';
if (text && text !== prompt && text !== modifiedPrompt &&
text.length > 20 && !text.includes('Grok something') &&
!text.includes('What can I help') && !text.includes('@')) {
responses.push({
text: text,
element: element,
timestamp: Date.now()
});
if (chatbotName === 'gemini:latest') {
responseSelectors = [
// Gemini-specific selectors
'[data-test-id*="conversation-turn"] [data-test-id*="response"]',
'[data-test-id*="model-response-text"]',
'.model-response-text',
'message-content',
'[role="presentation"] > div',
'.response-container',
'.conversation-turn [data-test-id*="response"]',
// Generic Gemini patterns
'main [class*="response"]',
'main [class*="message"]:not([class*="user"])',
'main div[style*="white-space"]:not([contenteditable])',
// Broader fallbacks
'p:not([class*="input"]):not([class*="prompt"])',
'div:not([class*="input"]):not([class*="button"]):not([contenteditable]) > span'
];
} else {
responseSelectors = [
// Grok/Twitter-specific selectors
'div[data-testid*="cellInnerDiv"]',
'article[data-testid*="tweet"] div',
'div[role="article"] div',
'div[class*="css-"] span:not([class*="button"])',
'main div div span:not([class*="input"])',
// Generic fallbacks
'p:not([class*="input"]):not([class*="prompt"])',
'div:not([class*="input"]):not([class*="button"]) > span'
];
}
let allCandidates = [];
const currentTime = Date.now();
// Collect all potential response elements
for (const selector of responseSelectors) {
try {
const elements = container.querySelectorAll(selector);
for (const element of elements) {
const text = element.textContent ? element.textContent.trim() : '';
if (text &&
text !== prompt &&
text !== modifiedPrompt &&
text.length > 10 &&
!text.includes('Ask Gemini') &&
!text.includes('Send message') &&
!text.includes('Grok something') &&
!text.includes('What can I help') &&
!text.match(/^[@#]\\w+/) && // Skip handles and hashtags
!text.match(/^\\d+[smhd]$/) && // Skip time indicators
!element.querySelector('input, button, textarea') && // Skip interactive elements
!element.hasAttribute('contenteditable')) { // Skip editable content
allCandidates.push({
text: text,
element: element,
timestamp: currentTime,
selector: selector,
length: text.length
});
}
}
} catch (e) {
// Skip invalid selectors
continue;
}
}
// Sort by DOM order and find the last substantial response
responses.sort((a, b) => {
if (allCandidates.length === 0) return '';
// Filter for likely new responses (different from previous)
const newCandidates = allCandidates.filter(candidate => {
return !previousResponseText ||
candidate.text !== previousResponseText &&
candidate.text !== lastContent &&
candidate.text.length > Math.max(10, previousResponseText.length * 0.1);
});
const candidatePool = newCandidates.length > 0 ? newCandidates : allCandidates;
// Sort by relevance: longer text preferred, then by DOM position
candidatePool.sort((a, b) => {
// Heavily prefer longer responses
const lengthDiff = b.length - a.length;
if (Math.abs(lengthDiff) > 20) return lengthDiff;
// Then prefer later DOM elements (more recent)
const position = a.element.compareDocumentPosition(b.element);
return position & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1;
});
// Return the longest response that seems complete
for (const response of responses.reverse()) {
if (response.text.length > 100) {
return response.text;
}
}
return responses.length > 0 ? responses[responses.length - 1].text : '';
return candidatePool[0]?.text || '';
};
const checkForResponse = () => {
const currentResponse = extractLatestResponse();
if (currentResponse && currentResponse.length > lastContent.length) {
progressiveContent = currentResponse;
lastContent = currentResponse;
stableCount = 0;
} else if (currentResponse === lastContent && currentResponse.length > 0) {
stableCount++;
if (stableCount >= 3 && progressiveContent.length > 50) {
// Content seems stable and substantial
return progressiveContent;
// For Gemini with formatted response, return immediately if found
if (chatbotName === 'gemini:latest' && currentResponse) {
const startMarker = `RESPONSE_ID_[${requestId}]_START:`;
const allText = container.textContent || '';
if (allText.includes(startMarker) && currentResponse.length > 10) {
return currentResponse;
}
}
// Enhanced stability checking with better timing
if (currentResponse && currentResponse.length > 0) {
if (currentResponse !== lastContent) {
// New content detected
if (currentResponse.length > lastContent.length ||
(currentResponse.length > 20 && !lastContent)) {
progressiveContent = currentResponse;
lastContent = currentResponse;
stableCount = 0;
lastResponseTimestamp = Date.now();
// Quick return for substantial responses
if (currentResponse.length > 150 &&
!currentResponse.includes(requestId) &&
currentResponse !== prompt &&
currentResponse !== modifiedPrompt) {
return currentResponse;
}
}
} else if (currentResponse === lastContent && currentResponse.length > 15) {
stableCount++;
const timeSinceLastResponse = Date.now() - lastResponseTimestamp;
// More responsive stability check
if ((stableCount >= 2 && timeSinceLastResponse > 2000) ||
(stableCount >= 1 && currentResponse.length > 100 && timeSinceLastResponse > 1500)) {
previousResponseText = currentResponse;
return progressiveContent || currentResponse;
}
}
}
......@@ -914,42 +1199,60 @@ async def detect_progressive_response(page, container_selector, prompt, modified
};
const observer = new MutationObserver((mutations) => {
const result = checkForResponse();
if (result) {
observer.disconnect();
resolveOnce(result);
}
// Small delay to allow DOM to settle
setTimeout(() => {
if (!resolved) {
const result = checkForResponse();
if (result) {
observer.disconnect();
resolveOnce(result);
}
}
}, 150);
});
observer.observe(container, {
childList: true,
subtree: true,
characterData: true
characterData: true,
attributes: true
});
// Periodic checks
// More frequent periodic checks for better responsiveness
const interval = setInterval(() => {
const result = checkForResponse();
if (result) {
clearInterval(interval);
observer.disconnect();
resolveOnce(result);
if (!resolved) {
const result = checkForResponse();
if (result) {
clearInterval(interval);
observer.disconnect();
resolveOnce(result);
}
}
}, 2000);
}, 1000);
// Timeout
// Multiple timeout stages for better reliability
setTimeout(() => {
clearInterval(interval);
observer.disconnect();
if (progressiveContent.length > 20) {
if (!resolved && progressiveContent && progressiveContent.length > 20) {
clearInterval(interval);
observer.disconnect();
resolveOnce(progressiveContent);
} else {
resolveOnce("Error: Progressive detection timeout");
}
}, 60000);
}, 25000); // Early timeout for fast responses
setTimeout(() => {
if (!resolved) {
clearInterval(interval);
observer.disconnect();
if (progressiveContent && progressiveContent.length > 10) {
resolveOnce(progressiveContent);
} else {
resolveOnce("Error: Progressive detection timeout - no substantial content found");
}
}
}, 45000); // Main timeout
});
}""",
[container_selector, prompt, modified_prompt, request_id]
[container_selector, prompt, modified_prompt, request_id, chatbot_name]
)
except Exception as e:
logging.error(f"Error in progressive detection: {e}")
......@@ -1020,7 +1323,22 @@ async def start_browser(args):
else:
browser = await p.chromium.launch_persistent_context(
user_data_dir="./playwright_data",
headless=False
headless=False,
# Better cookie and session handling
accept_downloads=True,
bypass_csp=True,
ignore_https_errors=True,
java_script_enabled=True,
# Persistent storage is handled automatically by user_data_dir
locale='en-US',
timezone_id='America/New_York',
# Additional args for better compatibility
args=[
'--disable-blink-features=AutomationControlled',
'--disable-web-security',
'--allow-running-insecure-content',
'--disable-features=VizDisplayCompositor'
]
)
browser_context = browser
pages = {}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment