# Fixture Manager - Comprehensive Python Daemon System

A sophisticated Python daemon system for Linux servers with internet exposure, implementing a secure web dashboard and RESTful API with robust authentication mechanisms. The system provides advanced file upload capabilities with real-time progress tracking and a comprehensive fixture management system.

## Features

### Core Functionality
- **Secure Web Dashboard**: Modern web interface with authentication and authorization
- **RESTful API**: Comprehensive API with JWT authentication
- **MySQL Database Integration**: Robust database connectivity with connection pooling
- **Advanced File Upload System**: Real-time progress tracking with SHA1 checksum verification
- **Dual-Format Support**: Intelligent parsing of CSV/XLSX fixture files
- **Two-Stage Upload Workflow**: Fixture files followed by mandatory ZIP uploads
- **Daemon Process Management**: Full Linux daemon with systemd integration

### Security Features
- **Multi-layer Authentication**: Session-based and JWT token authentication
- **API Token Management**: User-generated tokens for external application access
- **Rate Limiting**: Protection against brute force attacks
- **File Validation**: Comprehensive security checks and malicious content detection
- **SQL Injection Protection**: Parameterized queries and ORM usage
- **CSRF Protection**: Cross-site request forgery prevention
- **Security Headers**: Comprehensive HTTP security headers
- **Input Sanitization**: All user inputs are validated and sanitized

### Database Schema
- **Normalized Design**: Optimized relational database structure
- **Primary Matches Table**: Core fixture data with system fields
- **Secondary Outcomes Table**: Dynamic result columns with foreign key relationships
- **API Token Management**: Secure token storage with usage tracking
- **File Upload Tracking**: Complete upload lifecycle management
- **System Logging**: Comprehensive audit trail
- **Session Management**: Secure user session handling

## Installation

### Prerequisites
- Linux server (Ubuntu 18.04+, CentOS 7+, or similar)
- Python 3.8+
- MySQL 5.7+ or MariaDB 10.3+
- Root or sudo access

### Quick Installation
```bash
# Clone the repository
git clone <repository-url>
cd fixture-manager

# Make installation script executable
chmod +x install.sh

# Run installation (as root)
sudo ./install.sh
```

### Manual Installation
1. **Install System Dependencies**:
   ```bash
   # Ubuntu/Debian
   apt-get update
   apt-get install python3 python3-pip python3-venv mysql-server nginx supervisor

   # CentOS/RHEL
   yum install python3 python3-pip mysql-server nginx supervisor
   ```

2. **Create System User**:
   ```bash
   useradd --system --home-dir /var/lib/fixture-daemon fixture
   ```

3. **Install Python Dependencies**:
   ```bash
   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   ```

4. **Configure Database**:
   ```bash
   mysql -u root -p < database/schema.sql
   ```

5. **Configure Environment**:
   ```bash
   cp .env.example .env
   # Edit .env with your configuration
   # Note: For PyInstaller deployments, configuration will migrate to mbetterd.conf automatically
   ```

## Configuration

### Configuration File (mbetterd.conf)
The system automatically migrates from `.env` to `mbetterd.conf` stored in persistent directories for PyInstaller compatibility. Configuration settings include:

```bash
# Database Configuration
MYSQL_HOST=localhost
MYSQL_PORT=3306
MYSQL_USER=fixture_user
MYSQL_PASSWORD=secure_password
MYSQL_DATABASE=fixture_manager

# Security Configuration
SECRET_KEY=your-secret-key-here
JWT_SECRET_KEY=your-jwt-secret-key
BCRYPT_LOG_ROUNDS=12

# File Upload Configuration
UPLOAD_FOLDER=/var/lib/fixture-daemon/uploads
MAX_CONTENT_LENGTH=524288000  # 500MB
MAX_CONCURRENT_UPLOADS=5

# Server Configuration
HOST=0.0.0.0
PORT=5000
DEBUG=false
```

### Database Schema
The system automatically creates the following tables:
- `users` - User authentication and management
- `matches` - Core fixture data with system fields
- `match_outcomes` - Dynamic outcome results
- `api_tokens` - User-generated API tokens for external access
- `file_uploads` - Upload tracking and progress
- `system_logs` - Comprehensive logging
- `user_sessions` - Session management

## Usage

### Daemon Management
```bash
# Start the daemon
sudo systemctl start fixture-daemon

# Stop the daemon
sudo systemctl stop fixture-daemon

# Restart the daemon
sudo systemctl restart fixture-daemon

# Check status
sudo systemctl status fixture-daemon

# View logs
journalctl -u fixture-daemon -f
```

### Direct Daemon Control
```bash
# Start in foreground (for debugging)
python daemon.py start --foreground

# Start as daemon
python daemon.py start

# Stop daemon
python daemon.py stop

# Restart daemon
python daemon.py restart

# Check status
python daemon.py status

# Reload configuration
python daemon.py reload
```

### Web Interface
Access the web dashboard at `http://your-server-ip/`

**Default Credentials**:
- Username: `admin`
- Password: `admin123`

**⚠️ Important**: Change the default password immediately after installation!

### API Usage

#### Authentication Methods

**1. Session-Based Authentication (Web Interface)**
```bash
# Login via web interface
curl -X POST http://your-server/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "admin123"}'
```

**2. API Token Authentication (Recommended for External Apps)**
```bash
# Use API token in Authorization header (recommended)
curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  http://your-server/api/fixtures

# Alternative: Use X-API-Token header
curl -H "X-API-Token: YOUR_API_TOKEN" \
  http://your-server/api/matches

# Alternative: Use query parameter (less secure)
curl "http://your-server/api/match/123?token=YOUR_API_TOKEN"
```

**3. JWT Token Authentication (Legacy)**
```bash
# Login and get JWT token
curl -X POST http://your-server/auth/api/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "admin123"}'
```

#### Upload Fixture File
```bash
# Upload CSV/XLSX fixture file
curl -X POST http://your-server/upload/api/fixture \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -F "file=@fixtures.csv"
```

#### Upload ZIP File
```bash
# Upload ZIP file for specific match
curl -X POST http://your-server/upload/api/zip/123 \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -F "file=@match_data.zip"
```

#### Get Matches
```bash
# Get all matches with pagination
curl -X GET "http://your-server/api/matches?page=1&per_page=20" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

#### Get Fixtures
```bash
# Get all fixtures
curl -X GET "http://your-server/api/fixtures" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
```

#### Get Match Details
```bash
# Get specific match with outcomes
curl -X GET "http://your-server/api/match/123" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
```

#### Get Fixture Updates (New!)
The `/api/updates` endpoint provides incremental synchronization for fixture data:

```bash
# Get last N fixtures (default behavior, N configured in system settings)
curl -X GET "http://your-server/api/updates" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Get fixtures updated after specific unix timestamp
curl -X GET "http://your-server/api/updates?from=1704067200" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# POST method also supported with JSON body
curl -X POST "http://your-server/api/updates" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"from": 1704067200}'

# Get recent fixtures without timestamp filter
curl -X POST "http://your-server/api/updates" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'
```

**Features:**
- **Incremental Updates**: Use `from` parameter for efficient data synchronization
- **Flexible Methods**: Supports both GET (query params) and POST (JSON body)
- **Configurable Limits**: Respects system setting for maximum fixtures returned
- **Authenticated ZIP Downloads**: Secure direct download URLs with token authentication
- **Hybrid Authentication**: Works with both JWT and API tokens automatically
- **Smart Fallback**: Gracefully handles existing data without active timestamps

#### Download ZIP Files (Authenticated)
```bash
# Download ZIP file for specific match (requires authentication)
curl -X GET "http://your-server/api/download/zip/123" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -o "match_123.zip"
```

## File Format Requirements

### Fixture Files (CSV/XLSX)
**Required Columns**:
- `Match #` (integer) - Unique match identifier
- `Fighter1 (Township)` (varchar255) - First fighter details
- `Fighter2 (Township)` (varchar255) - Second fighter details
- `Venue (Kampala Township)` (varchar255) - Match venue

**Optional Columns**:
- Any numeric columns will be automatically detected as outcome results
- Values must be numeric (float with 2-decimal precision)

**Example CSV**:
```csv
Match #,Fighter1 (Township),Fighter2 (Township),Venue (Kampala Township),Score1,Score2,Duration
1,John Doe (Central),Jane Smith (North),Stadium A (Kampala),85.5,92.3,12.5
2,Mike Johnson (East),Sarah Wilson (West),Arena B (Kampala),78.2,81.7,15.2
```

### ZIP Files
- Must be uploaded after fixture file processing
- Associated with specific match records
- Triggers match activation upon successful upload
- SHA1 checksum verification for integrity

## Architecture

### System Components
1. **Flask Web Application**: Core web framework with blueprints
2. **SQLAlchemy ORM**: Database abstraction and management
3. **JWT Authentication**: Stateless API authentication
4. **File Upload Handler**: Chunked uploads with progress tracking
5. **Fixture Parser**: Intelligent CSV/XLSX parsing
6. **Security Layer**: Multi-layer security implementation
7. **Logging System**: Comprehensive audit and monitoring
8. **Daemon Manager**: Linux daemon process management

### Security Architecture
- **Authentication**: Multi-factor with session and JWT support
- **Authorization**: Role-based access control (RBAC)
- **Input Validation**: Comprehensive sanitization and validation
- **File Security**: Malicious content detection and quarantine
- **Network Security**: Rate limiting and DDoS protection
- **Data Protection**: Encryption at rest and in transit

### Database Design
- **Normalized Schema**: Third normal form compliance
- **Foreign Key Constraints**: Referential integrity
- **Indexing Strategy**: Optimized query performance
- **Transaction Management**: ACID compliance
- **Connection Pooling**: Efficient resource utilization

## Monitoring and Maintenance

### Log Files
- **Application Logs**: `/var/log/fixture-daemon.log`
- **System Logs**: `journalctl -u fixture-daemon`
- **Database Logs**: MySQL error logs
- **Web Server Logs**: Nginx access/error logs

### Health Monitoring
```bash
# Check system health
curl http://your-server/health

# Get system statistics
curl -H "Authorization: Bearer TOKEN" http://your-server/api/statistics
```

### Backup and Recovery
```bash
# Manual backup
/opt/fixture-manager/backup.sh

# Restore from backup
mysql -u fixture_user -p fixture_manager < backup.sql
```

### Maintenance Tasks
The daemon automatically performs:
- **Session Cleanup**: Expired sessions removed hourly
- **Log Rotation**: Old logs archived daily
- **File Cleanup**: Failed uploads cleaned every 6 hours
- **Database Optimization**: Statistics updated nightly

## Troubleshooting

### Common Issues

1. **Database Connection Failed**
   ```bash
   # Check MySQL service
   systemctl status mysql
   
   # Verify credentials
   mysql -u fixture_user -p
   ```

2. **File Upload Errors**
   ```bash
   # Check permissions
   ls -la /var/lib/fixture-daemon/uploads
   
   # Check disk space
   df -h
   ```

3. **Daemon Won't Start**
   ```bash
   # Check logs
   journalctl -u fixture-daemon -n 50
   
   # Test configuration
   python daemon.py start --foreground
   ```

4. **Permission Denied**
   ```bash
   # Fix ownership
   chown -R fixture:fixture /var/lib/fixture-daemon
   
   # Fix permissions
   chmod 755 /opt/fixture-manager
   ```

### Debug Mode
```bash
# Run in debug mode
export DEBUG=true
python daemon.py start --foreground --config development
```

## API Token Management

### Creating API Tokens

**Via Web Interface:**
1. Login to the web dashboard
2. Navigate to "API Tokens" from the main navigation
3. Click "Create New Token"
4. Provide a descriptive name (e.g., "Mobile App", "Dashboard Integration")
5. Copy the generated token immediately (it's only shown once)
6. Use the token in your external applications

**Token Features:**
- **Secure Generation**: Cryptographically secure random tokens
- **Named Tokens**: Descriptive names for easy identification
- **Expiration Management**: Default 1-year expiration, extendable
- **Usage Tracking**: Last used timestamp and IP address
- **Lifecycle Management**: Revoke, extend, or delete tokens
- **Security**: SHA256 hashed storage, one-time display

### Token Management Operations

**Create Token:**
```bash
# Via API (requires session authentication)
curl -X POST http://your-server/profile/tokens/create \
  -H "Content-Type: application/json" \
  -H "Cookie: session=YOUR_SESSION_COOKIE" \
  -d '{"name": "My API Integration"}'
```

**List User Tokens:**
```bash
# Via web interface at /profile/tokens
# Shows all tokens with status, creation date, expiration, and usage info
```

**Revoke Token:**
```bash
# Via API (requires session authentication)
curl -X POST http://your-server/profile/tokens/123/revoke \
  -H "Cookie: session=YOUR_SESSION_COOKIE"
```

**Extend Token Expiration:**
```bash
# Via API (requires session authentication)
curl -X POST http://your-server/profile/tokens/123/extend \
  -H "Content-Type: application/json" \
  -H "Cookie: session=YOUR_SESSION_COOKIE" \
  -d '{"days": 365}'
```

**Delete Token:**
```bash
# Via API (requires session authentication)
curl -X DELETE http://your-server/profile/tokens/123/delete \
  -H "Cookie: session=YOUR_SESSION_COOKIE"
```

## API Documentation

### Authentication Endpoints
- `POST /auth/api/login` - User login
- `POST /auth/api/logout` - User logout
- `POST /auth/api/refresh` - Refresh JWT token
- `GET /auth/api/profile` - Get user profile

### Token Management Endpoints
- `GET /profile/tokens` - Token management page (web interface)
- `POST /profile/tokens/create` - Create new API token
- `POST /profile/tokens/{id}/revoke` - Revoke API token
- `POST /profile/tokens/{id}/extend` - Extend token expiration
- `DELETE /profile/tokens/{id}/delete` - Delete API token

### Protected API Endpoints (Require API Token)
- `GET /api/fixtures` - List all fixtures with match counts
- `GET /api/matches` - List matches with pagination and filtering
- `GET /api/match/{id}` - Get match details with outcomes
- `GET|POST /api/updates` - **New!** Get fixture updates with incremental sync support

### Upload Endpoints
- `POST /upload/api/fixture` - Upload fixture file
- `POST /upload/api/zip/{match_id}` - Upload ZIP file
- `GET /upload/api/progress/{upload_id}` - Get upload progress
- `GET /upload/api/uploads` - List user uploads

### Administration
- `GET /api/admin/users` - List users (admin)
- `PUT /api/admin/users/{id}` - Update user (admin)
- `GET /api/admin/logs` - System logs (admin)
- `GET /api/admin/system-info` - System information (admin)

## Performance Optimization

### Database Optimization
- Connection pooling with 10 connections
- Query optimization with proper indexing
- Prepared statements for security
- Transaction batching for bulk operations

### File Upload Optimization
- Chunked uploads for large files
- Concurrent upload support (configurable)
- Progress tracking with minimal overhead
- Automatic cleanup of failed uploads

### Caching Strategy
- Session caching with Redis (optional)
- Static file caching with Nginx
- Database query result caching
- API response caching for read-heavy endpoints

## Security Considerations

### Production Deployment
1. **Change Default Credentials**: Update admin password immediately
2. **SSL/TLS Configuration**: Enable HTTPS with valid certificates
3. **Firewall Configuration**: Restrict access to necessary ports only
4. **Regular Updates**: Keep system and dependencies updated
5. **Backup Strategy**: Implement regular automated backups
6. **Monitoring**: Set up comprehensive monitoring and alerting

### Security Best Practices
- Regular security audits
- Penetration testing
- Vulnerability scanning
- Access log monitoring
- Incident response procedures

## Building Single Executable

The project can be packaged as a single executable file for easy distribution with **cross-platform persistent directories**:

### Quick Build
```bash
# Run the automated build script
python build.py
```

### Manual Build
```bash
# Install build dependencies
pip install -r requirements-build.txt

# Build with PyInstaller
pyinstaller --clean fixture-manager.spec
```

The executable will be created in the `dist/` directory and includes:
- All Python dependencies
- Complete Flask application
- Database utilities and models
- Web dashboard and API
- Configuration templates
- **Cross-platform persistent directory support**

**Executable Size**: ~80-120MB
**No Python Installation Required** on target systems
**Cross-Platform Compatibility**: Windows, macOS, and Linux

### PyInstaller Features
- **Persistent Data Storage**: Files persist between application restarts
- **Cross-Platform Directories**: Uses OS-appropriate locations (AppData, Library, /opt)
- **Configuration Migration**: Automatic .env to mbetterd.conf migration
- **Upload Directory Persistence**: ZIP files and fixtures stored outside temp directories
- **Platform Detection**: Automatic PyInstaller environment detection

See [BUILD.md](BUILD.md) for detailed build instructions and troubleshooting.

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Support

For support and questions:
- Check the troubleshooting section
- Review system logs
- See BUILD.md for executable build issues
- Contact system administrator

## API Token Security Best Practices

### For Developers
1. **Store Tokens Securely**: Never commit tokens to version control
2. **Use Environment Variables**: Store tokens in environment variables or secure config files
3. **Rotate Tokens Regularly**: Generate new tokens periodically and revoke old ones
4. **Monitor Usage**: Check token usage logs for suspicious activity
5. **Use Descriptive Names**: Name tokens clearly to identify their purpose
6. **Minimum Permissions**: Only use tokens for their intended purpose

### For System Administrators
1. **Monitor Token Activity**: Review token usage logs regularly
2. **Set Expiration Policies**: Enforce reasonable token expiration periods
3. **Audit Token Access**: Regular audits of active tokens and their usage
4. **Revoke Unused Tokens**: Remove tokens that haven't been used recently
5. **Secure Database**: Ensure API token table is properly secured
6. **Backup Considerations**: Include token management in backup/recovery procedures

### Example Integration

**Python Example:**
```python
import requests

# Store token securely (environment variable recommended)
API_TOKEN = "your-api-token-here"
BASE_URL = "http://your-server"

headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

# Get all fixtures
response = requests.get(f"{BASE_URL}/api/fixtures", headers=headers)
fixtures = response.json()

# Get specific match details
match_id = 123
response = requests.get(f"{BASE_URL}/api/match/{match_id}", headers=headers)
match_details = response.json()
```

**JavaScript Example:**
```javascript
const API_TOKEN = process.env.API_TOKEN; // Store in environment variable
const BASE_URL = 'http://your-server';

const headers = {
    'Authorization': `Bearer ${API_TOKEN}`,
    'Content-Type': 'application/json'
};

// Get all matches
fetch(`${BASE_URL}/api/matches?page=1&per_page=20`, { headers })
    .then(response => response.json())
    .then(data => console.log(data));

// Get fixtures
fetch(`${BASE_URL}/api/fixtures`, { headers })
    .then(response => response.json())
    .then(data => console.log(data));
```

**cURL Examples:**
```bash
# Set token as environment variable
export API_TOKEN="your-api-token-here"

# Get fixtures
curl -H "Authorization: Bearer $API_TOKEN" \
     http://your-server/api/fixtures

# Get matches with filtering
curl -H "Authorization: Bearer $API_TOKEN" \
     "http://your-server/api/matches?fixture_id=abc123&active_only=true"

# Get specific match
curl -H "Authorization: Bearer $API_TOKEN" \
     http://your-server/api/match/123
```

---

**Version**: 1.2.2
**Last Updated**: 2025-08-21
**Minimum Requirements**: Python 3.8+, MySQL 5.7+, Linux/Windows/macOS

### Recent Updates (v1.2.2) - Bug Fix
- ✅ **Fixture Parser Fighter Column Fix**: Fixed critical bug where both fighter1 and fighter2 were incorrectly mapped to fighter1 column during XLSX upload
  - Enhanced [`FixtureParser.detect_required_columns()`](app/upload/fixture_parser.py:179) with specific fighter number matching logic
  - Prevents cross-mapping of fighter columns during partial column name matching
  - Ensures accurate fighter data separation in database records
  - Maintains compatibility with all existing column naming conventions

### Updates (v1.2.1) - PyInstaller Enhancement
- ✅ **Cross-Platform Persistent Directories**: Windows (%APPDATA%), macOS (~/Library/Application Support), Linux (/opt/MBetter)
- ✅ **Configuration Migration**: Automatic .env to mbetterd.conf migration for PyInstaller deployments
- ✅ **Authenticated ZIP Downloads**: Secure API endpoint for ZIP file downloads with token authentication
- ✅ **PyInstaller Detection**: Automatic detection and optimization for PyInstaller environments
- ✅ **Persistent Upload Storage**: Uploads stored outside PyInstaller temp directories
- ✅ **Migration Utility**: migrate_config.py script for environment transition
- ✅ **Platform-Specific Paths**: OS-appropriate directory structures for all platforms

### Updates (v1.2.0) - API Enhancement
- ✅ **New `/api/updates` Endpoint**: Incremental fixture synchronization with timestamp-based filtering
- ✅ **Hybrid Authentication**: JWT and API token support with automatic fallback
- ✅ **Fixture Active Time Tracking**: Automatic timestamp management for fixture activation
- ✅ **SHA1-based ZIP Naming**: Consistent file naming across all upload methods
- ✅ **Configurable API Limits**: System setting for controlling API response sizes
- ✅ **Data Backfill Utility**: Migration tool for existing fixture data
- ✅ **Enhanced Database Schema**: New indexed columns and optimized queries
- ✅ **Flexible HTTP Methods**: Both GET and POST support for API endpoints
- ✅ **Fallback Mechanisms**: Graceful degradation for legacy data compatibility

### Previous Updates (v1.1.0)
- ✅ **API Token Management**: Complete user-generated token system
- ✅ **Enhanced Security**: SHA256 token hashing with usage tracking
- ✅ **Web Interface**: Professional token management UI
- ✅ **Multiple Auth Methods**: Bearer tokens, headers, and query parameters
- ✅ **Token Lifecycle**: Create, revoke, extend, and delete operations
- ✅ **Usage Monitoring**: Last used timestamps and IP tracking
- ✅ **Database Migration**: Automatic schema updates with versioning
- ✅ **REST API Endpoints**: Protected fixture and match data access
- ✅ **Documentation**: Comprehensive API and security guidelines