Fix chroot/sys unmount issue by targeting only chroot-specific rg/fd processes

- Added 9999-cleanup-processes.hook.chroot to terminate only chroot-specific processes
- Enhanced build.sh and cleanup.sh with targeted process termination
- Only terminate rg/fd processes running inside chroot environment
- Added documentation file SYS_UNMOUNT_ISSUE.md

SYS_UNMOUNT_ISSUE.md

+# Fix for chroot/sys Unmount Issue During ISO Creation
+
+## Problem
+During the live CD creation process, the build fails with the error:
+```
+P: Begin unmounting /sys...
+umount: /working/mlivecd/chroot/sys: target is busy.
+E: An unexpected failure occurred, exiting...
+P: Begin unmounting filesystems...
+```
+
+This issue occurs when processes like `rg` (ripgrep) or `fd` (fdfind) are accessing the chroot's `/sys` filesystem, preventing it from being unmounted cleanly.
+
+## Solution Implemented
+
+### 1. Cleanup Hook
+A new hook `config/hooks/live/9999-cleanup-processes.hook.chroot` has been added that:
+- Terminates `rg`/`ripgrep` processes **only inside the chroot environment**
+- Terminates `fd`/`fdfind` processes **only inside the chroot environment**
+- Kills any processes accessing virtual filesystems (`/sys`, `/proc`) **only inside the chroot**
+- Uses both graceful termination and force kill as needed
+- Specifically targets only processes running inside the chroot to avoid affecting system processes
+
+### 2. Enhanced Build Script
+The `build.sh` script now includes:
+- A `cleanup_chroot()` function that terminates problematic processes **only inside the chroot**
+- Retry logic that attempts to rebuild after cleanup if unmount issues occur
+- Detection of "target is busy" errors to trigger the retry mechanism
+
+### 3. Enhanced Cleanup Script
+The `cleanup.sh` script has been enhanced with the same targeted process termination capabilities.
+
+## How It Works
+1. During the chroot phase, if `rg` or `fd` processes are running inside the chroot and accessing the filesystem, they will be terminated
+2. The solution checks if processes are actually running inside the chroot by examining `/proc/$pid/cwd`, `/proc/$pid/exe`, and `/proc/$pid/root`
+3. Only processes with `/chroot/` in their path or working directory are targeted
+4. If the build fails due to unmount issues, the script automatically retries after cleanup
+5. Manual cleanup with `cleanup.sh` also handles these processes
+
+## Testing
+To test if the fix works:
+1. Run `sudo ./build.sh`
+2. If the issue was present, it should now either:
+   - Complete successfully on the first try, or
+   - Complete successfully after the automatic retry
+
+## Additional Notes
+- The hook is numbered `9999` to ensure it runs last in the chroot hooks sequence
+- The solution handles both graceful termination and force kill of processes
+- The retry mechanism provides a fallback if the initial cleanup doesn't resolve the issue
+- Only chroot-specific processes are targeted, ensuring system processes remain unaffected
\ No newline at end of file

build.sh

@@ -34,9 +34,91 @@ if [ "$EUID" -ne 0 ]; then
     exit 1
 fi
 
+# Function to cleanup chroot processes and mounts
+cleanup_chroot() {
+    echo "Cleaning up chroot processes and mounts..."
+    
+    # Kill any processes that might be accessing chroot filesystems
+    if [ -d "chroot" ]; then
+        echo "Terminating processes accessing chroot filesystems..."
+        
+        # Kill rg/ripgrep processes specifically inside chroot
+        echo "Checking for and terminating rg/ripgrep processes inside chroot..."
+        # Find processes with /chroot/ in their path or working directory
+        for pid in $(pgrep -f "rg" 2>/dev/null || true); do
+            # Check if the process is running inside the chroot
+            if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+                echo "Found rg process running inside chroot (PID: $pid), terminating it..."
+                kill -TERM "$pid" 2>/dev/null || true
+            fi
+        done
+
+        # Give processes time to terminate gracefully
+        sleep 1
+
+        # Force kill any remaining rg processes inside chroot
+        for pid in $(pgrep -f "rg" 2>/dev/null || true); do
+            # Check if the process is running inside the chroot
+            if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+                echo "Force killing rg process still running inside chroot (PID: $pid)..."
+                kill -9 "$pid" 2>/dev/null || true
+            fi
+        done
+        
+        # Kill fd/fdfind processes specifically inside chroot
+        echo "Checking for and terminating fd/fdfind processes inside chroot..."
+        for pid in $(pgrep -f "fd" 2>/dev/null || true); do
+            # Check if the process is running inside the chroot
+            if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+                echo "Found fd process running inside chroot (PID: $pid), terminating it..."
+                kill -TERM "$pid" 2>/dev/null || true
+            fi
+        done
+
+        # Give processes time to terminate gracefully
+        sleep 1
+
+        # Force kill any remaining fd processes inside chroot
+        for pid in $(pgrep -f "fd" 2>/dev/null || true); do
+            # Check if the process is running inside the chroot
+            if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+               [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+                echo "Force killing fd process still running inside chroot (PID: $pid)..."
+                kill -9 "$pid" 2>/dev/null || true
+            fi
+        done
+        
+        # Use fuser to kill processes accessing chroot mount points
+        for mount_point in chroot/sys chroot/proc chroot/dev chroot/dev/pts; do
+            if mountpoint -q "$mount_point" 2>/dev/null; then
+                echo "Killing processes accessing $mount_point..."
+                fuser -km "$mount_point" 2>/dev/null || true
+            fi
+        done
+        
+        sleep 2
+        
+        # Try to unmount chroot filesystems
+        for mount_point in chroot/dev/pts chroot/dev chroot/proc chroot/sys; do
+            if mountpoint -q "$mount_point" 2>/dev/null; then
+                echo "Unmounting $mount_point..."
+                umount "$mount_point" 2>/dev/null || umount -f "$mount_point" 2>/dev/null || true
+            fi
+        done
+    fi
+}
+
 echo "Starting ISO build..."
 echo "Running lb config..."
 lb config 2>&1 | tee -a build.log
+
 echo "Running lb build..."
 lb build 2>&1 | tee -a build.log
 
@@ -47,5 +129,26 @@ if [ $BUILD_EXIT_CODE -eq 0 ]; then
     echo "ISO file: $(ls live-image-amd64.iso 2>/dev/null || ls *.iso 2>/dev/null || echo 'Not found')"
 else
     echo "Build failed. Check build.log for errors."
-    exit 1
+    
+    # Check if the failure was due to unmount issues
+    if grep -q "target is busy" build.log 2>/dev/null; then
+        echo "Detected unmount issue. Attempting cleanup and retry..."
+        cleanup_chroot
+        
+        echo "Retrying build after cleanup..."
+        lb build 2>&1 | tee -a build_retry.log
+        BUILD_EXIT_CODE=$?
+        
+        if [ $BUILD_EXIT_CODE -eq 0 ]; then
+            echo "Build completed successfully after retry!"
+            echo "ISO file: $(ls live-image-amd64.iso 2>/dev/null || ls *.iso 2>/dev/null || echo 'Not found')"
+        else
+            echo "Build failed again after retry. Check build.log and build_retry.log for errors."
+            cleanup_chroot
+            exit 1
+        fi
+    else
+        cleanup_chroot
+        exit 1
+    fi
 fi
\ No newline at end of file

cleanup.sh

@@ -65,6 +65,72 @@ fi
 if [ -d "chroot" ]; then
     echo "Checking for mounted filesystems in chroot..."
 
+    # Kill any processes that might be blocking unmount
+    echo "Terminating processes that might block unmount..."
+    
+    # Kill rg/ripgrep processes specifically inside chroot
+    echo "Checking for and terminating rg/ripgrep processes inside chroot..."
+    # Find processes with /chroot/ in their path or working directory
+    for pid in $(pgrep -f "rg" 2>/dev/null || true); do
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Found rg process running inside chroot (PID: $pid), terminating it..."
+            kill -TERM "$pid" 2>/dev/null || true
+        fi
+    done
+
+    # Give processes time to terminate gracefully
+    sleep 1
+
+    # Force kill any remaining rg processes inside chroot
+    for pid in $(pgrep -f "rg" 2>/dev/null || true); do
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Force killing rg process still running inside chroot (PID: $pid)..."
+            kill -9 "$pid" 2>/dev/null || true
+        fi
+    done
+    
+    # Kill fd/fdfind processes specifically inside chroot
+    echo "Checking for and terminating fd/fdfind processes inside chroot..."
+    for pid in $(pgrep -f "fd" 2>/dev/null || true); do
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Found fd process running inside chroot (PID: $pid), terminating it..."
+            kill -TERM "$pid" 2>/dev/null || true
+        fi
+    done
+
+    # Give processes time to terminate gracefully
+    sleep 1
+
+    # Force kill any remaining fd processes inside chroot
+    for pid in $(pgrep -f "fd" 2>/dev/null || true); do
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Force killing fd process still running inside chroot (PID: $pid)..."
+            kill -9 "$pid" 2>/dev/null || true
+        fi
+    done
+    
+    # Use fuser to kill processes accessing chroot mount points
+    for mount_point in chroot/sys chroot/proc chroot/dev chroot/dev/pts; do
+        if mountpoint -q "$mount_point" 2>/dev/null; then
+            echo "Killing processes accessing $mount_point..."
+            fuser -km "$mount_point" 2>/dev/null || true
+        fi
+    done
+    
+    sleep 2
+
     # Try to unmount common mount points
     mount_points=(
         "chroot/dev/pts"
@@ -78,7 +144,7 @@ if [ -d "chroot" ]; then
     for mount_point in "${mount_points[@]}"; do
         if mountpoint -q "$mount_point" 2>/dev/null; then
             echo "Unmounting $mount_point..."
-            umount "$mount_point" 2>/dev/null || true
+            umount "$mount_point" 2>/dev/null || umount -f "$mount_point" 2>/dev/null || true
         fi
     done
 

config/hooks/live/9999-cleanup-processes.hook.chroot

+#!/bin/bash
+
+set -e
+
+echo "Running final cleanup to prevent unmount issues..."
+
+# Kill any ripgrep/rg processes that might be running inside the chroot
+echo "Checking for and terminating rg/ripgrep processes inside chroot..."
+# Find processes with /chroot/ in their path or working directory
+for pid in $(pgrep -f "rg" 2>/dev/null || true); do
+    # Check if the process is running inside the chroot
+    if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+        echo "Found rg process running inside chroot (PID: $pid), terminating it..."
+        kill -TERM "$pid" 2>/dev/null || true
+    fi
+done
+
+# Give processes time to terminate gracefully
+sleep 1
+
+# Force kill any remaining rg processes inside chroot
+for pid in $(pgrep -f "rg" 2>/dev/null || true); do
+    # Check if the process is running inside the chroot
+    if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+        echo "Force killing rg process still running inside chroot (PID: $pid)..."
+        kill -9 "$pid" 2>/dev/null || true
+    fi
+done
+
+# Kill any fdfind/fd processes that might be running inside the chroot
+echo "Checking for and terminating fd/fdfind processes inside chroot..."
+for pid in $(pgrep -f "fd" 2>/dev/null || true); do
+    # Check if the process is running inside the chroot
+    if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+        echo "Found fd process running inside chroot (PID: $pid), terminating it..."
+        kill -TERM "$pid" 2>/dev/null || true
+    fi
+done
+
+# Give processes time to terminate gracefully
+sleep 1
+
+# Force kill any remaining fd processes inside chroot
+for pid in $(pgrep -f "fd" 2>/dev/null || true); do
+    # Check if the process is running inside the chroot
+    if [ -r "/proc/$pid/cwd" ] && readlink /proc/$pid/cwd 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/exe" ] && readlink /proc/$pid/exe 2>/dev/null | grep -q "/chroot/" ||
+       [ -r "/proc/$pid/root" ] && readlink /proc/$pid/root 2>/dev/null | grep -q "/chroot/"; then
+        echo "Force killing fd process still running inside chroot (PID: $pid)..."
+        kill -9 "$pid" 2>/dev/null || true
+    fi
+done
+
+# Kill any other processes that might be accessing /sys or /proc inside the chroot
+echo "Terminating any processes that might be accessing virtual filesystems inside chroot..."
+for proc in $(lsof +D /sys 2>/dev/null | awk 'NR>1 {print $2}' | sort -u 2>/dev/null || true); do
+    if [ -n "$proc" ] && [ "$proc" != "$$" ]; then
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$proc/cwd" ] && readlink /proc/$proc/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/exe" ] && readlink /proc/$proc/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/root" ] && readlink /proc/$proc/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Killing process $proc accessing /sys inside chroot"
+            kill -TERM "$proc" 2>/dev/null || true
+        fi
+    fi
+done
+
+for proc in $(lsof +D /proc 2>/dev/null | awk 'NR>1 {print $2}' | sort -u 2>/dev/null || true); do
+    if [ -n "$proc" ] && [ "$proc" != "$$" ]; then
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$proc/cwd" ] && readlink /proc/$proc/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/exe" ] && readlink /proc/$proc/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/root" ] && readlink /proc/$proc/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Killing process $proc accessing /proc inside chroot"
+            kill -TERM "$proc" 2>/dev/null || true
+        fi
+    fi
+done
+
+# Give processes time to terminate gracefully
+sleep 2
+
+# Force kill any remaining processes accessing virtual filesystems inside the chroot
+for proc in $(lsof +D /sys 2>/dev/null | awk 'NR>1 {print $2}' | sort -u 2>/dev/null || true); do
+    if [ -n "$proc" ] && [ "$proc" != "$$" ]; then
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$proc/cwd" ] && readlink /proc/$proc/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/exe" ] && readlink /proc/$proc/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/root" ] && readlink /proc/$proc/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Force killing process $proc still accessing /sys inside chroot"
+            kill -9 "$proc" 2>/dev/null || true
+        fi
+    fi
+done
+
+for proc in $(lsof +D /proc 2>/dev/null | awk 'NR>1 {print $2}' | sort -u 2>/dev/null || true); do
+    if [ -n "$proc" ] && [ "$proc" != "$$" ]; then
+        # Check if the process is running inside the chroot
+        if [ -r "/proc/$proc/cwd" ] && readlink /proc/$proc/cwd 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/exe" ] && readlink /proc/$proc/exe 2>/dev/null | grep -q "/chroot/" ||
+           [ -r "/proc/$proc/root" ] && readlink /proc/$proc/root 2>/dev/null | grep -q "/chroot/"; then
+            echo "Force killing process $proc still accessing /proc inside chroot"
+            kill -9 "$proc" 2>/dev/null || true
+        fi
+    fi
+done
+
+# Sync filesystem to ensure all writes are flushed
+sync
+
+echo "Process cleanup completed successfully"
\ No newline at end of file