Commit 597a2376 authored by Lisa (AI Assistant)'s avatar Lisa (AI Assistant)

docs: update Android capability status

parent afd6c51a
Pipeline #327 canceled with stages
......@@ -12,9 +12,9 @@ git@git.nexlab.net:lisa/hermes-node-android.git
## Current status
Buildable first test APK.
Working Android node APK that connects to the Hermes Node Gateway and implements a meaningful first real capability set.
Implemented enough to install on a phone and connect it to the Hermes Node Gateway:
Implemented:
- Android/Kotlin app project with Gradle wrapper.
- Config UI for gateway URL, token, node name, TLS mode, and advertised capabilities.
......@@ -23,29 +23,46 @@ Implemented enough to install on a phone and connect it to the Hermes Node Gatew
- Reconnect loop and heartbeat frames.
- App-local config via `SharedPreferences`; no hardcoded token.
- Status display in the app.
- Permission/setup helpers in the UI for camera, microphone, Accessibility, and screenshot consent.
- Debug APK build verified locally.
Baseline capability handlers:
Implemented capability handlers:
- `computer_control`
- `launch_app`: implemented via package launch intent.
- `open_url`: implemented via `ACTION_VIEW`.
- `open_intent`: implemented for implicit Android intents.
- gesture/key/text actions return explicit unsupported until AccessibilityService backend is added.
- `tap`, `swipe`, `long_press`: implemented via AccessibilityService gestures.
- `type_text`: implemented via Accessibility focused-node text injection.
- `key_press`: implemented for Android global actions like `back`, `home`, `recents`, `notifications`, `quick_settings`.
- `desktop_observe`
- `screen_info`: implemented.
- `active_window`: implemented when AccessibilityService is enabled.
- `clipboard_get`: implemented.
- cursor/mouse-position compatibility returns Android-specific note.
- screenshots/active-window/clipboard return explicit unsupported until Android permission backends are added.
- `list_windows` and `window_geometry`: compatibility implementations.
- screenshot consent flow is wired, but bitmap capture/export is still incomplete.
- `audio_control`
- `get_audio_status`: implemented.
- other actions return explicit unsupported until media/mic backend is added.
- `list_audio_devices`: implemented via `AudioDeviceInfo`.
- `capture_input`: implemented via `AudioRecord`, returning WAV as base64.
- `capture_output`: explicitly unsupported on stock Android.
- `camera_control`
- `list_cameras`: implemented via Camera2.
- `get_camera_status`: implemented.
- capture/injection actions return explicit unsupported until CameraX/MediaProjection UX exists.
- `capture_frame`: implemented via Camera2 + `ImageReader`, returning base64 JPEG.
- `inject_video`: explicitly unsupported on stock Android.
- `exec`
- advertised only if enabled in UI, but backend intentionally returns unsupported for now.
Still incomplete by platform reality or next-stage engineering:
- `desktop_observe.screenshot` / `region_screenshot`: MediaProjection consent is wired, but the capture pipeline still needs completion.
- `audio_control.play_audio`: needs cache/download and playback backend.
- `camera_control.capture_video`: needs longer-lived visible recording pipeline and output policy.
- `browser_control`: not implemented yet.
- `exec`: intentionally disabled as non-parity Android shell execution.
## Capability naming decision
Use existing gateway capability names where practical:
......@@ -102,11 +119,18 @@ app/build/outputs/apk/debug/app-debug.apk
4. Tap **Save & start node**.
5. Android should show a persistent foreground-service notification.
6. On Hermes, check `node_list` / gateway node status for the Android node name.
7. Smoke-test safe actions:
7. In the Android app, also tap:
- **Grant camera/microphone permissions**
- **Open Accessibility settings** and enable Hermes Node if you want gestures/active window support
- **Grant screenshot consent** if you want to prepare for screenshot testing
8. Smoke-test actions from Hermes:
- `desktop_observe``screen_info`
- `desktop_observe``active_window`
- `audio_control``get_audio_status`
- `audio_control``list_audio_devices`
- `camera_control``list_cameras`
- `computer_control``open_url`
- `computer_control``tap` / `swipe` after Accessibility is enabled
## Repository layout
......
......@@ -8,15 +8,21 @@ The Android agent should expose **the same gateway capability names** as Linux w
Meaning on Android: phone/device control.
Current first-test APK actions:
Current implemented actions:
- `launch_app`: implemented; starts app by package name.
- `open_url`: implemented; opens URL via Android intent.
- `open_intent`: implemented for implicit Android intent bridge.
- `key_press`: returns explicit unsupported until AccessibilityService/privileged input backend exists.
- `tap`, `swipe`, `long_press`: return explicit unsupported until AccessibilityService gesture dispatch exists.
- `type_text`: returns explicit unsupported until accessibility/IME-backed text entry exists.
- `get_active_window`: route through `desktop_observe`; foreground app/activity needs UsageStats or Accessibility grant.
- `key_press`: implemented for Android global actions via AccessibilityService (`back`, `home`, `recents`, `notifications`, `quick_settings`, `power_dialog`).
- `tap`, `swipe`, `long_press`: implemented via AccessibilityService gesture dispatch.
- `type_text`: implemented via Accessibility node focused-text injection.
- `mouse_move`: implemented as Android no-op compatibility response; Android has no persistent cursor.
- `get_active_window`: still routed through `desktop_observe`.
Explicit limitations that remain honest:
- Arbitrary raw key injection like desktop Enter/volume semantics is not available through normal unprivileged Android app APIs.
- Anything requiring shell-level input injection or hidden APIs is intentionally not faked.
Keep the gateway name `computer_control` unless gateway-side UX later needs a friendly alias. Compatibility beats naming purity here.
......@@ -24,33 +30,35 @@ Keep the gateway name `computer_control` unless gateway-side UX later needs a fr
Meaning on Android: screen/device observation.
Current first-test APK actions:
Current implemented actions:
- `screen_info`: implemented; display metrics, density, orientation, interactive state.
- `active_window` / `get_active_window`: explicit unsupported until UsageStats or AccessibilityService grant flow exists.
- `screenshot` / `region_screenshot`: explicit unsupported until MediaProjection consent flow exists.
- `clipboard_get`: explicit unsupported until Android foreground/API restrictions are handled.
- `active_window` / `get_active_window`: implemented via AccessibilityService when enabled.
- `clipboard_get`: implemented.
- `cursor_position` / `mouse_position`: implemented as compatibility shim; Android has touch focus, not a persistent cursor.
- `list_windows`: implemented as Android foreground-window model compatibility.
- `window_geometry`: implemented as screen geometry fallback.
- `screenshot` / `region_screenshot`: consent flow is wired, but bitmap capture/export is still incomplete until the MediaProjection virtual-display pipeline is finished.
### `audio_control`
Current first-test APK actions:
Current implemented actions:
- `get_audio_status`: implemented; stream volume, ringer mode, speakerphone flag, mic mute flag.
- `list_audio_devices`: explicit unsupported until AudioDeviceInfo enumeration is wired.
- `play_audio`: explicit unsupported until download/cache + MediaPlayer/ExoPlayer backend exists.
- `capture_input`: explicit unsupported until microphone permission and foreground recording UX exist.
- `capture_output`: explicit unsupported; normal Android apps generally cannot capture arbitrary output audio.
- `list_audio_devices`: implemented via `AudioDeviceInfo`.
- `capture_input`: implemented via `AudioRecord`, returning WAV payload as base64 when microphone permission is granted.
- `play_audio`: still incomplete until download/cache + MediaPlayer/ExoPlayer backend exists.
- `capture_output`: explicitly unsupported; normal Android apps generally cannot capture arbitrary output audio.
### `camera_control`
Current first-test APK actions:
Current implemented actions:
- `list_cameras`: implemented with Camera2 camera inventory.
- `list_cameras`: implemented with Camera2 camera inventory and JPEG size hints.
- `get_camera_status`: implemented with backend/status metadata.
- `capture_frame`: explicit unsupported until CameraX/Camera2 capture session and permission UX exist.
- `capture_video`: explicit unsupported until CameraX/Camera2 capture session and visible UX exist.
- `inject_video`: explicit unsupported on normal Android; no fake parity.
- `capture_frame`: implemented with Camera2 + `ImageReader`, returning base64 JPEG when camera permission is granted.
- `capture_video`: still incomplete; this needs a longer-lived visible recording pipeline and storage/streaming policy.
- `inject_video`: explicitly unsupported on normal Android; no fake parity.
### `browser_control`
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment