<!-- Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net> -->
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
<!-- Copyleft: GNU GPLv3 or later applies to this file. -->

# Hermes Node Protocol
# Copyright (c) 2026 Stefy (nextime) Lanza <stefy@nexlab.net>
# All rights reserved.
# 
# This software is released under the MIT License with a copyleft clause.
# See the LICENSE file for full terms.
# Hermes Node Protocol - Deployment Guide

**Version:** 1.0  
**Date:** 2026-04-29

Complete deployment guide for setting up the Hermes Node Protocol — a reverse-connection node execution system that replaces OpenClaw sexec with a WebSocket-based architecture.

---

## Overview

The Hermes Node Protocol enables remote command execution without SSH keys. Nodes connect to a central gateway via WebSocket, and commands are routed through this persistent connection. The existing `sexec.sh` permission system is preserved.

**Architecture:**
```
Remote Nodes (sissy, zeiss, spank, ganeti1, ganeti2)
    ↓ WebSocket (node-initiated)
Gateway (Hermes host)
    ↓ HTTP API
Hermes Agent (you)
```

**Key Benefits:**
- ✅ No SSH keys stored on gateway
- ✅ Firewall-friendly (nodes connect out)
- ✅ Reuses existing sexec permission system
- ✅ Token-based authentication
- ✅ Real-time command streaming

---

## Prerequisites

### Gateway Host (Hermes)
- Python 3.7+
- pip3
- Root access (for SysV init service)
- Ports 8765 (WebSocket) and 8766 (HTTP) available

### Remote Nodes
- Python 3.7+
- pip3
- Existing `sexec.sh` installation with `config.json`
- Network access to gateway host

---

## Part 1: Gateway Installation

### Step 1: Install Gateway

On the Hermes host:

```bash
cd ~/hermes-node-protocol/gateway
sudo ./install.sh
```

This will:
1. Install Python dependencies (websockets, aiohttp)
2. Create `hermes` system user
3. Create config directory `/etc/hermes-node-gateway/`
4. Generate random tokens for each node
5. Install SysV init service
6. Create log file `/var/log/hermes-node-gateway.log`

### Step 2: Review Configuration

```bash
sudo cat /etc/hermes-node-gateway/config.json
```

**Example config:**
```json
{
  "websocket_port": 8765,
  "http_port": 8766,
  "bind_address": "0.0.0.0",
  "tokens": {
    "sissy": "a1b2c3d4e5f6...",
    "zeiss": "f6e5d4c3b2a1...",
    "spank": "1a2b3c4d5e6f...",
    "ganeti1": "6f5e4d3c2b1a...",
    "ganeti2": "abcdef123456..."
  }
}
```

**Important:** Save these tokens — you'll need them for each node's configuration.

### Step 3: Start Gateway

```bash
/etc/init.d/hermes-node-gateway start
/etc/init.d/hermes-node-gateway status
```

**Check logs:**
```bash
tail -f /var/log/hermes-node-gateway.log
```

You should see:
```
Starting Hermes Node Gateway...
HTTP API listening on 0.0.0.0:8766
WebSocket server listening on 0.0.0.0:8765
Gateway is running
```

### Step 4: Verify Gateway

```bash
# Check HTTP API
curl http://localhost:8766/nodes

# Should return:
{"nodes": []}
```

Gateway is now ready to accept node connections.

---

## Part 2: Node Agent Installation

Repeat these steps for **each remote node** (sissy, zeiss, spank, ganeti1, ganeti2).

### Step 1: Copy Installer to Node

From the gateway host:

```bash
# For sissy
scp -r ~/hermes-node-protocol/node-agent openclaw@192.168.42.115:/tmp/

# For zeiss
scp -r ~/hermes-node-protocol/node-agent nextime@192.168.42.3:/tmp/

# For spank
scp -r ~/hermes-node-protocol/node-agent openclaw@192.168.231.26:/tmp/

# For ganeti1
scp -r ~/hermes-node-protocol/node-agent root@192.168.250.1:/tmp/

# For ganeti2
scp -r ~/hermes-node-protocol/node-agent root@192.168.11.1:/tmp/
```

### Step 2: Install Agent on Node

SSH to the node:

```bash
ssh openclaw@192.168.42.115  # or appropriate user@ip
```

Run installer:

```bash
cd /tmp/node-agent
./install.sh
```

This will:
1. Install Python dependencies (websockets)
2. Create config directory `/etc/hermes-node/`
3. Install agent script to `/usr/local/bin/hermes-node-agent`
4. Create example config with random token
5. Install SysV init service
6. Update init runlevel links

### Step 3: Configure Node

Edit the config file:

```bash
nano /etc/hermes-node/config.json
```

**Update these fields:**

```json
{
  "gateway_url": "ws://192.168.42.115:8765",
  "node_name": "sissy",
  "token": "PASTE-TOKEN-FROM-GATEWAY-CONFIG",
  "sexec_path": "/home/openclaw/.openclaw/skills/sexec/sexec.sh",
  "reconnect_interval": 5,
  "heartbeat_interval": 30
}
```

**Important:**
- `gateway_url`: WebSocket URL of your gateway (use gateway's IP)
- `node_name`: Must match the name in gateway's tokens config
- `token`: Copy from gateway's `/etc/hermes-node-gateway/config.json`
- `sexec_path`: Path to your existing sexec.sh installation

### Step 4: Verify sexec Installation

Make sure sexec is installed and working:

```bash
/home/openclaw/.openclaw/skills/sexec/sexec.sh run --command "hostname"
```

Should return the hostname without errors.

### Step 5: Start Node Agent

```bash
/etc/init.d/hermes-node-agent start
/etc/init.d/hermes-node-agent status
```

**Check logs:**
```bash
tail -f /var/log/hermes-node-agent.log
```

You should see:
```
Connecting to gateway: ws://192.168.42.115:8765
Connected to gateway
Sent registration for node 'sissy'
Registration acknowledged by gateway (version 1.0)
```

### Step 6: Verify Connection

Back on the gateway host, check connected nodes:

```bash
curl http://localhost:8766/nodes | jq
```

Should show:
```json
{
  "nodes": [
    {
      "name": "sissy",
      "status": "connected",
      "last_seen": 1714392000,
      "uptime": 123,
      "version": "1.0",
      "capabilities": ["exec", "sysinfo"]
    }
  ]
}
```

---

## Part 3: Testing

### Test 1: Simple Command

```bash
curl -X POST http://localhost:8766/nodes/sissy/exec \
  -H "Content-Type: application/json" \
  -d '{"command": ["hostname"], "timeout": 10}' | jq
```

**Expected output:**
```json
{
  "id": "cmd-a1b2c3d4",
  "status": "completed",
  "stdout": "sissy\n",
  "stderr": "",
  "exit_code": 0,
  "error": null,
  "duration_ms": 234
}
```

### Test 2: Command with Output

```bash
curl -X POST http://localhost:8766/nodes/sissy/exec \
  -H "Content-Type: application/json" \
  -d '{"command": ["df", "-h"], "timeout": 10}' | jq -r '.stdout'
```

Should display disk usage.

### Test 3: Using Helper Script

```bash
~/hermes-node-protocol/scripts/node-exec.sh sissy hostname
~/hermes-node-protocol/scripts/node-exec.sh sissy df -h
~/hermes-node-protocol/scripts/node-exec.sh sissy uptime
```

### Test 4: All Nodes

```bash
for node in sissy zeiss spank ganeti1 ganeti2; do
  echo "=== $node ==="
  ~/hermes-node-protocol/scripts/node-exec.sh $node hostname 2>/dev/null || echo "  (not connected)"
done
```

---

## Part 4: Integration with Hermes

The `hermes-node-exec` skill is already installed at `~/.hermes/skills/hermes-node-exec/`.

### Usage from Hermes

When you ask me to run commands on nodes, I'll use the HTTP API:

**You say:**
> "Check disk usage on sissy"

**I execute:**
```bash
curl -s -X POST http://localhost:8766/nodes/sissy/exec \
  -H "Content-Type: application/json" \
  -d '{"command": ["df", "-h"], "timeout": 30}' | jq -r '.stdout'
```

**You say:**
> "List Ganeti instances on ganeti1"

**I execute:**
```bash
curl -s -X POST http://localhost:8766/nodes/ganeti1/exec \
  -H "Content-Type: application/json" \
  -d '{"command": ["sudo", "gnt-instance", "list"], "timeout": 30}' | jq -r '.stdout'
```

---

## Troubleshooting

### Node Won't Connect

**Symptom:** Node agent logs show connection errors

**Check:**
1. Gateway is running: `/etc/init.d/hermes-node-gateway status`
2. Firewall allows port 8765: `sudo ufw status`
3. Gateway URL is correct in node config
4. Token matches between node and gateway configs

**Debug:**
```bash
# On node
tail -n 50 /var/log/hermes-node-agent.log

# On gateway
sudo tail -n 50 /var/log/hermes-node-gateway.log
```

### Invalid Token

**Symptom:** Node logs show "Invalid token"

**Solution:**
1. Check gateway config: `sudo cat /etc/hermes-node-gateway/config.json`
2. Copy correct token to node config: `/etc/hermes-node/config.json`
3. Restart node agent: `/etc/init.d/hermes-node-agent restart`

### Command Denied

**Symptom:** Command returns "Denied: ..."

**Solution:**
The command is in the node's sexec deny list. Either:
1. Remove from deny list: Edit `~/.openclaw/skills/sexec/config.json` on the node
2. Use a different command

### Command Requires Approval

**Symptom:** Command returns "approval_required"

**Solution:**
The command is in the sexec "ask" list. Get user approval, then re-run with `"approved": true`:

```bash
curl -X POST http://localhost:8766/nodes/sissy/exec \
  -H "Content-Type: application/json" \
  -d '{"command": ["sudo", "..."], "timeout": 30, "approved": true}'
```

### Gateway Not Responding

**Symptom:** `curl: (7) Failed to connect`

**Check:**
```bash
/etc/init.d/hermes-node-gateway status
sudo netstat -tlnp | grep 8766
```

**Restart:**
```bash
/etc/init.d/hermes-node-gateway restart
```

---

## Security Considerations

### 1. Token Security

- Tokens are stored in `/etc/hermes-node-gateway/config.json` (mode 600, owned by hermes)
- Each node has a unique token
- Tokens are 64-character random hex strings
- If a token is compromised, regenerate and update both gateway and node configs

### 2. Network Security

- Gateway listens on `0.0.0.0` by default (all interfaces)
- Consider binding to specific interface: `"bind_address": "192.168.42.115"`
- Use firewall rules to restrict access:
  ```bash
  sudo ufw allow from 192.168.42.0/24 to any port 8765
  sudo ufw allow from 192.168.42.0/24 to any port 8766
  ```

### 3. Permission System

- Each node enforces its own sexec allow/deny/ask lists
- Gateway cannot bypass node permissions
- Commands in "ask" list require explicit user approval
- Commands in "deny" list are rejected immediately

### 4. TLS/WSS (Future Enhancement)

Current version uses unencrypted WebSocket (ws://). For production:
- Use WSS (WebSocket Secure) with TLS certificates
- Encrypt HTTP API with HTTPS
- Use certificate-based authentication instead of tokens

---

## Maintenance

### View Gateway Logs

```bash
tail -f /var/log/hermes-node-gateway.log
```

### View Node Logs

```bash
ssh user@node 'tail -f /var/log/hermes-node-agent.log'
```

### Restart Gateway

```bash
/etc/init.d/hermes-node-gateway restart
```

### Restart Node Agent

```bash
ssh user@node '/etc/init.d/hermes-node-agent restart'
```

### Update Gateway

```bash
cd ~/hermes-node-protocol/gateway
sudo cp hermes_node_gateway.py /usr/local/bin/hermes-node-gateway
/etc/init.d/hermes-node-gateway restart
```

### Update Node Agent

```bash
# On each node
cd /tmp/node-agent
sudo cp hermes_node_agent.py /usr/local/bin/hermes-node-agent
/etc/init.d/hermes-node-agent restart
```

---

## Migration from OpenClaw

If you're migrating from OpenClaw sexec:

### What to Keep
- ✅ Keep `sexec.sh` on all nodes
- ✅ Keep `config.json` permission files
- ✅ Keep existing allow/deny/ask lists

### What to Change
- ❌ Remove OpenClaw gateway (optional)
- ❌ Remove OpenClaw node agents (optional)
- ✅ Install Hermes Node Gateway
- ✅ Install Hermes Node Agents
- ✅ Update Hermes skills to use new HTTP API

### Coexistence

You can run both systems in parallel during migration:
- OpenClaw gateway on different ports
- Hermes gateway on ports 8765/8766
- Both can use the same sexec.sh installations

---

## Next Steps

1. ✅ Gateway installed and running
2. ✅ All nodes connected
3. ✅ Test commands working
4. ✅ Hermes skill installed

**You're ready to use the system!**

Try asking me:
- "Check disk usage on all nodes"
- "List Ganeti instances on ganeti1"
- "Check uptime on zeiss"

I'll use the Hermes Node Protocol to execute these commands.

---

## Files Reference

| File | Location | Purpose |
|------|----------|---------|
| Protocol Spec | `~/hermes-node-protocol/PROTOCOL.md` | Message format and architecture |
| Gateway Script | `~/hermes-node-protocol/gateway/hermes_node_gateway.py` | Gateway server code |
| Gateway Config | `/etc/hermes-node-gateway/config.json` | Gateway configuration and tokens |
| Gateway Service | `/etc/init.d/hermes-node-gateway` | SysV init script |
| Node Agent Script | `~/hermes-node-protocol/node-agent/hermes_node_agent.py` | Node agent code |
| Node Agent Config | `/etc/hermes-node/config.json` | Node configuration (on each node) |
| Node Agent Service | `/etc/init.d/hermes-node-agent` | SysV init script |
| Helper Script | `~/hermes-node-protocol/scripts/node-exec.sh` | CLI wrapper |
| Hermes Skill | `~/.hermes/skills/hermes-node-exec/SKILL.md` | Skill documentation |

---

## Support

For issues or questions:
1. Check logs (gateway and node)
2. Verify network connectivity
3. Confirm tokens match
4. Test with simple commands first
5. Review the protocol spec for message formats

**Common issues are documented in the Troubleshooting section above.**
