upgrade: fall back to the public mirror when the default remote is unreachable

`coderai-docker --upgrade` was unusable: git.nexlab.net's certificate expired on
2026-09-15, the clone failed, and the script gave up — although the same
production branch sits on the public GitHub mirror. Nobody running the published
image has a stake in that host, so an unreachable default is now a detour, not a
dead end.

- packaging/linux/launcher/coderai-upgrade: clone_from() per remote; a connection
  /TLS/DNS failure on the DEFAULT remote retries the mirror and logs both the
  reason and the source it ended up using. A repo the operator named
  (--upgrade-repo / CODERAI_UPGRADE_REPO) or an SSH-key clone is never silently
  replaced, and a ref that does not exist stays an error — papering over that
  with the mirror would install the wrong code.
- Two bugs found while testing it, both silent:
  * `if ! cmd; then rc=$?` captured the status of the negation (0 on failure),
    so the fallback branch could never run;
  * reqs_content() ran under `set -o pipefail` with a `[[ -f ]] && grep` group,
    so a tree without packaging/common/requirements-oci.txt (or with no deps at
    all) aborted the upgrade one line after "upgrading x -> y", with no message.
- tests/test_upgrade_launcher.py: the real script against local git repos — no
  network — for the healthy default, the fallback, the explicit repo, a missing
  ref and an up-to-date install.

The certificate itself is on the host serving git.nexlab.net (94.23.108.54, not
lisa) and still needs renewing there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi

packaging/linux/launcher/coderai-upgrade

@@ -20,7 +20,9 @@
 #
 # Configuration (all optional; the host runner passes these as env):
 #   CODERAI_UPGRADE_REPO   git URL to fetch (https or ssh). Default: the nexlab
-#                          HTTPS repo, or its SSH form when a key is provided.
+#                          HTTPS repo, or its SSH form when a key is provided;
+#                          when THAT default is unreachable (expired cert, DNS,
+#                          outage) the public GitHub mirror is used instead.
 #   CODERAI_UPGRADE_REF    branch/tag/commit to fetch. Default: production.
 #   CODERAI_UPGRADE_FORCE  "1" to upgrade even if not strictly newer.
 #   CODERAI_UPGRADE_SSH_KEY  path (inside the container) to an SSH private key;
@@ -30,6 +32,12 @@ set -euo pipefail
 APP_DIR="/opt/coderai/app"
 DEFAULT_HTTPS_REPO="https://git.nexlab.net/nexlab/coderai.git"
 DEFAULT_SSH_REPO="git@git.nexlab.net:nexlab/coderai.git"
+# The public mirror, tried when the default remote cannot be reached at all —
+# an expired certificate or a DNS/outage on one host must not make --upgrade
+# impossible, least of all for someone running the published image who has no
+# stake in that host. Only used as a FALLBACK to the defaults above: a repo the
+# operator named explicitly is never silently replaced.
+MIRROR_REPO="https://github.com/nextime/coderai.git"
 
 REF="${CODERAI_UPGRADE_REF:-production}"
 FORCE="${CODERAI_UPGRADE_FORCE:-0}"
@@ -42,8 +50,10 @@ PYBIN="/opt/coderai/python/bin/python3"
 
 # Pick the default repo form to match the auth method: SSH URL when a key was
 # supplied, HTTPS otherwise. An explicit CODERAI_UPGRADE_REPO always wins.
+REPO_EXPLICIT=0
 if [[ -n "${CODERAI_UPGRADE_REPO:-}" ]]; then
   REPO="$CODERAI_UPGRADE_REPO"
+  REPO_EXPLICIT=1
 elif [[ -n "$SSH_KEY" ]]; then
   REPO="$DEFAULT_SSH_REPO"
 else
@@ -85,12 +95,17 @@ vercmp() {
 # stripped so the diff below reflects real dependency changes, not formatting.
 reqs_content() {
   local d="$1"
+  # `set -o pipefail` is on, so every stage's status counts: a tree without
+  # requirements-oci.txt made the group exit 1 and killed the whole upgrade one
+  # line after "upgrading x -> y" with no message at all. A missing file and an
+  # empty result are both valid answers here.
   { [[ -f "$d/requirements.txt" ]] && cat "$d/requirements.txt"
     [[ -f "$d/packaging/common/requirements-oci.txt" ]] \
       && grep -vE '^[[:space:]]*-r[[:space:]]' "$d/packaging/common/requirements-oci.txt"
+    true
   } 2>/dev/null \
     | sed 's/#.*//' | sed 's/[[:space:]]*$//' \
-    | grep -vE '^[[:space:]]*$' | sort -u
+    | { grep -vE '^[[:space:]]*$' || true; } | sort -u
 }
 
 # Refresh the launcher scripts and service configs that live OUTSIDE the app tree
@@ -161,21 +176,57 @@ cleanup(){ rm -rf "$WORK"; }
 trap cleanup EXIT
 
 log "fetching '$REF'…"
+
 # Shallow single-branch clone: minimal transfer, no history. A ref that is a tag
 # or bare commit also works with --branch for tags; for a raw commit we fall back
 # to a full-ish fetch.
-if ! git clone --depth 1 --branch "$REF" --single-branch "$REPO" "$WORK/src" 2>"$WORK/clone.err"; then
-  if grep -qiE "Could not find remote branch|not found in upstream" "$WORK/clone.err"; then
-    log "ref '$REF' is not a branch/tag; trying to fetch it as a commit…"
+clone_from() {   # clone_from <repo> <errfile> -> 0 ok, 1 unreachable, 2 no such ref
+  local repo="$1" err="$2"
+  rm -rf "$WORK/src"
+  if git clone --depth 1 --branch "$REF" --single-branch "$repo" "$WORK/src" 2>"$err"; then
+    return 0
+  fi
+  if grep -qiE "Could not find remote branch|not found in upstream" "$err"; then
+    log "ref '$REF' is not a branch/tag on $repo; trying to fetch it as a commit…"
+    rm -rf "$WORK/src"
     git init -q "$WORK/src"
-    git -C "$WORK/src" remote add origin "$REPO"
-    git -C "$WORK/src" fetch -q --depth 1 origin "$REF" || { cat "$WORK/clone.err" >&2; die "fetch of ref '$REF' failed"; }
-    git -C "$WORK/src" checkout -q FETCH_HEAD
+    git -C "$WORK/src" remote add origin "$repo"
+    if git -C "$WORK/src" fetch -q --depth 1 origin "$REF" 2>>"$err"; then
+      git -C "$WORK/src" checkout -q FETCH_HEAD
+      return 0
+    fi
+    return 2
+  fi
+  return 1
+}
+
+# NOTE: `if ! cmd; then rc=$?` captures the status of the NEGATION (0 when the
+# command failed), not the command's — which is exactly how the fallback below
+# was first written, and why it never ran. Capture it with `|| rc=$?`.
+rc=0
+clone_from "$REPO" "$WORK/clone.err" || rc=$?
+if [[ "$rc" -ne 0 ]]; then
+  # A TLS/DNS/connection problem with the default remote is not the operator's
+  # fault and not a reason to give up: the public mirror carries the same
+  # production branch. A ref that simply does not exist IS a reason to stop.
+  if [[ "$rc" -eq 1 && "$REPO_EXPLICIT" -eq 0 && "$REPO" != "$MIRROR_REPO" && -z "$SSH_KEY" ]]; then
+    log "cannot reach $REPO:"
+    sed 's/^/[upgrade]   /' "$WORK/clone.err" >&2
+    log "falling back to the public mirror $MIRROR_REPO …"
+    if ! clone_from "$MIRROR_REPO" "$WORK/clone2.err"; then
+      cat "$WORK/clone.err" "$WORK/clone2.err" >&2
+      die "neither $REPO nor the mirror could be cloned (network? proxy? wrong ref '$REF'?)"
+    fi
+    REPO="$MIRROR_REPO"
   else
     cat "$WORK/clone.err" >&2
-    die "clone failed (auth? network? wrong repo/ref?)"
+    if [[ "$rc" -eq 2 ]]; then
+      die "ref '$REF' does not exist on $REPO (use CODERAI_UPGRADE_REF / --upgrade-ref)"
+    fi
+    die "clone of $REPO failed (auth? network? certificate? use --upgrade-repo to name another)"
   fi
 fi
+log "source used:       $REPO"
 
 NEW_VER="$(read_version "$WORK/src/codai/__init__.py" || true)"
 [[ -n "$NEW_VER" ]] || die "fetched tree has no codai/__init__.py __version__"

packaging/linux/run_oci.sh

@@ -131,7 +131,9 @@ Upgrade (refresh the in-image code instead of running the server):
                       not strictly newer than what's installed.
   --upgrade-ref REF   Branch/tag/commit to upgrade to (default: production).
   --upgrade-repo URL  Git URL to fetch from (default: the nexlab HTTPS repo, or
-                      its SSH form when --ssh-key is given).
+                      its SSH form when --ssh-key is given; if that default
+                      cannot be reached — expired certificate, DNS, outage — the
+                      public GitHub mirror is used automatically).
   --ssh-key PATH      Host path to an SSH private key; mounted into the upgrade
                       container so git can authenticate over SSH.
   --no-pip            With --upgrade, refresh the code only; do not re-run pip

tests/test_upgrade_launcher.py

+"""The in-image upgrader (`packaging/linux/launcher/coderai-upgrade`).
+
+`coderai-docker --upgrade` is how a running install gets new code without a
+rebuild, and it broke for everyone when one git host's certificate expired: the
+default remote was unreachable and the script simply died, although the same
+branch sits on a public mirror. These tests run the real script against local
+git repositories — no network — and pin the behaviour that matters:
+
+* a healthy default remote is used and nothing else is contacted;
+* an unreachable default falls back to the mirror and says so;
+* a remote the operator named explicitly is never silently replaced;
+* a ref that does not exist is an error, not something to paper over with the
+  mirror (that would upgrade to the wrong code).
+"""
+
+import os
+import shutil
+import subprocess
+import time
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parents[1]
+SCRIPT = ROOT / "packaging" / "linux" / "launcher" / "coderai-upgrade"
+
+pytestmark = pytest.mark.skipif(shutil.which("git") is None or not SCRIPT.is_file(),
+                                reason="needs git and the upgrader script")
+
+
+def _git(*args, cwd):
+    subprocess.run(["git", *args], cwd=str(cwd), check=True,
+                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+
+
+def _repo(path: Path, version: str, branch: str = "production") -> str:
+    """A minimal coderai tree as a git repo, on `branch`."""
+    path.mkdir(parents=True, exist_ok=True)
+    _git("init", "-q", "-b", branch, cwd=path)
+    _git("config", "user.email", "t@t", cwd=path)
+    _git("config", "user.name", "t", cwd=path)
+    pkg = path / "codai"
+    pkg.mkdir(exist_ok=True)
+    (pkg / "__init__.py").write_text(f'__version__ = "{version}"\n')
+    (path / "requirements.txt").write_text("fastapi\n")
+    _git("add", "-A", cwd=path)
+    _git("commit", "-q", "-m", "x", cwd=path)
+    return str(path)
+
+
+def _app(path: Path, version: str) -> Path:
+    """A fake installed app tree the script upgrades in place.
+
+    The files are back-dated a day: the upgrade replaces the tree with rsync,
+    whose quick check is size + mtime, and two version strings of the same length
+    written in the same second look identical to it. In a real image the app tree
+    carries the build's timestamp, which is what this reproduces."""
+    (path / "codai").mkdir(parents=True, exist_ok=True)
+    (path / "codai" / "__init__.py").write_text(f'__version__ = "{version}"\n')
+    (path / "requirements.txt").write_text("fastapi\n")
+    old = time.time() - 86400
+    for f in (path, path / "codai", path / "codai" / "__init__.py",
+              path / "requirements.txt"):
+        os.utime(f, (old, old))
+    return path
+
+
+def _run(tmp_path, app: Path, *, default_repo: str, mirror: str,
+         explicit: str = "", ref: str = "production", ssh_key: str = ""):
+    """Run the script with its two defaults rewritten to local repos."""
+    src = SCRIPT.read_text()
+    src = src.replace('APP_DIR="/opt/coderai/app"', f'APP_DIR="{app}"')
+    src = src.replace('DEFAULT_HTTPS_REPO="https://git.nexlab.net/nexlab/coderai.git"',
+                      f'DEFAULT_HTTPS_REPO="{default_repo}"')
+    src = src.replace('MIRROR_REPO="https://github.com/nextime/coderai.git"',
+                      f'MIRROR_REPO="{mirror}"')
+    src = src.replace('PYBIN="/opt/coderai/python/bin/python3"',
+                      f'PYBIN="{sys.executable}"')
+    script = tmp_path / "upgrade.sh"
+    script.write_text(src)
+    script.chmod(0o755)
+    env = dict(os.environ, CODERAI_UPGRADE_REF=ref, CODERAI_UPGRADE_SKIP_PIP="1")
+    if explicit:
+        env["CODERAI_UPGRADE_REPO"] = explicit
+    if ssh_key:
+        env["CODERAI_UPGRADE_SSH_KEY"] = ssh_key
+    return subprocess.run([str(script)], capture_output=True, text=True, env=env,
+                          timeout=180)
+
+
+def test_the_default_remote_is_used_when_it_works(tmp_path):
+    app = _app(tmp_path / "app", "0.2.21")
+    good = _repo(tmp_path / "default", "0.2.22")
+    mirror = _repo(tmp_path / "mirror", "9.9.9")        # must NOT be used
+    r = _run(tmp_path, app, default_repo=good, mirror=mirror)
+    assert r.returncode == 0, r.stderr
+    assert "source used:       " + good in r.stderr
+    assert "mirror" not in r.stderr
+    assert '"0.2.22"' in (app / "codai" / "__init__.py").read_text()
+
+
+def test_an_unreachable_default_falls_back_to_the_mirror(tmp_path):
+    app = _app(tmp_path / "app", "0.2.21")
+    mirror = _repo(tmp_path / "mirror", "0.2.22")
+    # A remote that fails the way an expired certificate fails: cannot connect.
+    dead = "https://127.0.0.1:1/nexlab/coderai.git"
+    r = _run(tmp_path, app, default_repo=dead, mirror=mirror)
+    assert r.returncode == 0, r.stderr
+    assert "cannot reach " + dead in r.stderr
+    assert "falling back to the public mirror" in r.stderr
+    assert "source used:       " + mirror in r.stderr
+    assert '"0.2.22"' in (app / "codai" / "__init__.py").read_text()
+
+
+def test_a_repo_named_by_the_operator_is_never_replaced(tmp_path):
+    app = _app(tmp_path / "app", "0.2.21")
+    mirror = _repo(tmp_path / "mirror", "0.2.22")
+    r = _run(tmp_path, app, default_repo=_repo(tmp_path / "d", "0.2.22"),
+             mirror=mirror, explicit="https://127.0.0.1:1/mine.git")
+    assert r.returncode != 0
+    assert "falling back" not in r.stderr
+    assert "clone of https://127.0.0.1:1/mine.git failed" in r.stderr
+    assert '"0.2.21"' in (app / "codai" / "__init__.py").read_text()   # untouched
+
+
+def test_a_missing_ref_is_an_error_not_a_reason_to_try_the_mirror(tmp_path):
+    app = _app(tmp_path / "app", "0.2.21")
+    good = _repo(tmp_path / "default", "0.2.22")
+    mirror = _repo(tmp_path / "mirror", "0.2.22")
+    r = _run(tmp_path, app, default_repo=good, mirror=mirror, ref="no-such-branch")
+    assert r.returncode != 0
+    assert "falling back" not in r.stderr
+    assert "no-such-branch" in r.stderr
+    assert '"0.2.21"' in (app / "codai" / "__init__.py").read_text()
+
+
+def test_an_up_to_date_install_exits_10_and_changes_nothing(tmp_path):
+    app = _app(tmp_path / "app", "0.2.22")
+    good = _repo(tmp_path / "default", "0.2.22")
+    r = _run(tmp_path, app, default_repo=good, mirror=_repo(tmp_path / "m", "0.2.22"))
+    assert r.returncode == 10 and "already up to date" in r.stderr