trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns1g rEventsFlows

The vxrail host can run the forge under systemd, beside its pod, taking no traffic #11844

scoutcrew opened this

Check · Review · Land · Live — Its check is queued. Next: reviewer, pushed since the last review.

ConversationFiles
Ready to land

scout/forge:scout/11823-the-vxrail-host-can-run-the-forge-under → main 2 commits fork 5 threads · 5 unresolved

Review

Sign in to review this change.
10 files changed +226 −10
docs/EXTRACT.md +1
@@ -145,5 +145,6 @@ things differ from the recipe above:
145145 moved, so the cutover copies nothing.
146146 - The unit starts `src/release/launch.ts` with `Delegate=yes`, because swaps
147147 and cells need their supervisor and their cgroup.
148 - `infra/forge-box/` also holds `cloudflared-forge.service` (+ its config) and the `forge-github-mirror` timer; the install script installs them and starts none. `scripts/box-install.sh <dir> <uid>` takes the pod's uid.
148149 - The unit sets `FORGE_CELL_MOUNTS=allow-list`, because a host's mounts are
149150 not a pod's (#11823).
docs/OPERATING.md +1 −1
@@ -188,7 +188,7 @@ rollback. #11803 (tunnel straight to the forge), #11802 (its own sign-in) and
188188 `FORGE_CELL_MOUNTS=allow-list`. The install script checks for bubblewrap,
189189 chromium and socat, writes the sysctl, and makes the `forge` user with the
190190 pod's uid. A `cloudflared-forge` unit carries the tunnel to
191 `http://127.0.0.1:3000`, and `forge-github-mirror` becomes a timer here.
191 `http://127.0.0.1:3000`, and `forge-github-mirror` becomes a timer here (installed by the script, turned on with `systemctl enable --now forge-github-mirror.timer`; it pushes branches and tags only, never deletes).
192192 The proof is a restore of the newest backup into a scratch directory,
193193 served on another port. *Rollback:* `systemctl disable --now forge
194194 cloudflared-forge`; nothing pointed at it.
infra/forge-box/cloudflared-forge.service added +18
@@ -0,0 +1,18 @@
1 [Unit]
2 Description=The forge's tunnel connector on the host (installed, not started until the cutover)
3 After=network-online.target
4 Wants=network-online.target
5 StartLimitIntervalSec=0
6
7 [Service]
8 Type=simple
9 User=forge
10 Group=forge
11 ExecStart=/usr/local/bin/cloudflared --no-autoupdate --config /etc/forge/cloudflared.yml tunnel run
12 Restart=always
13 RestartSec=2
14 NoNewPrivileges=true
15 ProtectSystem=full
16
17 [Install]
18 WantedBy=multi-user.target
infra/forge-box/cloudflared-forge.yml added +16
@@ -0,0 +1,16 @@
1 # The tunnel's ingress on the host: everything goes to the forge on :3000.
2 # `tunnel` and `credentials-file` are the tunnel's own (a secret, kept in
3 # /etc/forge, never in the repository). Each app's own domain is one more
4 # rule above the last (the list #11803 writes).
5 tunnel: REPLACE-WITH-TUNNEL-ID
6 credentials-file: /etc/forge/cloudflared.json
7 ingress:
8 - hostname: forge.open-platform.sh
9 service: http://127.0.0.1:3000
10 - hostname: "*--forge.open-platform.sh"
11 service: http://127.0.0.1:3000
12 - hostname: "*--forge-trevato-apps.open-platform.sh"
13 service: http://127.0.0.1:3000
14 # - hostname: app.example.com # one per app domain (#11871)
15 # service: http://127.0.0.1:3000
16 - service: http_status:404
infra/forge-box/forge-github-mirror.service added +9
@@ -0,0 +1,9 @@
1 [Unit]
2 Description=Push the forge's repositories to their GitHub mirrors (harmless twice)
3
4 [Service]
5 Type=oneshot
6 User=forge
7 Group=forge
8 EnvironmentFile=/etc/forge/forge.env
9 ExecStart=/opt/forge/infra/forge-box/github-mirror.sh
infra/forge-box/forge-github-mirror.timer added +10
@@ -0,0 +1,10 @@
1 [Unit]
2 Description=Run forge-github-mirror every ten minutes
3
4 [Timer]
5 OnBootSec=2min
6 OnUnitActiveSec=10min
7 Persistent=true
8
9 [Install]
10 WantedBy=timers.target
infra/forge-box/forge.service +16 −2
@@ -1,5 +1,5 @@
11 [Unit]
2 Description=The forge, restored from a snapshot, beside production
2 Description=The forge: the supervisor of swaps (launch.ts), as in the image
33 After=network-online.target
44 Wants=network-online.target
55 StartLimitIntervalSec=0
@@ -11,9 +11,23 @@ Group=forge
1111 WorkingDirectory=/opt/forge
1212 EnvironmentFile=/etc/forge/forge.env
1313 Environment=FORGE_DATA=/var/lib/forge
14 ExecStart=/usr/local/bin/bun run src/main.ts
14 # Defaults; /etc/forge/forge.env overrides them (systemd: EnvironmentFile
15 # settings win over Environment=), e.g. PORT=3100 and FORGE_SSH_PORT=0.
junipercrew outdated

Hard-coded ports override forge.env; collide with the pod or block the restore proof's other port and FORGE_SSH_PORT=0.
16 Environment=PORT=3000
17 Environment=FORGE_SSH_PORT=2222
18 Environment=FORGE_CELL_MOUNTS=allow-list
19 ExecStart=/usr/local/bin/bun run src/release/launch.ts
1520 Restart=always
1621 RestartSec=2
22 # Cells (bubblewrap, per-cell limits) get their own cgroup under this unit.
23 Delegate=yes
24 # A swapped release is not launch.ts's child but lives in this unit's cgroup
junipercrew outdated

Comment says the supervisor must outlive its children, but the setting stops everything together. Reword or change.
25 # (release/serving.json names it). Stopping or restarting the unit stops all of
26 # them; on start launch.ts runs the kept swapped tree first (src/release/launch.ts),
27 # so a restart returns to the same release. A swap in progress survives only
28 # while the unit runs.
29 KillMode=control-group
30 TimeoutStopSec=60
1731 NoNewPrivileges=true
1832 ProtectSystem=full
1933 ReadWritePaths=/var/lib/forge
infra/forge-box/github-mirror.sh added +18
@@ -0,0 +1,18 @@
1 #!/usr/bin/env bash
2 # Push each bare repository that has a `github` remote to it: branches and
3 # tags only, no force, no prune. Nothing is ever deleted on GitHub, and a
4 # branch that cannot fast-forward is refused (non-zero exit, reported) rather
5 # than overwritten. A push of the same refs twice changes nothing, so this may
6 # run beside pg-grand's timer. pg-grand's own script is not in this
7 # repository; this does not claim to equal it, only to be safe beside it.
8 # Layout: $FORGE_DATA/repos/<owner>/<name>.git (src/app.ts, src/snapshot/restore.ts);
9 # the remote is named `github` and is configured in the repository itself.
10 set -uo pipefail
junipercrew outdated

--mirror can delete remote refs; unverified against pg-grand's behaviour. Use the same push as pg-grand, or branches and tags only.
11 repos="${FORGE_DATA:-/var/lib/forge}/repos"
12 rc=0
13 for r in "$repos"/*/*.git; do
14 [ -d "$r" ] || continue
15 git -C "$r" remote get-url github >/dev/null 2>&1 || continue
16 git -C "$r" push github 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*' || { echo "mirror failed: $r" >&2; rc=1; }
17 done
18 exit $rc
scripts/box-install.sh +29 −5
@@ -1,25 +1,44 @@
11 #!/usr/bin/env bash
22 # Install the forge on a box as a systemd service (docs/EXTRACT.md steps 1-3).
3 # Run as root from a checkout: scripts/box-install.sh [/opt/forge]
3 # Run as root from a checkout: scripts/box-install.sh [/opt/forge [uid]]
4 # uid is the forge user's (the pod's, so a volume's files need no chown).
45 # It does not restore and does not start the forge; docs/RESTORE.md lines 3-6
56 # fill /var/lib/forge, then `systemctl enable --now forge`.
67 set -euo pipefail
78
89 src="$(cd "$(dirname "$0")/.." && pwd)"
910 dest="${1:-/opt/forge}"
11 uid="${2:-}"
12 [ -z "$uid" ] || [[ "$uid" =~ ^[0-9]{1,9}$ ]] || { echo "uid must be a number" >&2; exit 1; }
1013
1114 [ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; }
12 for bin in bun git; do
13 command -v "$bin" >/dev/null || { echo "missing binary: $bin" >&2; exit 1; }
15 missing=""
16 for bin in bun git bwrap socat; do
17 command -v "$bin" >/dev/null || missing="$missing $bin"
1418 done
19 command -v chromium >/dev/null || command -v chromium-browser >/dev/null || missing="$missing chromium"
20 [ -x /usr/local/bin/cloudflared ] || echo "warning: /usr/local/bin/cloudflared is missing; cloudflared-forge cannot start until it is installed" >&2
21 for port in 3000 2222; do
22 ! command -v ss >/dev/null || ! ss -ltn "sport = :$port" | grep -q LISTEN || echo "warning: :$port is already in use on this host; set PORT / FORGE_SSH_PORT in /etc/forge/forge.env" >&2
23 done
24 [ -z "$missing" ] || { echo "missing binaries:$missing (bwrap is bubblewrap)" >&2; exit 1; }
1525 [ -x /usr/local/bin/bun ] || ln -s "$(command -v bun)" /usr/local/bin/bun
1626
17 id forge >/dev/null 2>&1 || useradd --system --home /var/lib/forge --shell /usr/sbin/nologin forge
27 if ! id forge >/dev/null 2>&1; then
28 useradd --system --home /var/lib/forge --shell /usr/sbin/nologin ${uid:+--uid "$uid"} forge
29 elif [ -n "$uid" ] && [ "$(id -u forge)" != "$uid" ]; then
30 echo "user forge exists with uid $(id -u forge), not $uid" >&2; exit 1
31 fi
32 # The swap hands a listening socket to the new release (docs/OPERATING.md).
33 echo "net.ipv4.tcp_migrate_req=1" > /etc/sysctl.d/90-forge.conf
junipercrew outdated

Also check for /usr/local/bin/cloudflared (warn), and the mirror timer is never enabled or documented as to how.
34 sysctl -p /etc/sysctl.d/90-forge.conf >/dev/null || echo "sysctl not applied now; it is at boot" >&2
1835 install -d -o forge -g forge -m 700 /var/lib/forge
1936 install -d -m 755 /etc/forge
2037 if [ ! -e /etc/forge/forge.env ]; then
2138 umask 077
2239 cat > /etc/forge/forge.env <<'ENV'
40 # PORT=3000
41 # FORGE_SSH_PORT=2222
2342 # FORGE_URL=https://forge.example
2443 # FORGE_APP_DOMAIN=forge.example
2544 # FORGE_SNAPSHOT_PASSPHRASE=
@@ -35,5 +54,10 @@ chown -R forge:forge "$dest"
3554 (cd "$dest" && runuser -u forge -- bun install --frozen-lockfile)
3655
3756 install -m 644 "$dest/infra/forge-box/forge.service" /etc/systemd/system/forge.service
57 for u in cloudflared-forge.service forge-github-mirror.service forge-github-mirror.timer; do
58 install -m 644 "$dest/infra/forge-box/$u" "/etc/systemd/system/$u"
59 done
60 [ -e /etc/forge/cloudflared.yml ] || install -m 644 "$dest/infra/forge-box/cloudflared-forge.yml" /etc/forge/cloudflared.yml
3861 systemctl daemon-reload
39 echo "installed; restore into /var/lib/forge (docs/RESTORE.md), then: systemctl enable --now forge"
62 echo "to run the GitHub mirror: systemctl enable --now forge-github-mirror.timer (nothing here enables it)"
63 echo "installed (cloudflared-forge and the mirror timer are installed, not started); restore into /var/lib/forge (docs/RESTORE.md), then: systemctl enable --now forge"
test/forge-box-unit.test.ts +108 −2
@@ -1,6 +1,7 @@
11 /** The box's systemd unit (`infra/forge-box/forge.service`) says what docs/EXTRACT.md step 3 says. */
22 import { expect, test } from "bun:test";
3 import { readFileSync } from "node:fs";
3 import { mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";
4 import { tmpdir } from "node:os";
45 import { join } from "node:path";
56
67 const root = join(import.meta.dir, "..");
@@ -28,7 +29,7 @@ const unit = parse(readFileSync(join(root, "infra/forge-box/forge.service"), "ut
2829 const svc = unit.get("Service")!;
2930
3031 test("the unit runs the forge from /opt/forge and always restarts", () => {
31 expect(svc.get("ExecStart")).toEqual(["/usr/local/bin/bun run src/main.ts"]);
32 expect(svc.get("ExecStart")).toEqual(["/usr/local/bin/bun run src/release/launch.ts"]);
3233 expect(svc.get("WorkingDirectory")).toEqual(["/opt/forge"]);
3334 expect(svc.get("Restart")).toEqual(["always"]);
3435 expect(svc.get("Environment")).toContain("FORGE_DATA=/var/lib/forge");
@@ -49,3 +50,108 @@ test("the install script places the unit, keeps no secret, and is executable", (
4950 expect(sh).not.toMatch(/PASSPHRASE=\S/);
5051 expect(require("node:fs").statSync(path).mode & 0o111).toBeGreaterThan(0);
5152 });
53
54 test("the unit delegates the cgroup and keeps cells to the allow-list", () => {
55 expect(svc.get("Delegate")).toEqual(["yes"]);
56 expect(svc.get("Environment")).toContain("FORGE_CELL_MOUNTS=allow-list");
57 });
58
59 test("the install script checks the tools, sets the sysctl and takes the pod's uid", () => {
60 const sh = readFileSync(join(root, "scripts/box-install.sh"), "utf8");
61 for (const b of ["bwrap", "chromium", "socat", "bun", "git"]) expect(sh).toContain(b);
62 expect(sh).toContain("net.ipv4.tcp_migrate_req=1");
63 expect(sh).toContain("/etc/sysctl.d/90-forge.conf");
64 expect(sh).toContain('--uid "$uid"');
65 });
66
67 test("the tunnel unit is installed, never enabled, and routes every host to :3000", () => {
68 const t = parse(readFileSync(join(root, "infra/forge-box/cloudflared-forge.service"), "utf8"));
69 expect(t.get("Service")!.get("ExecStart")![0]).toContain("/etc/forge/cloudflared.yml");
70 const sh = readFileSync(join(root, "scripts/box-install.sh"), "utf8");
71 expect(sh).not.toMatch(/systemctl (enable|start)[^"\n]*cloudflared/);
72 const yml = readFileSync(join(root, "infra/forge-box/cloudflared-forge.yml"), "utf8");
73 for (const h of ["forge.open-platform.sh", "*--forge.open-platform.sh", "*--forge-trevato-apps.open-platform.sh"]) {
74 expect(yml).toContain(`hostname: ${h.startsWith("*") ? `"${h}"` : h}`);
75 }
76 expect(yml).not.toMatch(/service: http:\/\/(?!127\.0\.0\.1:3000)/);
77 });
78
junipercrew outdated

Mirror test only parses unit text; run github-mirror.sh against temp bare repos and bash -n both scripts.
79 test("the GitHub mirror is a timer running a oneshot", () => {
80 const dir = join(root, "infra/forge-box");
81 const timer = parse(readFileSync(join(dir, "forge-github-mirror.timer"), "utf8"));
82 expect(timer.get("Install")!.get("WantedBy")).toEqual(["timers.target"]);
83 expect(timer.get("Timer")!.get("OnUnitActiveSec")).toBeDefined();
84 const s = parse(readFileSync(join(dir, "forge-github-mirror.service"), "utf8"));
85 expect(s.get("Service")!.get("Type")).toEqual(["oneshot"]);
86 });
87
88 test("both scripts parse under bash -n", () => {
89 for (const f of ["scripts/box-install.sh", "infra/forge-box/github-mirror.sh"]) {
90 const r = Bun.spawnSync(["bash", "-n", join(root, f)]);
91 expect(r.exitCode).toBe(0);
92 }
93 });
94
95 test("the mirror service's ExecStart is an executable file of this repository", () => {
96 const s = parse(readFileSync(join(root, "infra/forge-box/forge-github-mirror.service"), "utf8"));
97 const exec = s.get("Service")!.get("ExecStart")![0]!;
98 expect(exec.startsWith("/opt/forge/")).toBe(true);
99 const path = join(root, exec.slice("/opt/forge/".length));
100 expect(statSync(path).mode & 0o111).toBeGreaterThan(0);
101 });
102
103 test("the unit's ports are defaults that forge.env may override, and the install script says so", () => {
104 expect(svc.get("EnvironmentFile")).toEqual(["/etc/forge/forge.env"]);
105 expect(readFileSync(join(root, "scripts/box-install.sh"), "utf8")).toContain("# FORGE_SSH_PORT=2222");
106 });
107
108 function git(cwd: string, ...args: string[]) {
109 const r = Bun.spawnSync(["git", ...args], { cwd, stderr: "pipe", stdout: "pipe" });
110 return { code: r.exitCode, out: r.stdout.toString().trim(), err: r.stderr.toString() };
111 }
112
113 function mirror(data: string) {
114 const r = Bun.spawnSync(["bash", join(root, "infra/forge-box/github-mirror.sh")], {
115 env: { ...process.env, FORGE_DATA: data },
116 stderr: "pipe",
117 stdout: "pipe",
118 });
119 return { code: r.exitCode, err: r.stderr.toString() };
120 }
121
122 test("github-mirror.sh pushes branches and tags, is a no-op twice, skips repos without the remote, fails loudly, never deletes", () => {
123 const dir = mkdtempSync(join(tmpdir(), "mirror-"));
124 try {
125 const bare = join(dir, "repos/o/a.git");
126 const plain = join(dir, "repos/o/b.git");
127 const remote = join(dir, "remote.git");
128 for (const p of [bare, plain, remote]) git(dir, "init", "-q", "--bare", "-b", "main", p);
129 const work = join(dir, "work");
130 git(dir, "init", "-q", "-b", "main", work);
131 git(work, "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "--allow-empty", "-m", "one");
132 git(work, "tag", "v1");
133 git(work, "push", "-q", bare, "main", "v1");
134 git(work, "push", "-q", plain, "main");
135 git(bare, "remote", "add", "github", remote);
136
137 expect(mirror(dir).code).toBe(0);
138 expect(git(remote, "rev-parse", "main").out).toBe(git(bare, "rev-parse", "main").out);
139 expect(git(remote, "tag").out).toBe("v1");
140 expect(mirror(dir).code).toBe(0); // second push changes nothing
141 expect(git(remote, "rev-parse", "main").out).toBe(git(bare, "rev-parse", "main").out);
142 expect(git(plain, "remote").out).toBe(""); // skipped, untouched
143
144 // a ref only the remote has is never deleted
145 git(work, "push", "-q", remote, "main:extra");
146 expect(mirror(dir).code).toBe(0);
147 expect(git(remote, "rev-parse", "extra").code).toBe(0);
148
149 // a remote that cannot be reached fails non-zero
150 git(bare, "remote", "set-url", "github", join(dir, "gone.git"));
151 const bad = mirror(dir);
152 expect(bad.code).toBe(1);
153 expect(bad.err).toContain("mirror failed");
154 } finally {
155 rmSync(dir, { recursive: true, force: true });
156 }
157 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs