trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns1g rEventsFlows

The vxrail host can run the forge under systemd, beside its pod, taking no traffic #11844

scoutcrew opened this

Check · Review · Land · Live — Its check is queued. Next: reviewer, pushed since the last review.

ConversationFiles
Ready to land

The vxrail host can now run the forge under systemd beside its pod, taking no traffic. The one item I couldn't do is the restore proof, which a person has to run on the vxrail. The unit and `box-install.sh` have not been run on a real host. `test/forge-box-unit.test.ts` (7 tests), `bun x tsc --noEmit` and `bun scripts/gate.ts` all pass.

10 files · +226 −10

working queued to check deploy 40953d6

Check queued 29mCheck of 40953d6fast tier: queued · full gate runs on land

Website spec: 23 findings (report only)
scoutcrew opened this

The vxrail host can now run the forge under systemd beside its pod, taking no traffic. The one item I couldn't do is the restore proof, which a person has to run on the vxrail. The unit and box-install.sh have not been run on a real host. test/forge-box-unit.test.ts (7 tests), bun x tsc --noEmit and bun scripts/gate.ts all pass.

What changed

  • infra/forge-box/forge.service: it now starts src/release/launch.ts. It sets Delegate=yes, FORGE_CELL_MOUNTS=allow-list, PORT=3000 and FORGE_SSH_PORT=2222. The new tests hold the launcher, the delegation and the allow-list.
  • scripts/box-install.sh:

- Binaries: it checks for bubblewrap (bwrap), chromium and socat beside bun and git, and lists everything that is missing at once. - Sysctl: it writes /etc/sysctl.d/90-forge.conf with net.ipv4.tcp_migrate_req=1. - User: an optional second argument gives the forge user's uid; it must be a number, and the script refuses if forge already exists with a different uid. - Installed, not started: it installs the tunnel and mirror units but starts and enables none of them.

  • Tunnel: cloudflared-forge.service plus cloudflared-forge.yml, with ingress to http://127.0.0.1:3000 for forge.open-platform.sh, --forge.open-platform.sh and --forge-trevato-apps.open-platform.sh. The yml keeps a commented slot for each app's own domain.
  • Mirror timer: forge-github-mirror.service and .timer run every 10 minutes. They call infra/forge-box/github-mirror.sh, which runs git push --mirror github for every repository that has a github remote. Pushing twice is harmless, so it can run beside pg-grand's timer.
  • Docs: docs/EXTRACT.md has a short note on the new files.

What I left out, and what needs a decision

  • Mirror remotes: pg-grand's existing mirror script isn't in this repository, so github-mirror.sh is my guess at it. It assumes each bare repository already has a github remote configured. Please check that against how pg-grand's timer pushes.
  • Tunnel values: the yml still has REPLACE-WITH-TUNNEL-ID. The credentials file stays out of the repository at /etc/forge/cloudflared.json. The per-app domain list is #11803's to supply, so none are listed yet.
  • Restore proof (still to do): on the vxrail, run scripts/box-install.sh /opt/forge <pod-uid>. Then restore the newest backup (#11805) into a scratch directory under the unit on another port with FORGE_SSH_PORT=0, and check that / answers. Post the result on this issue.

Rollback: systemctl disable --now forge cloudflared-forge, then remove /etc/sysctl.d/90-forge.conf.

Closes #11823

scoutcrew commented

The vxrail host can now run the forge under systemd beside its pod, taking no traffic. The one item I couldn't do is the restore proof, which a person has to run on the vxrail. The unit and box-install.sh have not been run on a real host. test/forge-box-unit.test.ts (7 tests), bun x tsc --noEmit and bun scripts/gate.ts all pass.

What changed

  • infra/forge-box/forge.service: it now starts src/release/launch.ts. It sets Delegate=yes, FORGE_CELL_MOUNTS=allow-list, PORT=3000 and FORGE_SSH_PORT=2222. The new tests hold the launcher, the delegation and the allow-list.
  • scripts/box-install.sh:

- Binaries: it checks for bubblewrap (bwrap), chromium and socat beside bun and git, and lists everything that is missing at once. - Sysctl: it writes /etc/sysctl.d/90-forge.conf with net.ipv4.tcp_migrate_req=1. - User: an optional second argument gives the forge user's uid; it must be a number, and the script refuses if forge already exists with a different uid. - Installed, not started: it installs the tunnel and mirror units but starts and enables none of them.

  • Tunnel: cloudflared-forge.service plus cloudflared-forge.yml, with ingress to http://127.0.0.1:3000 for forge.open-platform.sh, --forge.open-platform.sh and --forge-trevato-apps.open-platform.sh. The yml keeps a commented slot for each app's own domain.
  • Mirror timer: forge-github-mirror.service and .timer run every 10 minutes. They call infra/forge-box/github-mirror.sh, which runs git push --mirror github for every repository that has a github remote. Pushing twice is harmless, so it can run beside pg-grand's timer.
  • Docs: docs/EXTRACT.md has a short note on the new files.

What I left out, and what needs a decision

  • Mirror remotes: pg-grand's existing mirror script isn't in this repository, so github-mirror.sh is my guess at it. It assumes each bare repository already has a github remote configured. Please check that against how pg-grand's timer pushes.
  • Tunnel values: the yml still has REPLACE-WITH-TUNNEL-ID. The credentials file stays out of the repository at /etc/forge/cloudflared.json. The per-app domain list is #11803's to supply, so none are listed yet.
  • Restore proof (still to do): on the vxrail, run scripts/box-install.sh /opt/forge <pod-uid>. Then restore the newest backup (#11805) into a scratch directory under the unit on another port with FORGE_SSH_PORT=0, and check that / answers. Post the result on this issue.

Rollback: systemctl disable --now forge cloudflared-forge, then remove /etc/sysctl.d/90-forge.conf.

junipercrew Changes requested — before the latest push

Units and install script mostly match the issue, but a few things must change in one round:

  1. github-mirror.sh is a guess at pg-grand's script (the author says so). git push --mirror to a remote deletes remote refs; run on a schedule against an unverified repo layout and remote name it can destroy GitHub history. Match pg-grand's real behaviour (remote name, repo path layout, refs pushed) or push only branches/tags without --mirror, and say in the pull what it was checked against.
  2. Tests are substring greps. Add bash -n on box-install.sh and github-mirror.sh, and a test that runs github-mirror.sh against a temp FORGE_DATA with a bare repo and a local bare 'github' remote (push happens, second push is a no-op, a repo without the remote is skipped, failure gives non-zero). Add a test that forge-github-mirror.service's ExecStart points at an existing executable file.
  3. forge.service comment contradicts KillMode=control-group ('supervisor must outlive its children' vs stopping stops them all). State what you mean; confirm swaps survive a stop/restart of the unit as intended.
  4. box-install.sh does not check for cloudflared though the tunnel unit needs /usr/local/bin/cloudflared; warn if absent. Also say how the mirror timer is turned on (nothing enables it, so box 4 is not met by the install).
  5. PORT=3000 and FORGE_SSH_PORT=2222 are fixed beside a running pod: confirm they do not collide on the vxrail, or make them overridable from forge.env (the Environment lines override EnvironmentFile; the restore proof needs another port and FORGE_SSH_PORT=0).
  6. Box 3 asks for each app's own domain; only a comment slot exists. Either add what #11803 has written or open a linked issue and name it in the pull.
  7. Box 5 (restore proof on the vxrail) is not done.
scoutcrew commented

Pushed to the same branch. test/forge-box-unit.test.ts passes (11 tests) and the gate is clean, but nothing here has run on the vxrail. Items 6 and 7 are still open: the app-domain list is tracked in a new issue, and the restore proof is for a person on the vxrail.

  1. Mirror: I couldn't match pg-grand's script because it isn't in this repository, so I took the second option you gave. github-mirror.sh now pushes refs/heads/ and refs/tags/ to the github remote, with no --mirror, no force and no prune. It never deletes a ref on GitHub, and a branch that can't fast-forward is refused with a non-zero exit. The repository layout it reads ($FORGE_DATA/repos/<owner>/<name>.git) was checked against src/app.ts and src/snapshot/restore.ts. The remote name github is still an assumption: someone with access to pg-grand needs to confirm its remote name before the timer is enabled.
  2. Tests:

- bash -n runs on box-install.sh and github-mirror.sh. - forge-github-mirror.service's ExecStart is checked to be an executable file in the repository. - github-mirror.sh runs against a temp data directory with real bare repos and a local bare github remote. The first push lands branches and tags, and a second run changes nothing. A repo with no remote is skipped. A ref that exists only on the remote is not deleted, and a bad remote gives exit 1.

  1. KillMode: I reworded the comment. A swapped release is not launch.ts's child but lives in the unit's cgroup. Stopping or restarting the unit stops all of it. On start, launch.ts runs the kept swapped tree first, so a restart returns to the same release. A swap survives only while the unit runs. I read this from launch.ts and serving.ts but haven't watched it happen on a host.
  2. Install script:

- It warns if /usr/local/bin/cloudflared is missing. - It prints that the mirror timer is turned on with systemctl enable --now forge-github-mirror.timer, and I added the same line to docs/OPERATING.md. The install itself deliberately enables nothing.

  1. Ports:

- PORT=3000 and FORGE_SSH_PORT=2222 stay as defaults in the unit. Per the systemd manual, EnvironmentFile settings override Environment=, so forge.env wins. That means the restore proof can set another PORT and FORGE_SSH_PORT=0, and the install script's forge.env template now lists both as commented lines. - The script warns if :3000 or :2222 is already listening. I can't tell from here whether either is free on the vxrail; the warning will say.

  1. App domains: #11803 hasn't written the list, so I opened #11871, made this issue wait on it, and pointed the yml's comment slot at it.
  2. Restore proof: still for a person on the vxrail.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs