trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns18g rEventsFlows

The roll recipe computes whether a surge pod can schedule, and prints the headroom it lacks #1640

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/1588-the-roll-recipe-computes-whether-a-surge → main 1 commit fork

  • Merged. torque/1588-the-roll-recipe-computes-whether-a-surge was merged into main as 629076b.

Review

Sign in to review this change.
5 files changed +721 −0
docs/ops/ROLL.md added +78
@@ -0,0 +1,78 @@
1 # The roll recipe
2
3 How the forge's own release rolls onto the platform, and the step that comes
4 before it every time (#1588, part of #873).
5
6 The chart rolls with `maxSurge: 1, maxUnavailable: 0`
7 (`charts/app/templates/deployment.yaml`). The new pod comes up beside the old
8 one, and the old one drains only once the new one is ready. The volume is
9 local-path, which binds it to one node, so the new pod has to fit on **that
10 node**, next to the old one. If it does not, the roll cannot be zero-downtime.
11
12 ## 1. Before rolling: can a surge pod schedule?
13
14 The forge runs in namespace `$NS` as the Helm release `$REL`. The Deployment
15 and its `app` label carry that name.
16
17 ```sh
18 POD=$(kubectl -n "$NS" get pod -l app="$REL" -o jsonpath='{.items[0].metadata.name}')
19 NODE=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.spec.nodeName}')
20 kubectl get node "$NODE" -o json > node.json
21 kubectl get pods -A -o json --field-selector spec.nodeName="$NODE" > pods.json
22 # The forge's steady memory: the MEMORY column, read over a quiet minute
23 # (no crew run starting). Not a peak, and not a cold start.
24 kubectl -n "$NS" top pod "$POD" --no-headers
25
26 bun scripts/roll-preflight.ts --node node.json --pods pods.json \
27 --pod "$NS/$POD" --steady 3712Mi
28 ```
29
30 It prints its answer and exits:
31
32 | exit | means |
33 | ---- | -------------------------------------------------------------------- |
34 | 0 | a surge pod fits: the roll is zero-downtime, go to step 2 |
35 | 3 | it does not fit: the output's `short by` line is the headroom it lacks |
36 | 2 | an input is not what `kubectl` prints; nothing was judged |
37
38 `--json` prints the same answer as data.
39
40 What happens after an exit 3 is #1589's step. Until that lands, do not roll:
41 put the printed answer in front of the owner.
42
43 ### The rule
44
45 `scripts/roll.ts` applies it. It counts whole millicores and whole bytes, as
46 the scheduler does.
47
48 | quantity | is |
49 | -------- | --------------------------------------------------------------------------------------------------- |
50 | free | node `status.allocatable` − the requests of every pod on the node that has not Succeeded or Failed |
51 | need | cpu: the forge pod's request. Memory: the larger of its request and its steady memory |
52 | fits | free ≥ need, for cpu and for memory |
53 | short by | need − free, never below zero: the headroom that would make the roll zero-downtime |
54
55 A pod's request is counted the way the scheduler reckons it:
56
57 - containers and sidecars (init containers with `restartPolicy: Always`) are summed;
58 - each plain init container, together with the sidecars started before it, is a floor;
59 - the runtime class's overhead is added on top.
60
61 Steady memory is part of `need` because the request is only a floor. A pod
62 that requests 2 Gi but holds 3.6 Gi once warm schedules on 2 Gi free. Then
63 the two pods together hold more memory than the node has. The answer says
64 when this is the case: "the pod would schedule, but both pods' steady memory
65 would not".
66
67 ### Worked example
68
69 On an 8 GB node with 3900m cpu and 7680 Mi allocatable, 3100m cpu and
70 5760 Mi are requested, which leaves 800m and 1920 Mi free. The forge requests
71 1000m cpu and 2048 Mi, and its steady memory is 3712 Mi. So it needs 1000m
72 and 3712 Mi, and is short by 200m cpu and 1792 Mi. `test/roll.test.ts`
73 carries this case.
74
75 ## 2. Roll
76
77 Release as the release workflow does. Nothing in this recipe changes the
78 workflow or the chart.
scripts/gate.ts +1
@@ -261,6 +261,7 @@ export const GATE: readonly string[] = [
261261 "test/ssh-site.test.ts",
262262 "test/web-code.test.ts",
263263 "test/web-origins.test.ts",
264 "test/roll.test.ts",
264265 ];
265266
266267 export const EXCLUDED: readonly Exclusion[] = [
scripts/roll-preflight.ts added +99
@@ -0,0 +1,99 @@
1 /**
2 * The roll recipe's first step (`docs/ops/ROLL.md`, #1588): read what
3 * `kubectl` printed, answer with `scripts/roll.ts` whether a surge pod can
4 * schedule beside the running forge, print that answer, and exit:
5 *
6 * 0 it fits: the roll is zero-downtime
7 * 3 it does not fit: the answer names the headroom it lacks
8 * 2 the input is not what kubectl prints
9 *
10 * bun scripts/roll-preflight.ts --node node.json --pods pods.json \
11 * --pod forge/forge-7d9c-abcde --steady 3712Mi [--json]
12 *
13 * It runs nothing and reaches no cluster: the files are the operator's.
14 */
15
16 import { readFileSync } from "node:fs";
17 import { parseQuantity, report, surge } from "./roll.ts";
18
19 export interface Args {
20 readonly node: string;
21 readonly pods: string;
22 readonly pod: string;
23 readonly steady: string;
24 readonly json: boolean;
25 }
26
27 const VALUED = ["--node", "--pods", "--pod", "--steady"] as const;
28
29 export function parseArgs(argv: readonly string[]): Args {
30 const one: Record<string, string> = {};
31 let json = false;
32 for (let i = 0; i < argv.length; i++) {
33 const flag = argv[i] ?? "";
34 if (flag === "--json") {
35 json = true;
36 continue;
37 }
38 if (!(VALUED as readonly string[]).includes(flag))
39 throw new Error(`unknown argument ${JSON.stringify(flag).slice(0, 40)}`);
40 const value = argv[++i];
41 if (value === undefined) throw new Error(`${flag} wants a value`);
42 if (one[flag] !== undefined) throw new Error(`${flag} given twice`);
43 one[flag] = value;
44 }
45 const need = (f: (typeof VALUED)[number]): string => {
46 const v = one[f];
47 if (v === undefined) throw new Error(`${f} is required`);
48 return v;
49 };
50 return {
51 node: need("--node"),
52 pods: need("--pods"),
53 pod: need("--pod"),
54 steady: need("--steady"),
55 json,
56 };
57 }
58
59 function readJson(path: string): unknown {
60 let text: string;
61 try {
62 text = readFileSync(path, "utf8");
63 } catch {
64 throw new Error(`cannot read ${path}`);
65 }
66 try {
67 return JSON.parse(text);
68 } catch {
69 throw new Error(`${path} is not JSON — was it written with -o json?`);
70 }
71 }
72
73 /** The whole step as a function of the arguments: exit code and output. */
74 export function run(argv: readonly string[]): { code: number; out: string } {
75 try {
76 const args = parseArgs(argv);
77 const steady = parseQuantity(args.steady);
78 if (steady === null)
79 throw new Error(`--steady ${JSON.stringify(args.steady).slice(0, 40)} is not a quantity`);
80 const s = surge({
81 node: readJson(args.node),
82 pods: readJson(args.pods),
83 forgePod: args.pod,
84 steadyMemory: steady,
85 });
86 return {
87 code: s.fits ? 0 : 3,
88 out: args.json ? JSON.stringify(s, null, 2) : report(s),
89 };
90 } catch (e) {
91 return { code: 2, out: `roll-preflight: ${(e as Error).message}` };
92 }
93 }
94
95 if (import.meta.main) {
96 const { code, out } = run(process.argv.slice(2));
97 console.log(out);
98 process.exit(code);
99 }
scripts/roll.ts added +278
@@ -0,0 +1,278 @@
1 /**
2 * Whether a surge pod can schedule, and the headroom it lacks (#1588, part
3 * of #873).
4 *
5 * The chart rolls with `maxSurge: 1, maxUnavailable: 0`: the new forge pod
6 * comes up beside the old one and the old one drains only once the new one
7 * is ready. Local-path storage binds the volume to one node, so the new pod
8 * must fit on *that* node, beside the old one. When it does not, the roll
9 * cannot be zero-downtime.
10 *
11 * Plain data in, plain data out. `scripts/roll-preflight.ts` reads what
12 * `kubectl` printed and prints what this answers; nothing here talks to a
13 * cluster.
14 *
15 * The rule, in the scheduler's own terms (NodeResourcesFit):
16 *
17 * free = node.status.allocatable − Σ requests of the pods on the node
18 * need.cpu = the forge pod's cpu request
19 * need.mem = max(the forge pod's memory request, its steady memory)
20 * fits ⇔ free.cpu ≥ need.cpu ∧ free.mem ≥ need.mem
21 * short by = max(0, need − free), per resource
22 *
23 * Steady memory is in `need` because the request is a floor, not what the
24 * pod holds: a pod requesting 2 Gi that steadies at 3.6 Gi schedules on
25 * 2 Gi free, and then both pods together hold more than the node has.
26 */
27
28 /**
29 * Whole millicores and whole bytes, as the scheduler counts them (a
30 * quantity is rounded up to its unit, `Quantity.MilliValue`), so `4100m −
31 * 3100m` is exactly `1000m` and never a hair under it.
32 */
33 export interface Resources {
34 readonly milliCpu: number;
35 readonly memory: number;
36 }
37
38 const SCALE: Record<string, number> = {
39 Ki: 2 ** 10,
40 Mi: 2 ** 20,
41 Gi: 2 ** 30,
42 Ti: 2 ** 40,
43 Pi: 2 ** 50,
44 Ei: 2 ** 60,
45 n: 1e-9,
46 u: 1e-6,
47 m: 1e-3,
48 "": 1,
49 k: 1e3,
50 M: 1e6,
51 G: 1e9,
52 T: 1e12,
53 P: 1e15,
54 E: 1e18,
55 };
56
57 const QUANTITY =
58 /^([0-9]+(?:\.[0-9]+)?|\.[0-9]+)(?:([eE][+-]?[0-9]{1,2})|(Ki|Mi|Gi|Ti|Pi|Ei|n|u|m|k|M|G|T|P|E))?$/;
59
60 /**
61 * A Kubernetes resource quantity (`resource.Quantity`): `8130464Ki`, `2Gi`,
62 * `500m`, `1.5`, `12e6`. Answers the plain number (bytes, or cores), or
63 * null for anything that is not one — never a guess.
64 */
65 export function parseQuantity(text: unknown): number | null {
66 if (typeof text !== "string") return null;
67 const m = QUANTITY.exec(text.trim());
68 if (!m) return null;
69 const base = Number(m[1]);
70 if (m[2]) return base * 10 ** Number(m[2].slice(1));
71 const scale = SCALE[m[3] ?? ""];
72 return scale === undefined ? null : base * scale;
73 }
74
75 const ZERO: Resources = { milliCpu: 0, memory: 0 };
76
77 function add(a: Resources, b: Resources): Resources {
78 return { milliCpu: a.milliCpu + b.milliCpu, memory: a.memory + b.memory };
79 }
80
81 function max(a: Resources, b: Resources): Resources {
82 return {
83 milliCpu: Math.max(a.milliCpu, b.milliCpu),
84 memory: Math.max(a.memory, b.memory),
85 };
86 }
87
88 function obj(v: unknown): Record<string, unknown> {
89 return v && typeof v === "object" && !Array.isArray(v)
90 ? (v as Record<string, unknown>)
91 : {};
92 }
93
94 function list(v: unknown): unknown[] {
95 return Array.isArray(v) ? v : [];
96 }
97
98 /** `{cpu, memory}` from a requests/allocatable map; absent reads as zero. */
99 function resources(map: unknown, where: string): Resources {
100 const m = obj(map);
101 const read = (key: "cpu" | "memory"): number => {
102 if (m[key] === undefined) return 0;
103 const n = parseQuantity(m[key]);
104 if (n === null)
105 throw new Error(
106 `${where}: ${key} ${JSON.stringify(m[key]).slice(0, 40)} is not a quantity`,
107 );
108 return n;
109 };
110 // Round up to the unit, as the scheduler does; the epsilon keeps 0.1 × 1000
111 // from becoming 101.
112 return {
113 milliCpu: Math.ceil(read("cpu") * 1000 - 1e-6),
114 memory: Math.ceil(read("memory") - 1e-6),
115 };
116 }
117
118 /**
119 * What the scheduler reserves for a pod (Kubernetes ≥ 1.29, "Resource
120 * sharing within containers"): the containers and sidecars (init
121 * containers with `restartPolicy: Always`) summed; each plain init
122 * container, with the sidecars started before it, taken as a floor; plus
123 * the runtime class's overhead.
124 */
125 export function podRequest(pod: unknown): Resources {
126 const spec = obj(obj(pod)["spec"]);
127 const name = String(obj(obj(pod)["metadata"])["name"] ?? "?");
128 const req = (c: unknown) =>
129 resources(obj(obj(c)["resources"])["requests"], `pod ${name}`);
130 let sidecars = ZERO;
131 let initFloor = ZERO;
132 for (const c of list(spec["initContainers"])) {
133 if (obj(c)["restartPolicy"] === "Always") sidecars = add(sidecars, req(c));
134 else initFloor = max(initFloor, add(req(c), sidecars));
135 }
136 let sum = sidecars;
137 for (const c of list(spec["containers"])) sum = add(sum, req(c));
138 return add(max(sum, initFloor), resources(spec["overhead"], `pod ${name}`));
139 }
140
141 /** A pod the scheduler still counts against its node. */
142 function holdsNode(pod: unknown, node: string): boolean {
143 const phase = obj(obj(pod)["status"])["phase"];
144 return (
145 obj(obj(pod)["spec"])["nodeName"] === node &&
146 phase !== "Succeeded" &&
147 phase !== "Failed"
148 );
149 }
150
151 export interface SurgeInput {
152 /** `kubectl get node <node> -o json`. */
153 readonly node: unknown;
154 /** `kubectl get pods -A -o json` (pods on other nodes are ignored). */
155 readonly pods: unknown;
156 /** The running forge pod, `namespace/name`. */
157 readonly forgePod: string;
158 /** Its steady memory in bytes (`kubectl top pod`, over a quiet minute). */
159 readonly steadyMemory: number;
160 }
161
162 export interface Surge {
163 readonly node: string;
164 readonly allocatable: Resources;
165 readonly requested: Resources;
166 readonly podsOnNode: number;
167 readonly free: Resources;
168 readonly forgeRequest: Resources;
169 readonly steadyMemory: number;
170 readonly need: Resources;
171 /** Would the scheduler place the surge pod at all (requests only)? */
172 readonly schedulable: boolean;
173 /** Would it place it *and* would both pods' steady memory fit? */
174 readonly fits: boolean;
175 /** What more must be free on this node for the roll to be zero-downtime. */
176 readonly shortfall: Resources;
177 }
178
179 const POD =
180 /^([a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?)\/([a-z0-9](?:[-a-z0-9.]{0,251}[a-z0-9])?)$/;
181
182 /** Answer. Throws on input that is not what `kubectl` prints. */
183 export function surge(input: SurgeInput): Surge {
184 const pod = POD.exec(input.forgePod.trim());
185 if (!pod)
186 throw new Error(
187 `forge pod ${JSON.stringify(input.forgePod).slice(0, 80)} is not namespace/name`,
188 );
189 const [ns, podName] = [pod[1], pod[2]];
190 if (!Number.isFinite(input.steadyMemory) || input.steadyMemory <= 0)
191 throw new Error("steady memory must be a positive quantity");
192 const steadyMemory = Math.ceil(input.steadyMemory);
193
194 const nodeName = obj(obj(input.node)["metadata"])["name"];
195 if (typeof nodeName !== "string" || nodeName === "")
196 throw new Error("node: no metadata.name — is it `kubectl get node -o json`?");
197 const allocatable = resources(
198 obj(obj(input.node)["status"])["allocatable"],
199 `node ${nodeName}`,
200 );
201 if (allocatable.memory <= 0 || allocatable.milliCpu <= 0)
202 throw new Error(`node ${nodeName}: no status.allocatable cpu and memory`);
203
204 const items = list(obj(input.pods)["items"]);
205 const forge = items.find((p) => {
206 const m = obj(obj(p)["metadata"]);
207 return m["namespace"] === ns && m["name"] === podName;
208 });
209 if (!forge) throw new Error(`pods: no ${ns}/${podName} in the list`);
210 if (obj(obj(forge)["spec"])["nodeName"] !== nodeName)
211 throw new Error(
212 `${ns}/${podName} is not on node ${nodeName}: the volume's node is the one to judge`,
213 );
214
215 const onNode = items.filter((p) => holdsNode(p, nodeName));
216 const requested = onNode.reduce<Resources>((s, p) => add(s, podRequest(p)), ZERO);
217 const free = {
218 milliCpu: allocatable.milliCpu - requested.milliCpu,
219 memory: allocatable.memory - requested.memory,
220 };
221 const forgeRequest = podRequest(forge);
222 const need = {
223 milliCpu: forgeRequest.milliCpu,
224 memory: Math.max(forgeRequest.memory, steadyMemory),
225 };
226 return {
227 node: nodeName,
228 allocatable,
229 requested,
230 podsOnNode: onNode.length,
231 free,
232 forgeRequest,
233 steadyMemory,
234 need,
235 schedulable:
236 free.milliCpu >= forgeRequest.milliCpu && free.memory >= forgeRequest.memory,
237 fits: free.milliCpu >= need.milliCpu && free.memory >= need.memory,
238 shortfall: {
239 milliCpu: Math.max(0, need.milliCpu - free.milliCpu),
240 memory: Math.max(0, need.memory - free.memory),
241 },
242 };
243 }
244
245 const MiB = 2 ** 20;
246 /** Whole MiB, rounded up: a shortfall printed is never less than the real one. */
247 const mib = (bytes: number) =>
248 `${bytes < 0 ? "-" : ""}${Math.ceil(Math.abs(bytes) / MiB)} MiB`;
249 const cpu = (milli: number) => `${milli}m cpu`;
250
251 /** The answer as plain lines, for the roll recipe to print. */
252 export function report(s: Surge): string {
253 const row = (label: string, r: Resources, extra = "") =>
254 ` ${label.padEnd(18)}${mib(r.memory).padStart(10)} ${cpu(r.milliCpu).padStart(11)}${extra}`;
255 const head = s.fits
256 ? `surge on ${s.node}: fits — the roll is zero-downtime`
257 : s.schedulable
258 ? `surge on ${s.node}: does NOT fit — the pod would schedule, but both pods' steady memory would not`
259 : `surge on ${s.node}: does NOT fit — the surge pod cannot schedule`;
260 const lines = [
261 head,
262 row("node allocatable", s.allocatable),
263 row("requested now", s.requested, ` (${s.podsOnNode} pods)`),
264 row("free", s.free),
265 row(
266 "surge pod needs",
267 s.need,
268 ` (request ${mib(s.forgeRequest.memory)}, steady ${mib(s.steadyMemory)})`,
269 ),
270 ];
271 if (!s.fits)
272 lines.push(
273 row("short by", s.shortfall),
274 `zero-downtime needs ${mib(s.shortfall.memory)} and ${cpu(s.shortfall.milliCpu)} more free on ${s.node}` +
275 " (a larger node, or fewer requests on this one)",
276 );
277 return lines.join("\n");
278 }
test/roll.test.ts added +265
@@ -0,0 +1,265 @@
1 /**
2 * The roll recipe's surge check (#1588): can a surge pod schedule beside the
3 * running forge, and if not, how much headroom it lacks. Every number below
4 * is worked by hand in the comment beside it.
5 */
6
7 import { afterAll, describe, expect, test } from "bun:test";
8 import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { run } from "../scripts/roll-preflight.ts";
12 import { parseQuantity, podRequest, report, surge } from "../scripts/roll.ts";
13
14 const Mi = 2 ** 20;
15
16 const container = (cpu: string, memory: string) => ({
17 name: "c",
18 resources: { requests: { cpu, memory } },
19 });
20
21 function pod(
22 ns: string,
23 name: string,
24 spec: Record<string, unknown>,
25 phase = "Running",
26 nodeName = "droplet",
27 ) {
28 return {
29 metadata: { namespace: ns, name },
30 spec: { nodeName, ...spec },
31 status: { phase },
32 };
33 }
34
35 // Requests on node "droplet" (Running and Pending pods only):
36 // forge 1 cpu 2048 Mi
37 // two-box 1.5 cpu 3072 Mi (1000m/2Gi + 500m/1Gi)
38 // init-heavy 0.5 cpu 512 Mi (max(init 500m/512Mi, main 100m/256Mi))
39 // overhead 0.1 cpu 128 Mi (no container requests, runtime overhead)
40 // ─────────────────────────────────
41 // requested 3.1 cpu 5760 Mi
42 // Not counted: a Succeeded job (4 Gi), a Failed one (4 Gi) and a pod on
43 // another node (4 Gi).
44 const PODS = {
45 items: [
46 pod("forge", "forge-abc", { containers: [container("1", "2Gi")] }),
47 pod("db", "two-box", {
48 containers: [container("1000m", "2Gi"), container("500m", "1Gi")],
49 }),
50 pod("ops", "init-heavy", {
51 initContainers: [container("500m", "512Mi")],
52 containers: [container("100m", "256Mi")],
53 }),
54 pod("ops", "overhead", {
55 containers: [{ name: "c" }],
56 overhead: { cpu: "100m", memory: "128Mi" },
57 }),
58 pod("ops", "done-job", { containers: [container("1", "4Gi")] }, "Succeeded"),
59 pod("ops", "failed-job", { containers: [container("1", "4Gi")] }, "Failed"),
60 pod("ops", "elsewhere", { containers: [container("1", "4Gi")] }, "Running", "other"),
61 ],
62 };
63
64 const node = (cpu: string, memory: string) => ({
65 metadata: { name: "droplet" },
66 status: { allocatable: { cpu, memory } },
67 });
68
69 const STEADY = 3712 * Mi; // 3.625 GiB, what the forge holds once warm
70
71 const judge = (cpu: string, memory: string, steadyMemory = STEADY) =>
72 surge({ node: node(cpu, memory), pods: PODS, forgePod: "forge/forge-abc", steadyMemory });
73
74 describe("quantities", () => {
75 test("parse as kubectl prints them", () => {
76 expect(parseQuantity("8130464Ki")).toBe(8130464 * 1024);
77 expect(parseQuantity("2Gi")).toBe(2 * 2 ** 30);
78 expect(parseQuantity("1G")).toBe(1e9);
79 expect(parseQuantity("500m")).toBe(0.5);
80 expect(parseQuantity("1.5")).toBe(1.5);
81 expect(parseQuantity("12e6")).toBe(12e6);
82 expect(parseQuantity("250000n")).toBeCloseTo(0.00025, 12);
83 });
84 test("refuse what is not a quantity", () => {
85 for (const bad of ["", "Gi", "2 Gi", "-1", "1Zi", "1e", "0x10", "1Gi;rm"])
86 expect(parseQuantity(bad)).toBeNull();
87 expect(parseQuantity(5)).toBeNull();
88 });
89 });
90
91 describe("a pod's request, as the scheduler reckons it", () => {
92 test("init floor, sidecars summed, overhead added", () => {
93 // sidecar 100m/64Mi runs throughout; plain init 2 cpu/1Gi runs after it,
94 // so its floor is 2.1 cpu / 1088 Mi; containers + sidecar = 0.6 / 576 Mi.
95 // max → 2.1 cpu / 1088 Mi, + overhead 50m/32Mi → 2.15 cpu / 1120 Mi.
96 const r = podRequest(
97 pod("x", "y", {
98 initContainers: [
99 { ...container("100m", "64Mi"), restartPolicy: "Always" },
100 container("2", "1Gi"),
101 ],
102 containers: [container("500m", "512Mi")],
103 overhead: { cpu: "50m", memory: "32Mi" },
104 }),
105 );
106 expect(r).toEqual({ milliCpu: 2150, memory: 1120 * Mi });
107 });
108 test("tenths of a core are exact millicores", () => {
109 // 0.1 × 1000 in floating point is 100.00000000000001; the scheduler says 100.
110 expect(podRequest(pod("x", "y", { containers: [container("0.1", "1")] }))).toEqual({
111 milliCpu: 100,
112 memory: 1,
113 });
114 });
115 });
116
117 describe("does a surge pod schedule", () => {
118 test("the 8 GB droplet: it does not, and it says by how much", () => {
119 // allocatable 3.9 cpu / 7680 Mi − requested 3.1 / 5760 → free 0.8 / 1920 Mi
120 // need: cpu 1, memory max(2048, 3712) = 3712 Mi
121 // short: 0.2 cpu, 3712 − 1920 = 1792 Mi
122 const s = judge("3900m", "7680Mi");
123 expect(s.podsOnNode).toBe(4);
124 expect(s.requested).toEqual({ milliCpu: 3100, memory: 5760 * Mi });
125 expect(s.free).toEqual({ milliCpu: 800, memory: 1920 * Mi });
126 expect(s.forgeRequest).toEqual({ milliCpu: 1000, memory: 2048 * Mi });
127 expect(s.need).toEqual({ milliCpu: 1000, memory: 3712 * Mi });
128 expect(s.schedulable).toBe(false);
129 expect(s.fits).toBe(false);
130 expect(s.shortfall).toEqual({ milliCpu: 200, memory: 1792 * Mi });
131 const text = report(s);
132 expect(text).toContain("cannot schedule");
133 expect(text).toContain("1792 MiB and 200m cpu more free on droplet");
134 });
135
136 test("a 16 GB node: it fits, nothing is short", () => {
137 // allocatable 7.9 cpu / 12288 Mi − 3.1 / 5760 → free 4.8 / 6528 Mi ≥ 1 / 3712
138 const s = judge("7900m", "12Gi");
139 expect(s.free).toEqual({ milliCpu: 4800, memory: 6528 * Mi });
140 expect(s.schedulable).toBe(true);
141 expect(s.fits).toBe(true);
142 expect(s.shortfall).toEqual({ milliCpu: 0, memory: 0 });
143 const text = report(s);
144 expect(text).toContain("fits — the roll is zero-downtime");
145 expect(text).not.toContain("short by");
146 });
147
148 test("exactly enough is enough", () => {
149 // allocatable 4.1 cpu / 9472 Mi − 3.1 / 5760 → free 1 / 3712 Mi = need
150 const s = judge("4100m", "9472Mi");
151 expect(s.free).toEqual({ milliCpu: 1000, memory: 3712 * Mi });
152 expect(s.fits).toBe(true);
153 // one byte less is not
154 const t = judge("4100m", String(9472 * Mi - 1));
155 expect(t.fits).toBe(false);
156 expect(t.shortfall).toEqual({ milliCpu: 0, memory: 1 });
157 });
158
159 test("schedules on its request, but steady memory does not fit", () => {
160 // allocatable 4.1 cpu / 8000 Mi − 3.1 / 5760 → free 1 / 2240 Mi
161 // request 2048 Mi ≤ 2240: the scheduler places it; steady 3712 > 2240
162 // short: 0 cpu, 3712 − 2240 = 1472 Mi
163 const s = judge("4100m", "8000Mi");
164 expect(s.schedulable).toBe(true);
165 expect(s.fits).toBe(false);
166 expect(s.shortfall).toEqual({ milliCpu: 0, memory: 1472 * Mi });
167 expect(report(s)).toContain("would schedule, but both pods' steady memory would not");
168 });
169
170 test("steady memory below the request: the request is the need", () => {
171 // free 4.8 cpu / 6528 Mi; steady 1024 Mi < request 2048 Mi → need 2048 Mi
172 const s = judge("7900m", "12Gi", 1024 * Mi);
173 expect(s.need.memory).toBe(2048 * Mi);
174 expect(s.fits).toBe(true);
175 });
176
177 test("short on cpu alone", () => {
178 // allocatable 3.5 cpu / 12288 Mi − 3.1 / 5760 → free 0.4 / 6528 Mi
179 const s = judge("3500m", "12Gi");
180 expect(s.fits).toBe(false);
181 expect(s.shortfall).toEqual({ milliCpu: 600, memory: 0 });
182 });
183 });
184
185 describe("input that is not what kubectl prints", () => {
186 test("is refused, never guessed", () => {
187 const bad: Array<[Parameters<typeof surge>[0], RegExp]> = [
188 [{ node: node("4", "8Gi"), pods: PODS, forgePod: "forge-abc", steadyMemory: STEADY }, /namespace\/name/],
189 [{ node: node("4", "8Gi"), pods: PODS, forgePod: "forge/-x", steadyMemory: STEADY }, /namespace\/name/],
190 [{ node: node("4", "8Gi"), pods: PODS, forgePod: "forge/forge-abc", steadyMemory: 0 }, /steady/],
191 [{ node: node("4", "8Gi"), pods: PODS, forgePod: "forge/forge-abc", steadyMemory: NaN }, /steady/],
192 [{ node: {}, pods: PODS, forgePod: "forge/forge-abc", steadyMemory: STEADY }, /metadata\.name/],
193 [{ node: node("4", "lots"), pods: PODS, forgePod: "forge/forge-abc", steadyMemory: STEADY }, /not a quantity/],
194 [{ node: { metadata: { name: "droplet" } }, pods: PODS, forgePod: "forge/forge-abc", steadyMemory: STEADY }, /allocatable/],
195 [{ node: node("4", "8Gi"), pods: PODS, forgePod: "forge/missing", steadyMemory: STEADY }, /no forge\/missing/],
196 [{ node: node("4", "8Gi"), pods: PODS, forgePod: "ops/elsewhere", steadyMemory: STEADY }, /not on node/],
197 ];
198 for (const [input, why] of bad) expect(() => surge(input)).toThrow(why);
199 });
200 });
201
202 describe("the recipe's command", () => {
203 const dir = mkdtempSync(join(tmpdir(), "roll-"));
204 afterAll(() => rmSync(dir, { recursive: true, force: true }));
205 const files = (cpu: string, memory: string) => {
206 writeFileSync(join(dir, "node.json"), JSON.stringify(node(cpu, memory)));
207 writeFileSync(join(dir, "pods.json"), JSON.stringify(PODS));
208 return ["--node", join(dir, "node.json"), "--pods", join(dir, "pods.json"), "--pod", "forge/forge-abc"];
209 };
210
211 test("exit 3 and the headroom when it does not fit", () => {
212 const { code, out } = run([...files("3900m", "7680Mi"), "--steady", "3712Mi"]);
213 expect(code).toBe(3);
214 expect(out).toContain("short by");
215 expect(out).toContain("1792 MiB and 200m cpu more free");
216 });
217
218 test("exit 0 when it fits", () => {
219 const { code, out } = run([...files("7900m", "12Gi"), "--steady", "3712Mi"]);
220 expect(code).toBe(0);
221 expect(out).toContain("zero-downtime");
222 });
223
224 test("--json is the same answer as data", () => {
225 const { code, out } = run([...files("3900m", "7680Mi"), "--steady", "3712Mi", "--json"]);
226 expect(code).toBe(3);
227 const s = JSON.parse(out);
228 expect(s.fits).toBe(false);
229 expect(s.shortfall).toEqual({ milliCpu: 200, memory: 1792 * Mi });
230 });
231
232 test("exit 2 on arguments or files that are not right", () => {
233 const args = files("3900m", "7680Mi");
234 writeFileSync(join(dir, "text.json"), "NAME STATUS\n");
235 for (const argv of [
236 args, // no --steady
237 [...args, "--steady", "lots"],
238 [...args, "--steady", "1Gi", "--steady", "2Gi"],
239 [...args, "--steady", "1Gi", "--force"],
240 [...args, "--steady"],
241 ["--node", join(dir, "absent.json"), ...args.slice(2), "--steady", "1Gi"],
242 ["--node", join(dir, "text.json"), ...args.slice(2), "--steady", "1Gi"],
243 ]) {
244 const { code, out } = run(argv);
245 expect(code).toBe(2);
246 expect(out).toStartWith("roll-preflight: ");
247 }
248 });
249 });
250
251 describe("what this pull leaves alone", () => {
252 test("the chart still rolls with a surge pod", () => {
253 // The check above only means something while the chart surges.
254 const chart = readFileSync(
255 join(import.meta.dir, "../charts/app/templates/deployment.yaml"),
256 "utf8",
257 );
258 expect(chart).toMatch(/maxSurge: 1\n\s+maxUnavailable: 0/);
259 });
260 test("the roll recipe names the command and the rule", () => {
261 const doc = readFileSync(join(import.meta.dir, "../docs/ops/ROLL.md"), "utf8");
262 expect(doc).toContain("bun scripts/roll-preflight.ts");
263 expect(doc).toContain("short by");
264 });
265 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs