Public: anyone can read this; the crew can change it.
Bring existing leads in from a spreadsheet: the agent pastes or uploads a CSV, checks a preview, and each lead lands on the desk without a message going out #104
Check · Review · Land · Live done — Landed and live in production.
mason/speed-to-lead:mason/102-bring-existing-leads-in-from-a-spreadshe → main 2 commits fork
18 files
changed +1147
−8
README.md +4
| @@ -607,6 +607,10 @@ Every lead and its status, as a spreadsheet, for whoever may list leads, with no | ||
| 607 | 607 | |
| 608 | 608 | Columns: name, email, phone, source, status, arrived at, first reply at, minutes to first reply, last message at (the later of a message sent and a reply from the lead), texts allowed, lost reason, starred, and who referred them (name, email, phone). Times are ISO 8601. A field starting with `=`, `@`, `-`, `+`, tab or carriage return is prefixed with `'` so a spreadsheet never runs it as a formula; the one exception is a phone number written like `+1 206-555-0101` (a plus, then digits and the usual marks), which is kept as typed. |
| 609 | 609 | |
| 610 | ## Bringing your leads in | |
| 611 | ||
| 612 | From the desk, "Import leads (CSV)" (`/leads/import`): paste a CSV or choose a `.csv` file (up to 1 MB and 2,000 rows) with a header row, then press "Check the spreadsheet". Columns are found by header, in any case: `name`, `source` (zillow, realtor_com, website, open_house, referral) and `email` and/or `phone` are needed; `message`, `state` (or `status`: new/open, qualified, lost) and `arrived at` are optional, and other columns (the download's own, for instance, so your export imports as it is) are ignored. The preview shows each row as it will land and lists the rows it cannot use (no name, neither email nor phone, bad email, unknown source) with the reason; a row whose email or phone matches a lead already on the desk, or an earlier row, is named and not added again. Nothing is written until you press Import, and imported leads are history, not inquiries: no first reply is drafted, no follow-ups are scheduled, no response clock runs, they are not counted in the response record or the week's leads in, and consent to text is never carried over (log a call or reply on one to start working it). **HTTP:** `POST /leads/import/preview` and `POST /leads/import` with `{ "csv": "…" }` (the same cell as adding a lead; anon gets 401, a cross-site post 403), each import written to the audit as `imported`, one row per lead. **Terminal:** `I` on the desk, paste the CSV, ctrl-d to check it, `y` to import. **MCP:** the `preview_import` and `import_leads` tools (`csv`). | |
| 613 | ||
| 610 | 614 | ## Running it |
| 611 | 615 | |
| 612 | 616 | sh |
contract/openapi.json +333
| @@ -973,6 +973,324 @@ | ||
| 973 | 973 | } |
| 974 | 974 | } |
| 975 | 975 | }, |
| 976 | "/leads/import/preview": { | |
| 977 | "post": { | |
| 978 | "summary": "Check a spreadsheet before bringing it in: each row as it would land, marked new, duplicate (its email or phone matches a lead already on the desk, or an earlier row) or refused with the reason (no name, neither email nor phone, bad email, unknown source). At most 2000 rows and 1 MB. Nothing is written. Needs the same role as adding a lead.", | |
| 979 | "operationId": "postLeadsImportPreview", | |
| 980 | "requestBody": { | |
| 981 | "required": true, | |
| 982 | "content": { | |
| 983 | "application/json": { | |
| 984 | "schema": { | |
| 985 | "type": "object", | |
| 986 | "properties": { | |
| 987 | "csv": { | |
| 988 | "type": "string", | |
| 989 | "maxLength": 1000000, | |
| 990 | "description": "RFC 4180 CSV with a header row; at most 2000 rows. Columns by header, any case: name, source, email, phone, message, state (new/open, qualified, lost), arrived at. The app's own export imports as it is." | |
| 991 | } | |
| 992 | }, | |
| 993 | "required": [ | |
| 994 | "csv" | |
| 995 | ] | |
| 996 | } | |
| 997 | } | |
| 998 | } | |
| 999 | }, | |
| 1000 | "responses": { | |
| 1001 | "200": { | |
| 1002 | "description": "Check a spreadsheet before bringing it in: each row as it would land, marked new, duplicate (its email or phone matches a lead already on the desk, or an earlier row) or refused with the reason (no name, neither email nor phone, bad email, unknown source). At most 2000 rows and 1 MB. Nothing is written. Needs the same role as adding a lead.", | |
| 1003 | "content": { | |
| 1004 | "application/json": { | |
| 1005 | "schema": { | |
| 1006 | "type": "object", | |
| 1007 | "properties": { | |
| 1008 | "rows": { | |
| 1009 | "type": "array", | |
| 1010 | "items": { | |
| 1011 | "type": "object", | |
| 1012 | "properties": { | |
| 1013 | "line": { | |
| 1014 | "type": "integer", | |
| 1015 | "description": "The row's number in the sheet; the header is row 1." | |
| 1016 | }, | |
| 1017 | "name": { | |
| 1018 | "type": "string" | |
| 1019 | }, | |
| 1020 | "email": { | |
| 1021 | "type": "string" | |
| 1022 | }, | |
| 1023 | "phone": { | |
| 1024 | "type": "string" | |
| 1025 | }, | |
| 1026 | "source": { | |
| 1027 | "type": "string" | |
| 1028 | }, | |
| 1029 | "state": { | |
| 1030 | "enum": [ | |
| 1031 | "new", | |
| 1032 | "qualified", | |
| 1033 | "lost" | |
| 1034 | ] | |
| 1035 | }, | |
| 1036 | "outcome": { | |
| 1037 | "enum": [ | |
| 1038 | "new", | |
| 1039 | "duplicate", | |
| 1040 | "refused" | |
| 1041 | ] | |
| 1042 | }, | |
| 1043 | "reason": { | |
| 1044 | "type": "string", | |
| 1045 | "description": "Why a row is refused, or who a duplicate matches." | |
| 1046 | }, | |
| 1047 | "match": { | |
| 1048 | "type": "object", | |
| 1049 | "properties": { | |
| 1050 | "id": { | |
| 1051 | "type": "integer" | |
| 1052 | }, | |
| 1053 | "name": { | |
| 1054 | "type": "string" | |
| 1055 | } | |
| 1056 | } | |
| 1057 | } | |
| 1058 | }, | |
| 1059 | "required": [ | |
| 1060 | "line", | |
| 1061 | "name", | |
| 1062 | "email", | |
| 1063 | "phone", | |
| 1064 | "source", | |
| 1065 | "state", | |
| 1066 | "outcome" | |
| 1067 | ] | |
| 1068 | } | |
| 1069 | }, | |
| 1070 | "counts": { | |
| 1071 | "type": "object", | |
| 1072 | "properties": { | |
| 1073 | "new": { | |
| 1074 | "type": "integer" | |
| 1075 | }, | |
| 1076 | "duplicate": { | |
| 1077 | "type": "integer" | |
| 1078 | }, | |
| 1079 | "refused": { | |
| 1080 | "type": "integer" | |
| 1081 | } | |
| 1082 | } | |
| 1083 | } | |
| 1084 | } | |
| 1085 | } | |
| 1086 | } | |
| 1087 | } | |
| 1088 | }, | |
| 1089 | "default": { | |
| 1090 | "description": "{ error, message }: why not", | |
| 1091 | "content": { | |
| 1092 | "application/json": { | |
| 1093 | "schema": { | |
| 1094 | "$ref": "#/components/schemas/Error" | |
| 1095 | } | |
| 1096 | } | |
| 1097 | } | |
| 1098 | } | |
| 1099 | } | |
| 1100 | } | |
| 1101 | }, | |
| 1102 | "/leads/import": { | |
| 1103 | "post": { | |
| 1104 | "summary": "Bring leads in from a spreadsheet: every new, usable row becomes a lead (state new, qualified or lost) with no first reply drafted, no follow-ups scheduled, no response clock, and no count in the response record or the week report. A row whose email or phone matches a lead already on the desk is not added again; rows that cannot be used are returned with their reason. Consent to text is never imported. Nothing is sent. The audit holds a row per lead. 422 when no row can be added.", | |
| 1105 | "operationId": "postLeadsImport", | |
| 1106 | "requestBody": { | |
| 1107 | "required": true, | |
| 1108 | "content": { | |
| 1109 | "application/json": { | |
| 1110 | "schema": { | |
| 1111 | "type": "object", | |
| 1112 | "properties": { | |
| 1113 | "csv": { | |
| 1114 | "type": "string", | |
| 1115 | "maxLength": 1000000, | |
| 1116 | "description": "RFC 4180 CSV with a header row; at most 2000 rows. Columns by header, any case: name, source, email, phone, message, state (new/open, qualified, lost), arrived at. The app's own export imports as it is." | |
| 1117 | } | |
| 1118 | }, | |
| 1119 | "required": [ | |
| 1120 | "csv" | |
| 1121 | ] | |
| 1122 | } | |
| 1123 | } | |
| 1124 | } | |
| 1125 | }, | |
| 1126 | "responses": { | |
| 1127 | "201": { | |
| 1128 | "description": "Bring leads in from a spreadsheet: every new, usable row becomes a lead (state new, qualified or lost) with no first reply drafted, no follow-ups scheduled, no response clock, and no count in the response record or the week report. A row whose email or phone matches a lead already on the desk is not added again; rows that cannot be used are returned with their reason. Consent to text is never imported. Nothing is sent. The audit holds a row per lead. 422 when no row can be added.", | |
| 1129 | "content": { | |
| 1130 | "application/json": { | |
| 1131 | "schema": { | |
| 1132 | "type": "object", | |
| 1133 | "properties": { | |
| 1134 | "id": { | |
| 1135 | "type": "integer", | |
| 1136 | "description": "The first lead added." | |
| 1137 | }, | |
| 1138 | "added": { | |
| 1139 | "type": "array", | |
| 1140 | "items": { | |
| 1141 | "type": "object", | |
| 1142 | "properties": { | |
| 1143 | "id": { | |
| 1144 | "type": "integer" | |
| 1145 | }, | |
| 1146 | "name": { | |
| 1147 | "type": "string" | |
| 1148 | } | |
| 1149 | } | |
| 1150 | } | |
| 1151 | }, | |
| 1152 | "duplicates": { | |
| 1153 | "type": "array", | |
| 1154 | "items": { | |
| 1155 | "type": "object", | |
| 1156 | "properties": { | |
| 1157 | "line": { | |
| 1158 | "type": "integer", | |
| 1159 | "description": "The row's number in the sheet; the header is row 1." | |
| 1160 | }, | |
| 1161 | "name": { | |
| 1162 | "type": "string" | |
| 1163 | }, | |
| 1164 | "email": { | |
| 1165 | "type": "string" | |
| 1166 | }, | |
| 1167 | "phone": { | |
| 1168 | "type": "string" | |
| 1169 | }, | |
| 1170 | "source": { | |
| 1171 | "type": "string" | |
| 1172 | }, | |
| 1173 | "state": { | |
| 1174 | "enum": [ | |
| 1175 | "new", | |
| 1176 | "qualified", | |
| 1177 | "lost" | |
| 1178 | ] | |
| 1179 | }, | |
| 1180 | "outcome": { | |
| 1181 | "enum": [ | |
| 1182 | "new", | |
| 1183 | "duplicate", | |
| 1184 | "refused" | |
| 1185 | ] | |
| 1186 | }, | |
| 1187 | "reason": { | |
| 1188 | "type": "string", | |
| 1189 | "description": "Why a row is refused, or who a duplicate matches." | |
| 1190 | }, | |
| 1191 | "match": { | |
| 1192 | "type": "object", | |
| 1193 | "properties": { | |
| 1194 | "id": { | |
| 1195 | "type": "integer" | |
| 1196 | }, | |
| 1197 | "name": { | |
| 1198 | "type": "string" | |
| 1199 | } | |
| 1200 | } | |
| 1201 | } | |
| 1202 | }, | |
| 1203 | "required": [ | |
| 1204 | "line", | |
| 1205 | "name", | |
| 1206 | "email", | |
| 1207 | "phone", | |
| 1208 | "source", | |
| 1209 | "state", | |
| 1210 | "outcome" | |
| 1211 | ] | |
| 1212 | } | |
| 1213 | }, | |
| 1214 | "refused": { | |
| 1215 | "type": "array", | |
| 1216 | "items": { | |
| 1217 | "type": "object", | |
| 1218 | "properties": { | |
| 1219 | "line": { | |
| 1220 | "type": "integer", | |
| 1221 | "description": "The row's number in the sheet; the header is row 1." | |
| 1222 | }, | |
| 1223 | "name": { | |
| 1224 | "type": "string" | |
| 1225 | }, | |
| 1226 | "email": { | |
| 1227 | "type": "string" | |
| 1228 | }, | |
| 1229 | "phone": { | |
| 1230 | "type": "string" | |
| 1231 | }, | |
| 1232 | "source": { | |
| 1233 | "type": "string" | |
| 1234 | }, | |
| 1235 | "state": { | |
| 1236 | "enum": [ | |
| 1237 | "new", | |
| 1238 | "qualified", | |
| 1239 | "lost" | |
| 1240 | ] | |
| 1241 | }, | |
| 1242 | "outcome": { | |
| 1243 | "enum": [ | |
| 1244 | "new", | |
| 1245 | "duplicate", | |
| 1246 | "refused" | |
| 1247 | ] | |
| 1248 | }, | |
| 1249 | "reason": { | |
| 1250 | "type": "string", | |
| 1251 | "description": "Why a row is refused, or who a duplicate matches." | |
| 1252 | }, | |
| 1253 | "match": { | |
| 1254 | "type": "object", | |
| 1255 | "properties": { | |
| 1256 | "id": { | |
| 1257 | "type": "integer" | |
| 1258 | }, | |
| 1259 | "name": { | |
| 1260 | "type": "string" | |
| 1261 | } | |
| 1262 | } | |
| 1263 | } | |
| 1264 | }, | |
| 1265 | "required": [ | |
| 1266 | "line", | |
| 1267 | "name", | |
| 1268 | "email", | |
| 1269 | "phone", | |
| 1270 | "source", | |
| 1271 | "state", | |
| 1272 | "outcome" | |
| 1273 | ] | |
| 1274 | } | |
| 1275 | } | |
| 1276 | } | |
| 1277 | } | |
| 1278 | } | |
| 1279 | } | |
| 1280 | }, | |
| 1281 | "default": { | |
| 1282 | "description": "{ error, message }: why not", | |
| 1283 | "content": { | |
| 1284 | "application/json": { | |
| 1285 | "schema": { | |
| 1286 | "$ref": "#/components/schemas/Error" | |
| 1287 | } | |
| 1288 | } | |
| 1289 | } | |
| 1290 | } | |
| 1291 | } | |
| 1292 | } | |
| 1293 | }, | |
| 976 | 1294 | "/leads/{id}": { |
| 977 | 1295 | "get": { |
| 978 | 1296 | "summary": "One lead, with every message: sent, drafted, and the follow-ups to come. Ask for application/json.", |
| @@ -1304,6 +1622,10 @@ | ||
| 1304 | 1622 | "type": "boolean", |
| 1305 | 1623 | "description": "The agent keeps this lead close: starred open leads lead the desk." |
| 1306 | 1624 | }, |
| 1625 | "imported": { | |
| 1626 | "type": "boolean", | |
| 1627 | "description": "Brought in from a spreadsheet: it has no drafted reply, no follow-ups and no response clock, and is not counted as a lead in." | |
| 1628 | }, | |
| 1307 | 1629 | "referred_by": { |
| 1308 | 1630 | "type": "string", |
| 1309 | 1631 | "maxLength": 120, |
| @@ -1546,6 +1868,7 @@ | ||
| 1546 | 1868 | "lostAt", |
| 1547 | 1869 | "back_on", |
| 1548 | 1870 | "starred", |
| 1871 | "imported", | |
| 1549 | 1872 | "referred_by", |
| 1550 | 1873 | "referrer_email", |
| 1551 | 1874 | "referrer_phone", |
| @@ -3138,6 +3461,10 @@ | ||
| 3138 | 3461 | "type": "boolean", |
| 3139 | 3462 | "description": "The agent keeps this lead close: starred open leads lead the desk." |
| 3140 | 3463 | }, |
| 3464 | "imported": { | |
| 3465 | "type": "boolean", | |
| 3466 | "description": "Brought in from a spreadsheet: it has no drafted reply, no follow-ups and no response clock, and is not counted as a lead in." | |
| 3467 | }, | |
| 3141 | 3468 | "referred_by": { |
| 3142 | 3469 | "type": "string", |
| 3143 | 3470 | "maxLength": 120, |
| @@ -3356,6 +3683,7 @@ | ||
| 3356 | 3683 | "lostAt", |
| 3357 | 3684 | "back_on", |
| 3358 | 3685 | "starred", |
| 3686 | "imported", | |
| 3359 | 3687 | "referred_by", |
| 3360 | 3688 | "referrer_email", |
| 3361 | 3689 | "referrer_phone", |
| @@ -3624,6 +3952,10 @@ | ||
| 3624 | 3952 | "type": "boolean", |
| 3625 | 3953 | "description": "The agent keeps this lead close: starred open leads lead the desk." |
| 3626 | 3954 | }, |
| 3955 | "imported": { | |
| 3956 | "type": "boolean", | |
| 3957 | "description": "Brought in from a spreadsheet: it has no drafted reply, no follow-ups and no response clock, and is not counted as a lead in." | |
| 3958 | }, | |
| 3627 | 3959 | "referred_by": { |
| 3628 | 3960 | "type": "string", |
| 3629 | 3961 | "maxLength": 120, |
| @@ -3743,6 +4075,7 @@ | ||
| 3743 | 4075 | "lostAt", |
| 3744 | 4076 | "back_on", |
| 3745 | 4077 | "starred", |
| 4078 | "imported", | |
| 3746 | 4079 | "referred_by", |
| 3747 | 4080 | "referrer_email", |
| 3748 | 4081 | "referrer_phone", |
src/actions.ts +130
| @@ -50,6 +50,7 @@ import { | ||
| 50 | 50 | MAX_LEAD_NOTE, |
| 51 | 51 | MAX_QUERY, |
| 52 | 52 | findMatch, |
| 53 | insertImported, | |
| 53 | 54 | insertLead, |
| 54 | 55 | writeInAgain, |
| 55 | 56 | lastMessages, |
| @@ -94,6 +95,7 @@ import { MAX_ADDRESS, timeOf } from "./showings.ts"; | ||
| 94 | 95 | import { weekReport, type WeekReport } from "./report.ts"; |
| 95 | 96 | import { draftPriceChange, readChange, type Outcome as PriceChanged } from "./price-change.ts"; |
| 96 | 97 | import { dateOf } from "./putaway.ts"; |
| 98 | import { MAX_CSV, readTable } from "./import.ts"; | |
| 97 | 99 | import { MAX_SIGNIN_LINES, MAX_SIGNIN_TEXT, readSignIns, type SignIn } from "./signins.ts"; |
| 98 | 100 | import type { Who } from "./who.ts"; |
| 99 | 101 | |
| @@ -527,6 +529,134 @@ export function addSignIns( | ||
| 527 | 529 | return ok({ id: (added[0] ?? again[0])!.id, address, added, again, skipped }); |
| 528 | 530 | } |
| 529 | 531 | |
| 532 | /** One spreadsheet row as it would land. */ | |
| 533 | export type ImportRow = { | |
| 534 | readonly line: number; | |
| 535 | readonly name: string; | |
| 536 | readonly email: string; | |
| 537 | readonly phone: string; | |
| 538 | readonly source: Source | ""; | |
| 539 | readonly message: string; | |
| 540 | readonly state: "new" | "qualified" | "lost"; | |
| 541 | /** `new` lands; `duplicate` is already on the desk (or earlier in the file); `refused` cannot be used. */ | |
| 542 | readonly outcome: "new" | "duplicate" | "refused"; | |
| 543 | /** Why it is refused, or who it matches. */ | |
| 544 | readonly reason?: string; | |
| 545 | /** The lead already on the desk that a duplicate matches. */ | |
| 546 | readonly match?: { readonly id: number; readonly name: string }; | |
| 547 | /** When the lead arrived, if the sheet said so and it is a past moment; otherwise the import's moment. */ | |
| 548 | readonly arrived: string; | |
| 549 | }; | |
| 550 | ||
| 551 | export type ImportPreview = { | |
| 552 | readonly rows: readonly ImportRow[]; | |
| 553 | readonly counts: { readonly new: number; readonly duplicate: number; readonly refused: number }; | |
| 554 | }; | |
| 555 | ||
| 556 | const IMPORT_MESSAGE = "Brought in from a spreadsheet."; | |
| 557 | const IMPORT_STATES: Readonly<Record<string, ImportRow["state"]>> = { | |
| 558 | "": "new", | |
| 559 | new: "new", | |
| 560 | open: "new", | |
| 561 | contacted: "new", | |
| 562 | qualified: "qualified", | |
| 563 | lost: "lost", | |
| 564 | }; | |
| 565 | const ISO_MOMENT = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d{1,3})?)?Z$/; | |
| 566 | ||
| 567 | /** The CSV that arrived, read row by row against the same rules a lead added by hand meets, and against the desk. */ | |
| 568 | function readImport(db: Database, input: Record<string, unknown>, now: string): Result<ImportPreview> { | |
| 569 | if (typeof input.csv !== "string" || input.csv.trim() === "") | |
| 570 | return fail("invalid", "Paste the spreadsheet as CSV, or choose a .csv file."); | |
| 571 | if (input.csv.length > MAX_CSV) | |
| 572 | return fail("invalid", `A CSV is at most ${MAX_CSV.toLocaleString("en-US")} characters; split it and import it in parts.`); | |
| 573 | const table = readTable(input.csv); | |
| 574 | if (!table.ok) return fail("invalid", table.message); | |
| 575 | const seenEmail = new Map<string, number>(); | |
| 576 | const seenPhone = new Map<string, number>(); | |
| 577 | const rows = table.rows.map((raw): ImportRow => { | |
| 578 | const email = raw.email.toLowerCase(); | |
| 579 | const source = sourceOf(raw.source); | |
| 580 | const state = IMPORT_STATES[raw.state.toLowerCase()]; | |
| 581 | const message = lines(raw.message === "" ? IMPORT_MESSAGE : raw.message); | |
| 582 | const arrived = ISO_MOMENT.test(raw.arrived) && Date.parse(raw.arrived) <= Date.parse(now) ? new Date(raw.arrived).toISOString() : now; | |
| 583 | const row = { line: raw.line, name: raw.name, email, phone: raw.phone, source: source ?? "", message, state: state ?? "new", arrived } as const; | |
| 584 | const refuse = (reason: string): ImportRow => ({ ...row, outcome: "refused", reason }); | |
| 585 | if ([raw.name, email, raw.phone].some((f) => hasControl(f, false)) || hasControl(message, true)) | |
| 586 | return refuse("It holds control characters; a lead's details are plain text."); | |
| 587 | const wrong = contactProblem({ name: raw.name, email, phone: raw.phone }, true); | |
| 588 | if (wrong !== undefined) return refuse(wrong); | |
| 589 | if (source === undefined) return refuse("The source is one of zillow, realtor_com, website, open_house or referral."); | |
| 590 | if (state === undefined) return refuse("The state is new, open, qualified or lost."); | |
| 591 | if (message.length > 2000) return refuse("What they asked is at most 2000 characters."); | |
| 592 | const digits = raw.phone.replace(/\D/g, ""); | |
| 593 | const known = findMatch(db, row); | |
| 594 | if (known !== null) | |
| 595 | return { ...row, outcome: "duplicate", reason: `Already on the desk as ${known.name}.`, match: { id: known.id, name: known.name } }; | |
| 596 | const earlier = (email === "" ? undefined : seenEmail.get(email)) ?? (digits === "" ? undefined : seenPhone.get(digits)); | |
| 597 | if (earlier !== undefined) return { ...row, outcome: "duplicate", reason: `Same email or phone as row ${earlier} of this file.` }; | |
| 598 | if (email !== "") seenEmail.set(email, raw.line); | |
| 599 | if (digits !== "") seenPhone.set(digits, raw.line); | |
| 600 | return { ...row, outcome: "new" }; | |
| 601 | }); | |
| 602 | const count = (o: ImportRow["outcome"]): number => rows.filter((r) => r.outcome === o).length; | |
| 603 | return ok({ rows, counts: { new: count("new"), duplicate: count("duplicate"), refused: count("refused") } }); | |
| 604 | } | |
| 605 | ||
| 606 | /** | |
| 607 | * A pasted or uploaded CSV shown as it would land, each row marked as new, | |
| 608 | * already on the desk, or refused with its reason. Writes nothing. Same cell | |
| 609 | * as adding a lead (lead create), so nothing is audited. | |
| 610 | */ | |
| 611 | export function previewImport(db: Database, who: Who, input: Record<string, unknown>, at?: Date): Result<ImportPreview> { | |
| 612 | if (permit(who, "lead", "create") !== "yes") | |
| 613 | return refuse(who, "Sign in to import leads.", "You may not add a lead."); | |
| 614 | return readImport(db, input, stamp(at)); | |
| 615 | } | |
| 616 | ||
| 617 | /** What an import did. */ | |
| 618 | export type Imported = { | |
| 619 | readonly id: number; | |
| 620 | readonly added: readonly { id: number; name: string }[]; | |
| 621 | readonly duplicates: readonly ImportRow[]; | |
| 622 | readonly refused: readonly ImportRow[]; | |
| 623 | }; | |
| 624 | ||
| 625 | /** | |
| 626 | * Every usable row of the CSV becomes a lead, as history: no first reply is | |
| 627 | * drafted, no follow-up is scheduled, the response clock never starts and the | |
| 628 | * lead is not counted as a lead in. A row already on the desk is left alone, | |
| 629 | * a row that cannot be used is returned with its reason. The file is read | |
| 630 | * again here, never trusted from the preview; the trail holds a row per lead. | |
| 631 | */ | |
| 632 | export function importLeads(db: Database, who: Who, input: Record<string, unknown>, at?: Date): Result<Imported> { | |
| 633 | const now = stamp(at); | |
| 634 | const read = | |
| 635 | permit(who, "lead", "create") !== "yes" | |
| 636 | ? refuse<ImportPreview>(who, "Sign in to import leads.", "You may not add a lead.") | |
| 637 | : readImport(db, input, now); | |
| 638 | const usable = read.ok ? read.value.rows.filter((r) => r.outcome === "new") : []; | |
| 639 | if (!read.ok || usable.length === 0) { | |
| 640 | const failed: Result<Imported> = read.ok ? fail("invalid", "No row could be imported: none is new and usable. Check the preview.") : read; | |
| 641 | audited(db, who, "lead", "create", undefined, failed as Result<{ id: number }>); | |
| 642 | return failed; | |
| 643 | } | |
| 644 | const added = db.transaction(() => | |
| 645 | usable.map((r) => { | |
| 646 | const id = insertImported(db, { ...r, source: r.source as Source }, r.arrived, now); | |
| 647 | audited(db, who, "lead", "create", undefined, ok({ id }), "imported"); | |
| 648 | return { id, name: r.name }; | |
| 649 | }), | |
| 650 | )(); | |
| 651 | emit("lead.added", { id: added[0]!.id }); | |
| 652 | return ok({ | |
| 653 | id: added[0]!.id, | |
| 654 | added, | |
| 655 | duplicates: read.value.rows.filter((r) => r.outcome === "duplicate"), | |
| 656 | refused: read.value.rows.filter((r) => r.outcome === "refused"), | |
| 657 | }); | |
| 658 | } | |
| 659 | ||
| 530 | 660 | /** The fields of a lead's details, in the order they are told. */ |
| 531 | 661 | const CONTACT_FIELDS = ["name", "email", "phone"] as const; |
| 532 | 662 | export type ContactField = (typeof CONTACT_FIELDS)[number]; |
src/audit.ts +2 −2
| @@ -34,8 +34,8 @@ export interface AuditRow { | ||
| 34 | 34 | readonly detail: string | null; |
| 35 | 35 | } |
| 36 | 36 | |
| 37 | /** Messages sent are `reply` and `first_reply`; a close is `qualified` or `lost`; `reopened` puts it back; `opted_out` is a lead who asked to stop texts; `inbound` is what the lead wrote back, pasted in; `call` is a phone call the agent logged; `showing` books or moves a showing, `showing_off` cancels it; `edited` is a correction of the lead's name, email or phone; `held` is a text approved outside quiet hours, to go at the next 8:00 a.m.; `unhold` takes back a text held for quiet hours; `noted` is a change to the agent's private note on the lead; `merged` is a new arrival who was already a lead (the detail names the arrival's source and the lead): their message joined that lead; `openers` is a change to how first replies open (the profile); `cadence` is a change to the follow-up days (the profile); `skipped` sets one upcoming follow-up aside by hand, `unskipped` brings it back; `put_away` parks a lead until a date (the detail holds `{"on":"YYYY-MM-DD"}`), `brought_back` clears that date; `starred` keeps a lead close, `unstarred` lets it go; `unsent` takes back a send within its minute; `referred` is a lead added with the person who sent them (the detail holds `{"referrer":"name"}`); `thanked` approves the thank-you to a referrer, `thanks_held` approves it as a text held for quiet hours, `thanks_skipped` sets it aside; `price_change` is a listing's new price drafting a note to each lead who asked about it (the detail holds `{"address","price","drafted"}`). */ | |
| 38 | export const ACTS = ["reply", "first_reply", "qualified", "lost", "reopened", "opted_out", "inbound", "call", "showing", "showing_off", "edited", "held", "unhold", "openers", "cadence", "replies", "noted", "merged", "skipped", "unskipped", "put_away", "brought_back", "starred", "unstarred", "unsent", "referred", "thanked", "thanks_held", "thanks_skipped", "price_change"] as const; | |
| 37 | /** Messages sent are `reply` and `first_reply`; a close is `qualified` or `lost`; `reopened` puts it back; `opted_out` is a lead who asked to stop texts; `inbound` is what the lead wrote back, pasted in; `call` is a phone call the agent logged; `showing` books or moves a showing, `showing_off` cancels it; `edited` is a correction of the lead's name, email or phone; `held` is a text approved outside quiet hours, to go at the next 8:00 a.m.; `unhold` takes back a text held for quiet hours; `noted` is a change to the agent's private note on the lead; `merged` is a new arrival who was already a lead (the detail names the arrival's source and the lead): their message joined that lead; `openers` is a change to how first replies open (the profile); `cadence` is a change to the follow-up days (the profile); `skipped` sets one upcoming follow-up aside by hand, `unskipped` brings it back; `put_away` parks a lead until a date (the detail holds `{"on":"YYYY-MM-DD"}`), `brought_back` clears that date; `starred` keeps a lead close, `unstarred` lets it go; `unsent` takes back a send within its minute; `referred` is a lead added with the person who sent them (the detail holds `{"referrer":"name"}`); `thanked` approves the thank-you to a referrer, `thanks_held` approves it as a text held for quiet hours, `thanks_skipped` sets it aside; `price_change` is a listing's new price drafting a note to each lead who asked about it (the detail holds `{"address","price","drafted"}`); `imported` is a lead brought in from a spreadsheet (no message drafted, no clock). */ | |
| 38 | export const ACTS = ["reply", "first_reply", "qualified", "lost", "reopened", "opted_out", "inbound", "call", "showing", "showing_off", "edited", "held", "unhold", "openers", "cadence", "replies", "noted", "merged", "skipped", "unskipped", "put_away", "brought_back", "starred", "unstarred", "unsent", "referred", "thanked", "thanks_held", "thanks_skipped", "price_change", "imported"] as const; | |
| 39 | 39 | export type Act = (typeof ACTS)[number]; |
| 40 | 40 | |
| 41 | 41 | /** What a `merged` row's detail holds: where the new arrival came from, and the lead they already were. */ |
src/contract.test.ts +11
| @@ -31,6 +31,8 @@ import { | ||
| 31 | 31 | parseLeadEmail, |
| 32 | 32 | previewSignIns, |
| 33 | 33 | addSignIns, |
| 34 | importLeads, | |
| 35 | previewImport, | |
| 34 | 36 | saveProfile, |
| 35 | 37 | setNote, |
| 36 | 38 | setState, |
| @@ -107,6 +109,7 @@ const NEW = { | ||
| 107 | 109 | message: "Hello", |
| 108 | 110 | }; |
| 109 | 111 | |
| 112 | let sheets = 0; | |
| 110 | 113 | const auditRow = (): number => { |
| 111 | 114 | record(db, { |
| 112 | 115 | who: { role: "anon" }, |
| @@ -134,6 +137,14 @@ const doors: Record< | ||
| 134 | 137 | const sheet = { address: "1 Elm St", text: "Pat Lee, [email protected]" }; |
| 135 | 138 | expect(previewSignIns(who, sheet).ok).toBe(added); |
| 136 | 139 | expect(addSignIns(db, who, sheet).ok).toBe(added); |
| 140 | // And a spreadsheet of leads (POST /leads/import and /preview). | |
| 141 | const sheet2 = { csv: `name,email,source\nSheet Lead,sheet${++sheets}@example.com,zillow\n` }; | |
| 142 | expect(previewImport(db, who, sheet2).ok).toBe(added); | |
| 143 | expect(importLeads(db, who, sheet2).ok).toBe(added); | |
| 144 | // The MCP tools are the same cell. | |
| 145 | const viaTool = (name: string) => (tools.find((t) => t.name === name)!.run(db, who, { csv: `name,email,source\nTool Lead,tool${++sheets}@example.com,zillow\n` }) as { ok: boolean }).ok; | |
| 146 | expect(viaTool("preview_import")).toBe(added); | |
| 147 | expect(viaTool("import_leads")).toBe(added); | |
| 137 | 148 | return added; |
| 138 | 149 | }, |
| 139 | 150 | // The spreadsheet is the same cell through its own door (GET /leads.csv, the MCP tool export_leads, `x` in the terminal). |
src/db.ts +2
| @@ -310,6 +310,8 @@ export const migrations: string[] = [ | ||
| 310 | 310 | // Kept when the lead is later lost; cleared when it is reopened. Leads qualified before this existed use their last change. |
| 311 | 311 | `ALTER TABLE lead ADD COLUMN qualified_at TEXT`, |
| 312 | 312 | `UPDATE lead SET qualified_at = updated_at WHERE state = 'qualified'`, |
| 313 | // A lead brought in from a spreadsheet: it has no inquiry clock and is not a lead in for the record or the report. | |
| 314 | `ALTER TABLE lead ADD COLUMN imported INTEGER NOT NULL DEFAULT 0 CHECK (imported IN (0, 1))`, | |
| 313 | 315 | ]; |
| 314 | 316 | |
| 315 | 317 | /** |
src/import.test.ts added +192
| @@ -0,0 +1,192 @@ | ||
| 1 | /** Leads in from a spreadsheet: the CSV reader, the preview, the import, and that an imported lead is history, not an inquiry. */ | |
| 2 | import { afterAll, expect, test } from "bun:test"; | |
| 3 | import { mkdtempSync } from "node:fs"; | |
| 4 | import { tmpdir } from "node:os"; | |
| 5 | import { join } from "node:path"; | |
| 6 | import { addLead, importLeads, listLeads, previewImport, reportWeek, type ImportRow } from "./actions.ts"; | |
| 7 | import { openDb } from "./db.ts"; | |
| 8 | import { leadsCsv } from "./export.ts"; | |
| 9 | import { MAX_ROWS, parseCsv, readTable } from "./import.ts"; | |
| 10 | import { desk, lastMessages, messagesOf, responseRecord } from "./leads.ts"; | |
| 11 | import { seed } from "./sample.ts"; | |
| 12 | import { serve } from "./server.ts"; | |
| 13 | import { sign } from "./who.ts"; | |
| 14 | import type { Who } from "./who.ts"; | |
| 15 | ||
| 16 | const anon: Who = { role: "anon" }; | |
| 17 | const alice: Who = { role: "user", id: "u1", name: "alice" }; | |
| 18 | const fresh = () => openDb(mkdtempSync(join(tmpdir(), "app-import-"))); | |
| 19 | const HEAD = "name,email,phone,source,message,state\r\n"; | |
| 20 | ||
| 21 | test("a CSV is read as RFC 4180: quoted commas, doubled quotes, line breaks inside a cell, CRLF, LF and a BOM", () => { | |
| 22 | const text = 'a,b,c\r\n"x, y","say ""hi""","line1\nline2"\n1,,3\r\n\r\n'; | |
| 23 | expect(parseCsv(text).rows).toEqual([ | |
| 24 | ["a", "b", "c"], | |
| 25 | ["x, y", 'say "hi"', "line1\nline2"], | |
| 26 | ["1", "", "3"], | |
| 27 | ]); | |
| 28 | expect(parseCsv("a\r\n").rows).toEqual([["a"]]); | |
| 29 | expect(parseCsv('a,"b').error).toContain("never closed"); | |
| 30 | expect(parseCsv(`h\n${"x\n".repeat(MAX_ROWS + 1)}`).error).toContain("at most"); | |
| 31 | }); | |
| 32 | ||
| 33 | test("headers are found by name in any case, in any order; a missing one is named", () => { | |
| 34 | const read = readTable("Phone,NAME,Source\n206-555-0101,Pat,zillow\n"); | |
| 35 | expect(read.ok && read.rows[0]).toMatchObject({ line: 2, name: "Pat", phone: "206-555-0101", source: "zillow", email: "" }); | |
| 36 | expect(readTable("name,email\nPat,[email protected]\n")).toMatchObject({ ok: false }); | |
| 37 | expect(readTable("name,source,email\n")).toMatchObject({ ok: false }); | |
| 38 | expect(readTable("")).toMatchObject({ ok: false }); | |
| 39 | }); | |
| 40 | ||
| 41 | test("the preview marks a good row, a duplicate, and a refused row with its reason; nothing is written", () => { | |
| 42 | const db = fresh(); | |
| 43 | seed(db, new Date()); | |
| 44 | const known = desk(db, 1)[0]!; | |
| 45 | const before = desk(db, -1).length; | |
| 46 | const csv = | |
| 47 | HEAD + | |
| 48 | "Ada New,[email protected],,zillow,Loved the house,\r\n" + | |
| 49 | `Dup Person,${known.email.toUpperCase()},,website,hi,\r\n` + | |
| 50 | "Ada Again,[email protected],,website,hi,\r\n" + | |
| 51 | ",[email protected],,zillow,x,\r\n" + | |
| 52 | "No Contact,,,zillow,x,\r\n" + | |
| 53 | "Bad Mail,not-an-email,,zillow,x,\r\n" + | |
| 54 | "Odd Source,[email protected],,billboard,x,\r\n" + | |
| 55 | "Odd State,[email protected],,zillow,x,archived\r\n"; | |
| 56 | const p = previewImport(db, alice, { csv }); | |
| 57 | if (!p.ok) throw new Error(p.message); | |
| 58 | expect(p.value.rows.map((r) => r.outcome)).toEqual(["new", "duplicate", "duplicate", "refused", "refused", "refused", "refused", "refused"]); | |
| 59 | expect(p.value.rows[1]!.match).toEqual({ id: known.id, name: known.name }); | |
| 60 | expect(p.value.rows[2]!.reason).toContain("row 2"); | |
| 61 | expect(p.value.rows[3]!.reason).toContain("name"); | |
| 62 | expect(p.value.rows[4]!.reason).toContain("email or a phone"); | |
| 63 | expect(p.value.rows[5]!.reason).toContain("email"); | |
| 64 | expect(p.value.rows[6]!.reason).toContain("source"); | |
| 65 | expect(p.value.rows[7]!.reason).toContain("state"); | |
| 66 | expect(p.value.counts).toEqual({ new: 1, duplicate: 2, refused: 5 }); | |
| 67 | expect(desk(db, -1).length).toBe(before); | |
| 68 | }); | |
| 69 | ||
| 70 | test("an import lands each usable row with no message, no clock and no count; duplicates and bad rows are returned", () => { | |
| 71 | const db = fresh(); | |
| 72 | const csv = HEAD + "Ada,[email protected],,zillow,Asked about Oak St,qualified\r\nBo,,206-555-0111,referral,,lost\r\nAda Dup,[email protected],,zillow,x,\r\nBad,,,zillow,x,\r\n"; | |
| 73 | const r = importLeads(db, alice, { csv }); | |
| 74 | if (!r.ok) throw new Error(r.message); | |
| 75 | expect(r.value.added.map((a) => a.name)).toEqual(["Ada", "Bo"]); | |
| 76 | expect(r.value.duplicates.length).toBe(1); | |
| 77 | expect(r.value.refused.length).toBe(1); | |
| 78 | const leads = desk(db, -1); | |
| 79 | expect(leads.map((l) => [l.name, l.state, l.imported, l.replied_at, l.consent, l.message])).toEqual([ | |
| 80 | ["Bo", "lost", true, null, false, "Brought in from a spreadsheet."], | |
| 81 | ["Ada", "qualified", true, null, false, "Asked about Oak St"], | |
| 82 | ]); | |
| 83 | for (const l of leads) expect(messagesOf(db, l.id)).toEqual([]); | |
| 84 | // Not a lead in: the record and the week's report skip them, and the desk shows no clock running. | |
| 85 | expect(responseRecord(db, Date.now()).leads).toBe(0); | |
| 86 | const week = reportWeek(db, alice, {}); | |
| 87 | if (!week.ok) throw new Error(week.message); | |
| 88 | expect(week.value.this_week.leads).toBe(0); | |
| 89 | expect(week.value.this_week.qualified).toBe(0); | |
| 90 | // Importing the same file again adds nothing, and says so. | |
| 91 | const again = importLeads(db, alice, { csv }); | |
| 92 | expect(again).toMatchObject({ ok: false, error: "invalid" }); | |
| 93 | expect(desk(db, -1).length).toBe(2); | |
| 94 | }); | |
| 95 | ||
| 96 | test("a real inquiry beside imported leads still counts and still has its clock", () => { | |
| 97 | const db = fresh(); | |
| 98 | importLeads(db, alice, { csv: HEAD + "Old,[email protected],,zillow,x,\r\n" }); | |
| 99 | const added = addLead(db, alice, { name: "New", email: "[email protected]", source: "zillow", message: "hi" }); | |
| 100 | if (!added.ok) throw new Error(added.message); | |
| 101 | expect(responseRecord(db, Date.now()).leads).toBe(1); | |
| 102 | const listed = listLeads(db, alice); | |
| 103 | if (!listed.ok) throw new Error(listed.message); | |
| 104 | expect(listed.value.counts.needs).toBe(1); | |
| 105 | }); | |
| 106 | ||
| 107 | test("the app's own export imports cleanly, with awkward names intact", () => { | |
| 108 | const source = fresh(); | |
| 109 | seed(source, new Date()); | |
| 110 | const leads = desk(source, -1).map((l, i) => (i === 0 ? { ...l, name: 'Smith, "Jo" Jr', phone: "+1 206 555 0101" } : i === 1 ? { ...l, name: "=SUM(A1)" } : l)); | |
| 111 | const csv = leadsCsv(leads, lastMessages(source)); | |
| 112 | const target = fresh(); | |
| 113 | const p = previewImport(target, alice, { csv }); | |
| 114 | if (!p.ok) throw new Error(p.message); | |
| 115 | const bad = p.value.rows.filter((r: ImportRow) => r.outcome === "refused"); | |
| 116 | expect(bad).toEqual([]); | |
| 117 | expect(p.value.counts.new + p.value.counts.duplicate).toBe(leads.length); | |
| 118 | const r = importLeads(target, alice, { csv }); | |
| 119 | if (!r.ok) throw new Error(r.message); | |
| 120 | const names = desk(target, -1).map((l) => l.name); | |
| 121 | expect(names).toContain('Smith, "Jo" Jr'); | |
| 122 | expect(names).toContain("=SUM(A1)"); | |
| 123 | const same = desk(target, -1).find((l) => l.name === 'Smith, "Jo" Jr')!; | |
| 124 | expect(same.phone).toBe("+1 206 555 0101"); | |
| 125 | expect(r.value.added.length).toBe(leads.length - r.value.duplicates.length); | |
| 126 | }); | |
| 127 | ||
| 128 | test("who may: anon is refused, and the trail holds a row per lead and the refusal", () => { | |
| 129 | const db = fresh(); | |
| 130 | expect(previewImport(db, anon, { csv: HEAD })).toMatchObject({ ok: false, error: "signin_required" }); | |
| 131 | expect(importLeads(db, anon, { csv: HEAD + "A,[email protected],,zillow,x,\r\n" })).toMatchObject({ ok: false, error: "signin_required" }); | |
| 132 | expect(desk(db, -1).length).toBe(0); | |
| 133 | importLeads(db, alice, { csv: HEAD + "A,[email protected],,zillow,x,\r\nB,[email protected],,zillow,x,\r\n" }); | |
| 134 | const rows = db.query<{ act: string | null; outcome: string }, []>("SELECT act, outcome FROM audit ORDER BY id").all(); | |
| 135 | expect(rows.filter((r) => r.act === "imported" && r.outcome === "ok").length).toBe(2); | |
| 136 | expect(rows.some((r) => r.outcome === "signin_required")).toBe(true); | |
| 137 | }); | |
| 138 | ||
| 139 | test("the cap: too many rows, too many characters, hostile text", () => { | |
| 140 | const db = fresh(); | |
| 141 | const many = HEAD + "A,[email protected],,zillow,x,\r\n".repeat(MAX_ROWS + 1); | |
| 142 | expect(previewImport(db, alice, { csv: many })).toMatchObject({ ok: false, error: "invalid" }); | |
| 143 | expect(previewImport(db, alice, { csv: "x".repeat(1_000_001) })).toMatchObject({ ok: false, error: "invalid" }); | |
| 144 | expect(previewImport(db, alice, { csv: 5 })).toMatchObject({ ok: false, error: "invalid" }); | |
| 145 | const p = previewImport(db, alice, { csv: HEAD + 'A\u001b[2J,[email protected],,zillow,x,\r\n' }); | |
| 146 | expect(p.ok && p.value.rows[0]!.outcome).toBe("refused"); | |
| 147 | }); | |
| 148 | ||
| 149 | // The doors: the page, the API, same origin. | |
| 150 | const KEY = "a-key-only-the-forge-holds"; | |
| 151 | const server = serve({ port: 0, dataDir: mkdtempSync(join(tmpdir(), "app-import-srv-")), identityKey: KEY }); | |
| 152 | const base = `http://${server.hostname}:${server.port}`; | |
| 153 | afterAll(() => server.stop(true)); | |
| 154 | const id = `u1|alice|user|${Date.now() + 600_000}`; | |
| 155 | const who = { "x-forge-identity": id, "x-forge-signature": sign(id, KEY) }; | |
| 156 | ||
| 157 | test("the API previews and imports JSON; the page takes an uploaded file and a confirm", async () => { | |
| 158 | const csv = HEAD + "Zed Import,[email protected],,website,hello,\r\n"; | |
| 159 | const json = { ...who, "content-type": "application/json" }; | |
| 160 | const prev = await fetch(base + "/leads/import/preview", { method: "POST", headers: json, body: JSON.stringify({ csv }) }); | |
| 161 | expect(prev.status).toBe(200); | |
| 162 | expect((await prev.json()).counts.new).toBe(1); | |
| 163 | const before = (await (await fetch(base + "/leads", { headers: who })).json()).leads.length; | |
| 164 | expect((await (await fetch(base + "/leads", { headers: who })).json()).leads.length).toBe(before); | |
| 165 | ||
| 166 | const form = new FormData(); | |
| 167 | form.set("file", new File([csv], "old-crm.csv", { type: "text/csv" })); | |
| 168 | const page = await fetch(base + "/leads/import/preview", { method: "POST", headers: who, body: form }); | |
| 169 | expect(page.status).toBe(200); | |
| 170 | const text = await page.text(); | |
| 171 | expect(text).toContain("Zed Import"); | |
| 172 | expect(text).toContain("Import 1 lead"); | |
| 173 | ||
| 174 | const done = await fetch(base + "/leads/import", { method: "POST", headers: json, body: JSON.stringify({ csv }) }); | |
| 175 | expect(done.status).toBe(201); | |
| 176 | expect((await done.json()).added[0].name).toBe("Zed Import"); | |
| 177 | const after = (await (await fetch(base + "/leads", { headers: who })).json()).leads; | |
| 178 | expect(after.length).toBe(before + 1); | |
| 179 | expect(after.find((l: { name: string }) => l.name === "Zed Import").imported).toBe(true); | |
| 180 | expect((await fetch(base + "/leads/import", { headers: who })).status).toBe(200); | |
| 181 | expect(await (await fetch(base + "/", { headers: who })).text()).toContain('href="/leads/import"'); | |
| 182 | }); | |
| 183 | ||
| 184 | test("anon and cross-site are refused at the door", async () => { | |
| 185 | const body = JSON.stringify({ csv: HEAD + "A,[email protected],,zillow,x,\r\n" }); | |
| 186 | const h = { "content-type": "application/json" }; | |
| 187 | expect((await fetch(base + "/leads/import", { method: "POST", headers: h, body })).status).toBe(401); | |
| 188 | const cross = await fetch(base + "/leads/import", { method: "POST", headers: { ...who, ...h, "sec-fetch-site": "cross-site" }, body }); | |
| 189 | expect(cross.status).toBe(403); | |
| 190 | const big = await fetch(base + "/leads/import/preview", { method: "POST", headers: { ...who, ...h }, body: "x".repeat(4_000_000) }); | |
| 191 | expect(big.status).toBe(413); | |
| 192 | }); |
src/import.ts added +115
| @@ -0,0 +1,115 @@ | ||
| 1 | /** | |
| 2 | * Leads in from a spreadsheet: a CSV (RFC 4180, as `export.ts` writes it) | |
| 3 | * read into plain rows. Nothing is written here and nothing is judged | |
| 4 | * beyond the shape of the file: whether a row is a usable lead is | |
| 5 | * `readImport`'s question in actions.ts. Plain text in, plain data out. | |
| 6 | */ | |
| 7 | ||
| 8 | /** The most a CSV may hold: characters when pasted, bytes when uploaded. */ | |
| 9 | export const MAX_CSV = 1_000_000; | |
| 10 | /** The most data rows (the header not counted) one import may carry. */ | |
| 11 | export const MAX_ROWS = 2000; | |
| 12 | ||
| 13 | /** What a header names, lower-cased; `status` is the export's word for `state`. */ | |
| 14 | const HEADERS: Readonly<Record<string, Column>> = { | |
| 15 | name: "name", | |
| 16 | email: "email", | |
| 17 | phone: "phone", | |
| 18 | source: "source", | |
| 19 | message: "message", | |
| 20 | state: "state", | |
| 21 | status: "state", | |
| 22 | "arrived at": "arrived", | |
| 23 | }; | |
| 24 | ||
| 25 | export type Column = "name" | "email" | "phone" | "source" | "message" | "state" | "arrived"; | |
| 26 | ||
| 27 | /** One data row as the sheet holds it: every column a string, empty when absent. */ | |
| 28 | export type RawRow = Readonly<Record<Column, string>> & { | |
| 29 | /** The row's number in the sheet: the header is row 1. */ | |
| 30 | readonly line: number; | |
| 31 | }; | |
| 32 | ||
| 33 | /** The records of a CSV; `error` when it cannot be read as one. Blank lines are skipped. */ | |
| 34 | export function parseCsv(text: string): { rows: string[][]; error?: string } { | |
| 35 | const rows: string[][] = []; | |
| 36 | let row: string[] = []; | |
| 37 | let cell = ""; | |
| 38 | let quoted = false; | |
| 39 | let touched = false; | |
| 40 | const endCell = (): void => { | |
| 41 | row.push(cell); | |
| 42 | cell = ""; | |
| 43 | }; | |
| 44 | const endRow = (): void => { | |
| 45 | endCell(); | |
| 46 | if (!(row.length === 1 && row[0] === "" && !touched)) rows.push(row); | |
| 47 | row = []; | |
| 48 | touched = false; | |
| 49 | }; | |
| 50 | const input = text.charCodeAt(0) === 0xfeff ? text.slice(1) : text; | |
| 51 | for (let i = 0; i < input.length; i++) { | |
| 52 | const ch = input[i]!; | |
| 53 | if (quoted) { | |
| 54 | if (ch !== '"') cell += ch; | |
| 55 | else if (input[i + 1] === '"') { | |
| 56 | cell += '"'; | |
| 57 | i++; | |
| 58 | } else quoted = false; | |
| 59 | continue; | |
| 60 | } | |
| 61 | if (ch === '"' && cell === "") { | |
| 62 | quoted = true; | |
| 63 | touched = true; | |
| 64 | } else if (ch === ",") { | |
| 65 | endCell(); | |
| 66 | touched = true; | |
| 67 | } else if (ch === "\r" || ch === "\n") { | |
| 68 | if (ch === "\r" && input[i + 1] === "\n") i++; | |
| 69 | endRow(); | |
| 70 | if (rows.length > MAX_ROWS + 1) return { rows, error: tooMany }; | |
| 71 | } else cell += ch; | |
| 72 | } | |
| 73 | if (quoted) return { rows, error: "A quoted cell is never closed: check for a stray quote mark." }; | |
| 74 | if (cell !== "" || row.length > 0 || touched) endRow(); | |
| 75 | if (rows.length > MAX_ROWS + 1) return { rows, error: tooMany }; | |
| 76 | return { rows }; | |
| 77 | } | |
| 78 | ||
| 79 | const tooMany = `An import is at most ${MAX_ROWS} rows; split the file and import it in parts.`; | |
| 80 | ||
| 81 | /** The export guards a cell a spreadsheet would run as a formula with a leading quote mark; take it off again. */ | |
| 82 | const unguard = (cell: string): string => (/^'[=+\-@\t\r]/.test(cell) ? cell.slice(1) : cell); | |
| 83 | ||
| 84 | /** A CSV read under its header into rows, or why it cannot be. */ | |
| 85 | export function readTable(text: string): { ok: true; rows: RawRow[] } | { ok: false; message: string } { | |
| 86 | const parsed = parseCsv(text); | |
| 87 | if (parsed.error !== undefined) return { ok: false, message: parsed.error }; | |
| 88 | const [header, ...data] = parsed.rows; | |
| 89 | if (header === undefined) return { ok: false, message: "The file is empty: it needs a header row, then one lead per row." }; | |
| 90 | const at = new Map<Column, number>(); | |
| 91 | header.forEach((h, i) => { | |
| 92 | const column = HEADERS[h.trim().toLowerCase()]; | |
| 93 | if (column !== undefined && !at.has(column)) at.set(column, i); | |
| 94 | }); | |
| 95 | if (!at.has("name") || !at.has("source") || !(at.has("email") || at.has("phone"))) | |
| 96 | return { | |
| 97 | ok: false, | |
| 98 | message: "The first row must name the columns: name, source, and email and/or phone (message and state are optional).", | |
| 99 | }; | |
| 100 | if (data.length === 0) return { ok: false, message: "The file has a header row but no leads under it." }; | |
| 101 | const cell = (r: string[], c: Column): string => unguard((r[at.get(c) ?? -1] ?? "").trim()); | |
| 102 | return { | |
| 103 | ok: true, | |
| 104 | rows: data.map((r, i) => ({ | |
| 105 | line: i + 2, | |
| 106 | name: cell(r, "name"), | |
| 107 | email: cell(r, "email"), | |
| 108 | phone: cell(r, "phone"), | |
| 109 | source: cell(r, "source"), | |
| 110 | message: cell(r, "message"), | |
| 111 | state: cell(r, "state"), | |
| 112 | arrived: cell(r, "arrived"), | |
| 113 | })), | |
| 114 | }; | |
| 115 | } |
src/leads.ts +27 −3
| @@ -81,6 +81,8 @@ export interface Lead { | ||
| 81 | 81 | readonly referred_by: string; |
| 82 | 82 | readonly referrer_email: string; |
| 83 | 83 | readonly referrer_phone: string; |
| 84 | /** Brought in from a spreadsheet: no clock, no drafts, and not counted as a lead in (see `insertImported`). */ | |
| 85 | readonly imported: boolean; | |
| 84 | 86 | } |
| 85 | 87 | |
| 86 | 88 | export interface Message { |
| @@ -192,7 +194,7 @@ export const inView = (lead: DeskLead, view: View): boolean => { | ||
| 192 | 194 | export const starredFirst = (leads: readonly DeskLead[], now: number): DeskLead[] => { |
| 193 | 195 | if (!leads.some((l) => l.starred)) return [...leads]; |
| 194 | 196 | const rank = (l: DeskLead): number => |
| 195 | l.replied_at === null && !CLOSED.includes(l.state) && between(l.created_at, now) >= GREEN_MS | |
| 197 | l.replied_at === null && !l.imported && !CLOSED.includes(l.state) && between(l.created_at, now) >= GREEN_MS | |
| 196 | 198 | ? 0 |
| 197 | 199 | : l.starred && !CLOSED.includes(l.state) |
| 198 | 200 | ? 1 |
| @@ -232,8 +234,9 @@ export type NewLead = { | ||
| 232 | 234 | readonly referrer_phone?: string; |
| 233 | 235 | }; |
| 234 | 236 | |
| 235 | type LeadRecord = Omit<Lead, "consent" | "noteUpdatedAt" | "lostReason" | "lostAt" | "starred"> & { | |
| 237 | type LeadRecord = Omit<Lead, "consent" | "noteUpdatedAt" | "lostReason" | "lostAt" | "starred" | "imported"> & { | |
| 236 | 238 | consent: number; |
| 239 | imported: number; | |
| 237 | 240 | starred: number; |
| 238 | 241 | note_updated_at: string | null; |
| 239 | 242 | lost_reason: LostReason | null; |
| @@ -245,6 +248,7 @@ const lead = ({ note_updated_at, lost_reason, lost_at, qualified_at: _qualifiedA | ||
| 245 | 248 | ...row, |
| 246 | 249 | consent: row.consent === 1, |
| 247 | 250 | starred: row.starred === 1, |
| 251 | imported: row.imported === 1, | |
| 248 | 252 | noteUpdatedAt: note_updated_at, |
| 249 | 253 | lostReason: lost_reason, |
| 250 | 254 | lostAt: lost_at, |
| @@ -537,7 +541,7 @@ export function responseRecord( | ||
| 537 | 541 | // One pass: the earlier window is read with the same rules as the current one. |
| 538 | 542 | const rows = db |
| 539 | 543 | .query<{ source: Source; created_at: string; replied_at: string | null }, [string, string]>( |
| 540 | "SELECT source, created_at, replied_at FROM lead WHERE created_at >= ?1 AND created_at < ?2", | |
| 544 | "SELECT source, created_at, replied_at FROM lead WHERE imported = 0 AND created_at >= ?1 AND created_at < ?2", | |
| 541 | 545 | ) |
| 542 | 546 | .all(since, endIso) |
| 543 | 547 | .map((row) => ({ |
| @@ -634,6 +638,26 @@ export function insertLead( | ||
| 634 | 638 | })(); |
| 635 | 639 | } |
| 636 | 640 | |
| 641 | /** | |
| 642 | * A lead brought in from a spreadsheet, as the agent's own history and not a | |
| 643 | * new inquiry: no first reply is drafted, no cadence is scheduled, and the | |
| 644 | * response clock never starts (`imported` keeps it out of the record and the | |
| 645 | * week's leads in). Nothing can be sent to it until the agent acts on it. | |
| 646 | * Consent is never carried in: a spreadsheet is not written consent to text. | |
| 647 | */ | |
| 648 | export function insertImported( | |
| 649 | db: Database, | |
| 650 | input: Omit<NewLead, "consent" | "referred_by" | "referrer_email" | "referrer_phone"> & { state: "new" | "qualified" | "lost" }, | |
| 651 | arrivedAt: string, | |
| 652 | at: string, | |
| 653 | ): number { | |
| 654 | return db | |
| 655 | .query<{ id: number }, [string, string, string, string, string, string, string, string]>( | |
| 656 | "INSERT INTO lead (name, email, phone, source, message, consent, state, created_at, updated_at, imported) VALUES (?1, ?2, ?3, ?4, ?5, 0, ?6, ?7, ?8, 1) RETURNING id", | |
| 657 | ) | |
| 658 | .get(input.name, input.email, input.phone, input.source, input.message, input.state, arrivedAt, at)!.id; | |
| 659 | } | |
| 660 | ||
| 637 | 661 | const digitsOf = (phone: string): string => phone.replace(/\D/g, ""); |
| 638 | 662 | |
| 639 | 663 | /** |
src/mcp.ts +28
| @@ -38,6 +38,8 @@ import { | ||
| 38 | 38 | logLeadReply, |
| 39 | 39 | parseLeadEmail, |
| 40 | 40 | previewSignIns, |
| 41 | previewImport, | |
| 42 | importLeads, | |
| 41 | 43 | addSignIns, |
| 42 | 44 | type Result, |
| 43 | 45 | CADENCE_MAX_DAY, |
| @@ -46,6 +48,7 @@ import { | ||
| 46 | 48 | import { MAX_NOTE, OUTCOMES } from "./calls.ts"; |
| 47 | 49 | import { MAX_EMAIL_TEXT } from "./intake.ts"; |
| 48 | 50 | import { MAX_SIGNIN_TEXT } from "./signins.ts"; |
| 51 | import { MAX_CSV, MAX_ROWS } from "./import.ts"; | |
| 49 | 52 | import { MAX_ADDRESS } from "./showings.ts"; |
| 50 | 53 | import { OPENER_MAX, SOURCES } from "./drafts.ts"; |
| 51 | 54 | import { title } from "./env.ts"; |
| @@ -71,6 +74,17 @@ const ONE = { | ||
| 71 | 74 | required: ["id"], |
| 72 | 75 | }; |
| 73 | 76 | |
| 77 | const SPREADSHEET = { | |
| 78 | type: "object", | |
| 79 | properties: { | |
| 80 | csv: { | |
| 81 | type: "string", | |
| 82 | maxLength: MAX_CSV, | |
| 83 | description: `CSV with a header row, at most ${MAX_ROWS} rows. Columns by header: name, source, email, phone, message, state (new/open, qualified, lost). The export_leads CSV imports as it is.`, | |
| 84 | }, | |
| 85 | }, | |
| 86 | required: ["csv"], | |
| 87 | }; | |
| 74 | 88 | const SIGNINS = { |
| 75 | 89 | type: "object", |
| 76 | 90 | properties: { |
| @@ -188,6 +202,20 @@ export const tools: readonly Tool[] = [ | ||
| 188 | 202 | inputSchema: SIGNINS, |
| 189 | 203 | run: (db, who, args) => addSignIns(db, who, args), |
| 190 | 204 | }, |
| 205 | { | |
| 206 | name: "preview_import", | |
| 207 | description: | |
| 208 | "Check a spreadsheet of existing leads (CSV) before bringing it in: each row as new, already on the desk (same email or phone) or refused with the reason. Nothing is written; then call import_leads.", | |
| 209 | inputSchema: SPREADSHEET, | |
| 210 | run: (db, who, args) => previewImport(db, who, args), | |
| 211 | }, | |
| 212 | { | |
| 213 | name: "import_leads", | |
| 214 | description: | |
| 215 | "Bring existing leads in from a CSV. Each new, usable row becomes a lead as history: no first reply drafted, no follow-ups scheduled, no response clock, not counted as a lead in. Duplicates are skipped, bad rows returned with their reason. Consent to text is never imported. Nothing is sent.", | |
| 216 | inputSchema: SPREADSHEET, | |
| 217 | run: (db, who, args) => importLeads(db, who, args), | |
| 218 | }, | |
| 191 | 219 | { |
| 192 | 220 | name: "new_sample_inquiry", |
| 193 | 221 | description: |
src/pages/desk.ts +1 −1
| @@ -107,7 +107,7 @@ export function deskPage(input: { | ||
| 107 | 107 | ${search} |
| 108 | 108 | ${views} |
| 109 | 109 | ${list} |
| 110 | <p class="export"><a href="${escape(exportHref(show, q))}" download>Download leads (CSV)</a> · <a href="/report">This week's report</a></p> | |
| 110 | <p class="export"><a href="${escape(exportHref(show, q))}" download>Download leads (CSV)</a>${input.mayAdd ? ' · <a href="/leads/import">Import leads (CSV)</a>' : ""} · <a href="/report">This week's report</a></p> | |
| 111 | 111 | </div> |
| 112 | 112 | ${input.mayAdd ? `<form class="reset" method="post" action="/sample/inquiry"><button class="button button-quiet">Bring in a new inquiry</button></form>${pasteForm}${signInForm(input.signIns)}${addForm(input.prefill)}` : ""} |
| 113 | 113 | ${input.mayPrice === true ? priceForm : ""} |
src/pages/import.ts added +63
| @@ -0,0 +1,63 @@ | ||
| 1 | /** | |
| 2 | * Bringing leads in from a spreadsheet: the form, then the preview of each | |
| 3 | * row as it will land. Server-rendered, no JavaScript; all text is escaped. | |
| 4 | */ | |
| 5 | ||
| 6 | import type { ImportPreview, ImportRow } from "../actions.ts"; | |
| 7 | import type { Agent } from "../env.ts"; | |
| 8 | import { SOURCE_LABEL } from "../drafts.ts"; | |
| 9 | import { escape } from "../http.ts"; | |
| 10 | import { MAX_CSV, MAX_ROWS } from "../import.ts"; | |
| 11 | import type { Who } from "../who.ts"; | |
| 12 | import { layout } from "./layout.ts"; | |
| 13 | ||
| 14 | const line = (r: ImportRow): string => { | |
| 15 | const who = `${escape(r.name)} · ${r.email === "" ? "no email" : escape(r.email)} · ${r.phone === "" ? "no phone" : escape(r.phone)}`; | |
| 16 | if (r.outcome === "refused") | |
| 17 | return `<li data-row="refused"><b>Cannot use</b> row ${r.line}: ${who} <span class="muted">(${escape(r.reason ?? "")} It will be left out.)</span></li>`; | |
| 18 | if (r.outcome === "duplicate") | |
| 19 | return `<li data-row="duplicate"><b>Already here</b> row ${r.line}: ${who} <span class="muted">(${escape(r.reason ?? "")} It will not be added again.)</span></li>`; | |
| 20 | return `<li data-row="new">Row ${r.line}: ${who} · ${escape(SOURCE_LABEL[r.source as keyof typeof SOURCE_LABEL])} · ${r.state === "new" ? "open" : r.state}</li>`; | |
| 21 | }; | |
| 22 | ||
| 23 | export function importPage(input: { | |
| 24 | who: Who; | |
| 25 | agent: Agent; | |
| 26 | csv?: string | undefined; | |
| 27 | preview?: ImportPreview | undefined; | |
| 28 | notice?: string | undefined; | |
| 29 | }): string { | |
| 30 | const { preview, csv = "" } = input; | |
| 31 | const shown = | |
| 32 | preview === undefined | |
| 33 | ? "" | |
| 34 | : `<section class="stack" role="status"> | |
| 35 | <p class="muted"><b>${preview.counts.new}</b> will be added, ${preview.counts.duplicate} already on the desk, ${preview.counts.refused} cannot be used. Nothing is written until you press Import. Imported leads get no drafted reply, no follow-ups and no response clock, and do not count as leads in.</p> | |
| 36 | <ol>${preview.rows.map(line).join("")}</ol> | |
| 37 | ${ | |
| 38 | preview.counts.new === 0 | |
| 39 | ? `<p class="muted">Nothing here can be added. Fix the spreadsheet and read it again.</p>` | |
| 40 | : `<form class="stack" method="post" action="/leads/import"> | |
| 41 | <input type="hidden" name="csv" value="${escape(csv)}"> | |
| 42 | <button class="button">Import ${preview.counts.new} lead${preview.counts.new === 1 ? "" : "s"}</button> | |
| 43 | </form>` | |
| 44 | } | |
| 45 | <p class="muted">Or change the spreadsheet and read it again:</p> | |
| 46 | </section>`; | |
| 47 | return layout({ | |
| 48 | who: input.who, | |
| 49 | agent: input.agent, | |
| 50 | path: "/leads/import", | |
| 51 | heading: "Import leads", | |
| 52 | notice: input.notice, | |
| 53 | body: `<h1>Bring your leads in from a spreadsheet</h1> | |
| 54 | <p class="muted">A CSV with a header row. Columns, by header (any case): <b>name</b>, <b>source</b> and <b>email</b> and/or <b>phone</b> are needed; <b>message</b> and <b>state</b> (new, open, qualified or lost) are optional. Your own "Download leads (CSV)" file imports as it is. At most ${MAX_ROWS.toLocaleString("en-US")} rows and 1 MB.</p> | |
| 55 | ${shown} | |
| 56 | <form class="stack" method="post" action="/leads/import/preview" enctype="multipart/form-data"> | |
| 57 | <label>Choose a .csv file <input class="input" type="file" name="file" accept=".csv,text/csv"></label> | |
| 58 | <label>Or paste the CSV <textarea class="input" name="csv" rows="10" maxlength="${MAX_CSV}">${escape(csv)}</textarea></label> | |
| 59 | <button class="button">Check the spreadsheet</button> | |
| 60 | </form> | |
| 61 | <p><a href="/">Back to the desk</a></p>`, | |
| 62 | }); | |
| 63 | } |
src/pages/parts.ts +3
| @@ -21,6 +21,9 @@ export function clock(lead: Lead, now = Date.now(), owedSince: string | null = n | ||
| 21 | 21 | const took = between(lead.created_at, lead.replied_at); |
| 22 | 22 | return `<span class="clock clock-done" data-tone="${tone(took)}"><small>answered in</small> <span class="clock-time">${elapsed(took)}</span></span>`; |
| 23 | 23 | } |
| 24 | // Brought in from a spreadsheet: nobody is waiting on a reply, so no clock runs. | |
| 25 | if (lead.imported) | |
| 26 | return `<span class="clock clock-done" data-tone="ok"><small>imported</small></span>`; | |
| 24 | 27 | const waited = between(lead.created_at, now); |
| 25 | 28 | return `<span class="clock" data-tone="${tone(waited)}" data-since="${escape(lead.created_at)}" role="timer" aria-label="Waiting since the inquiry"><small>waiting</small> <span class="clock-time">${elapsed(waited)}</span></span>`; |
| 26 | 29 | } |
src/routes.ts +146
| @@ -26,6 +26,8 @@ import { | ||
| 26 | 26 | readOpeners, |
| 27 | 27 | listAudit, |
| 28 | 28 | exportLeads, |
| 29 | importLeads, | |
| 30 | previewImport, | |
| 29 | 31 | listLeads, |
| 30 | 32 | reportWeek, |
| 31 | 33 | parseLeadEmail, |
| @@ -73,6 +75,7 @@ import { | ||
| 73 | 75 | fields, |
| 74 | 76 | html, |
| 75 | 77 | json, |
| 78 | readBody, | |
| 76 | 79 | readForm, |
| 77 | 80 | readJson, |
| 78 | 81 | readNotice, |
| @@ -84,7 +87,9 @@ import { MAX_NOTE, OUTCOMES } from "./calls.ts"; | ||
| 84 | 87 | import { MAX_ADDRESS, MAX_AHEAD_DAYS } from "./showings.ts"; |
| 85 | 88 | import { MAX_EMAIL_TEXT } from "./intake.ts"; |
| 86 | 89 | import { MAX_SIGNIN_TEXT } from "./signins.ts"; |
| 90 | import { MAX_CSV, MAX_ROWS } from "./import.ts"; | |
| 87 | 91 | import { LOST_REASONS, lostSaid } from "./lost.ts"; |
| 92 | import type { ImportPreview } from "./actions.ts"; | |
| 88 | 93 | import { findLead, MAX_INBOUND, MAX_LEAD_NOTE, MAX_QUERY, Q_HELP, showingsAhead, VIEW_HELP, VIEWS, type LeadDetail, type NewLead } from "./leads.ts"; |
| 89 | 94 | import { dueSaid } from "./sentences.ts"; |
| 90 | 95 | import { undoable } from "./undo.ts"; |
| @@ -96,6 +101,7 @@ import { leadPage } from "./pages/lead.ts"; | ||
| 96 | 101 | import { duePage } from "./pages/due.ts"; |
| 97 | 102 | import { activityPage } from "./pages/activity.ts"; |
| 98 | 103 | import { reportPage } from "./pages/report.ts"; |
| 104 | import { importPage } from "./pages/import.ts"; | |
| 99 | 105 | import { profilePage } from "./pages/profile.ts"; |
| 100 | 106 | import { currentAgent } from "./profile.ts"; |
| 101 | 107 | import { tokensCss } from "./tokens.ts"; |
| @@ -202,6 +208,7 @@ const LEAD_FIELDS = { | ||
| 202 | 208 | description: "The day (YYYY-MM-DD, the profile's calendar) a lead put away comes back, with a check-in draft at 8:00; null when it is not put away.", |
| 203 | 209 | }, |
| 204 | 210 | starred: { type: "boolean", description: "The agent keeps this lead close: starred open leads lead the desk." }, |
| 211 | imported: { type: "boolean", description: "Brought in from a spreadsheet: it has no drafted reply, no follow-ups and no response clock, and is not counted as a lead in." }, | |
| 205 | 212 | referred_by: { type: "string", maxLength: 120, description: "For a referral: the name of who sent them; empty when nobody is named. Never part of a message to the lead." }, |
| 206 | 213 | referrer_email: { type: "string", format: "email", description: "The referrer's email; empty when none." }, |
| 207 | 214 | referrer_phone: { type: "string", description: "The referrer's phone; empty when none." }, |
| @@ -605,6 +612,33 @@ const DUE_SENT: Schema = { | ||
| 605 | 612 | required: ["sent", "skipped"], |
| 606 | 613 | }; |
| 607 | 614 | |
| 615 | const IMPORT_BODY_SCHEMA: Schema = { | |
| 616 | type: "object", | |
| 617 | properties: { | |
| 618 | csv: { | |
| 619 | type: "string", | |
| 620 | maxLength: MAX_CSV, | |
| 621 | description: `RFC 4180 CSV with a header row; at most ${MAX_ROWS} rows. Columns by header, any case: name, source, email, phone, message, state (new/open, qualified, lost), arrived at. The app's own export imports as it is.`, | |
| 622 | }, | |
| 623 | }, | |
| 624 | required: ["csv"], | |
| 625 | }; | |
| 626 | const IMPORT_ROW: Schema = { | |
| 627 | type: "object", | |
| 628 | properties: { | |
| 629 | line: { type: "integer", description: "The row's number in the sheet; the header is row 1." }, | |
| 630 | name: { type: "string" }, | |
| 631 | email: { type: "string" }, | |
| 632 | phone: { type: "string" }, | |
| 633 | source: { type: "string" }, | |
| 634 | state: { enum: ["new", "qualified", "lost"] }, | |
| 635 | outcome: { enum: ["new", "duplicate", "refused"] }, | |
| 636 | reason: { type: "string", description: "Why a row is refused, or who a duplicate matches." }, | |
| 637 | match: { type: "object", properties: { id: { type: "integer" }, name: { type: "string" } } }, | |
| 638 | }, | |
| 639 | required: ["line", "name", "email", "phone", "source", "state", "outcome"], | |
| 640 | }; | |
| 641 | ||
| 608 | 642 | const STATUS: Record<Failure, number> = { |
| 609 | 643 | signin_required: 401, |
| 610 | 644 | forbidden: 403, |
| @@ -760,6 +794,40 @@ const posted = async <T>( | ||
| 760 | 794 | return done(result, to, said); |
| 761 | 795 | }; |
| 762 | 796 | |
| 797 | /** The most an import's request body may be: a 1 MB file, a form's encoding of it, and its confirm. */ | |
| 798 | const IMPORT_BODY = 3 * MAX_CSV + 64 * 1024; | |
| 799 | ||
| 800 | /** | |
| 801 | * The CSV of an import's POST: `csv` as JSON or a form sends it, or an uploaded | |
| 802 | * `file`. `undefined` when too large; a file that is not text is read as empty. | |
| 803 | */ | |
| 804 | async function readImportFields(request: Request): Promise<Record<string, unknown> | undefined> { | |
| 805 | const body = await readBody(request, IMPORT_BODY); | |
| 806 | if (body === undefined) return undefined; | |
| 807 | if (scripted(request)) { | |
| 808 | try { | |
| 809 | return fields(JSON.parse(new TextDecoder().decode(body))); | |
| 810 | } catch { | |
| 811 | return {}; | |
| 812 | } | |
| 813 | } | |
| 814 | const form = await new Request(request.url, { | |
| 815 | method: "POST", | |
| 816 | headers: { "content-type": request.headers.get("content-type") ?? "" }, | |
| 817 | body: new Uint8Array(body), | |
| 818 | }) | |
| 819 | .formData() | |
| 820 | .catch(() => undefined); | |
| 821 | if (form === undefined) return {}; | |
| 822 | const file = form.get("file"); | |
| 823 | if (file instanceof File && file.size > 0) { | |
| 824 | if (file.size > MAX_CSV) return undefined; | |
| 825 | return { csv: await file.text() }; | |
| 826 | } | |
| 827 | const csv = form.get("csv"); | |
| 828 | return { csv: typeof csv === "string" ? csv : "" }; | |
| 829 | } | |
| 830 | ||
| 763 | 831 | /** A whole number from the query, or undefined. */ |
| 764 | 832 | const whole = (value: string | null): number | undefined => |
| 765 | 833 | value !== null && /^\d+$/.test(value) ? Number(value) : undefined; |
| @@ -813,6 +881,16 @@ function desk( | ||
| 813 | 881 | ); |
| 814 | 882 | } |
| 815 | 883 | |
| 884 | /** The import page, with the spreadsheet as typed and how it was read when it could be. */ | |
| 885 | function importView(c: Context, shown: { csv?: string; preview?: ImportPreview; status?: number; notice?: string } = {}): Response { | |
| 886 | const { notice, clear } = readNotice(c.request); | |
| 887 | return html( | |
| 888 | importPage({ who: c.who, agent: currentAgent(c.db), ...shown, notice: shown.notice ?? notice }), | |
| 889 | shown.status ?? 200, | |
| 890 | clear, | |
| 891 | ); | |
| 892 | } | |
| 893 | ||
| 816 | 894 | export const routes: readonly Route[] = [ |
| 817 | 895 | // The desk. Forms post and come back; the page stays live over /events. |
| 818 | 896 | { |
| @@ -1108,6 +1186,74 @@ export const routes: readonly Route[] = [ | ||
| 1108 | 1186 | status: 201, |
| 1109 | 1187 | }, |
| 1110 | 1188 | }, |
| 1189 | { | |
| 1190 | // Leads in from a spreadsheet. Declared before /leads/:id, which would take "import" for an id. | |
| 1191 | method: "GET", | |
| 1192 | path: "/leads/import", | |
| 1193 | handle: (c) => importView(c), | |
| 1194 | }, | |
| 1195 | { | |
| 1196 | // Reads the CSV and writes nothing; the form gets the page back with the preview. | |
| 1197 | method: "POST", | |
| 1198 | path: "/leads/import/preview", | |
| 1199 | handle: async (c) => { | |
| 1200 | const input = await readImportFields(c.request); | |
| 1201 | if (input === undefined) | |
| 1202 | return scripted(c.request) ? json({ error: "too_large" }, 413) : text("too large", 413); | |
| 1203 | const read = previewImport(c.db, c.who, input); | |
| 1204 | if (scripted(c.request)) return answer(read); | |
| 1205 | const csv = typeof input.csv === "string" ? input.csv : ""; | |
| 1206 | if (!read.ok) return importView(c, { csv, status: STATUS[read.error], notice: read.message }); | |
| 1207 | return importView(c, { csv, preview: read.value }); | |
| 1208 | }, | |
| 1209 | api: { | |
| 1210 | summary: `Check a spreadsheet before bringing it in: each row as it would land, marked new, duplicate (its email or phone matches a lead already on the desk, or an earlier row) or refused with the reason (no name, neither email nor phone, bad email, unknown source). At most ${MAX_ROWS} rows and 1 MB. Nothing is written. Needs the same role as adding a lead.`, | |
| 1211 | body: IMPORT_BODY_SCHEMA, | |
| 1212 | answer: { | |
| 1213 | type: "object", | |
| 1214 | properties: { | |
| 1215 | rows: list(IMPORT_ROW), | |
| 1216 | counts: { type: "object", properties: { new: { type: "integer" }, duplicate: { type: "integer" }, refused: { type: "integer" } } }, | |
| 1217 | }, | |
| 1218 | }, | |
| 1219 | }, | |
| 1220 | }, | |
| 1221 | { | |
| 1222 | // Every usable row becomes a lead as history: no drafted reply, no follow-ups, no response clock. | |
| 1223 | method: "POST", | |
| 1224 | path: "/leads/import", | |
| 1225 | handle: async (c) => { | |
| 1226 | const input = await readImportFields(c.request); | |
| 1227 | if (input === undefined) | |
| 1228 | return scripted(c.request) ? json({ error: "too_large" }, 413) : text("too large", 413); | |
| 1229 | const imported = importLeads(c.db, c.who, input); | |
| 1230 | if (scripted(c.request)) return answer(imported, 201); | |
| 1231 | if (!imported.ok && imported.error === "invalid") | |
| 1232 | return importView(c, { csv: typeof input.csv === "string" ? input.csv : "", status: 422, notice: imported.message }); | |
| 1233 | const v = imported.ok ? imported.value : undefined; | |
| 1234 | return done( | |
| 1235 | imported, | |
| 1236 | "/", | |
| 1237 | v === undefined | |
| 1238 | ? undefined | |
| 1239 | : `Imported ${v.added.length} lead${v.added.length === 1 ? "" : "s"} with no message drafted${v.duplicates.length === 0 ? "" : `, ${v.duplicates.length} already on the desk`}${v.refused.length === 0 ? "" : `, ${v.refused.length} left out`}`, | |
| 1240 | ); | |
| 1241 | }, | |
| 1242 | api: { | |
| 1243 | summary: `Bring leads in from a spreadsheet: every new, usable row becomes a lead (state new, qualified or lost) with no first reply drafted, no follow-ups scheduled, no response clock, and no count in the response record or the week report. A row whose email or phone matches a lead already on the desk is not added again; rows that cannot be used are returned with their reason. Consent to text is never imported. Nothing is sent. The audit holds a row per lead. 422 when no row can be added.`, | |
| 1244 | body: IMPORT_BODY_SCHEMA, | |
| 1245 | answer: { | |
| 1246 | type: "object", | |
| 1247 | properties: { | |
| 1248 | id: { type: "integer", description: "The first lead added." }, | |
| 1249 | added: list({ type: "object", properties: { id: { type: "integer" }, name: { type: "string" } } }), | |
| 1250 | duplicates: list(IMPORT_ROW), | |
| 1251 | refused: list(IMPORT_ROW), | |
| 1252 | }, | |
| 1253 | }, | |
| 1254 | status: 201, | |
| 1255 | }, | |
| 1256 | }, | |
| 1111 | 1257 | { |
| 1112 | 1258 | method: "GET", |
| 1113 | 1259 | path: "/leads/:id", |
src/sentences.ts +1
| @@ -128,6 +128,7 @@ export function deed( | ||
| 128 | 128 | tried: `put ${lead} away`, |
| 129 | 129 | }; |
| 130 | 130 | } |
| 131 | if (row.act === "imported") return { done: `imported ${lead} from a spreadsheet`, tried: "import leads from a spreadsheet" }; | |
| 131 | 132 | if (row.act === "price_change") { |
| 132 | 133 | const priced = priceWords(row.detail, esc); |
| 133 | 134 | return { done: `changed a listing's price${priced}`, tried: "change a listing's price" }; |
src/server.test.ts +2
| @@ -325,6 +325,8 @@ test("the MCP server lists the tools and calls them as the caller", async () => | ||
| 325 | 325 | "parse_lead_email", |
| 326 | 326 | "preview_sign_ins", |
| 327 | 327 | "add_sign_ins", |
| 328 | "preview_import", | |
| 329 | "import_leads", | |
| 328 | 330 | "new_sample_inquiry", |
| 329 | 331 | "send_reply", |
| 330 | 332 | "send_due_followups", |
src/terminal.test.ts +29 −1
| @@ -12,7 +12,7 @@ import { openDb } from "./db.ts"; | ||
| 12 | 12 | import { agent } from "./env.ts"; |
| 13 | 13 | import { seed } from "./sample.ts"; |
| 14 | 14 | import { subscribe } from "./live.ts"; |
| 15 | import { insertLead } from "./leads.ts"; | |
| 15 | import { desk, insertLead } from "./leads.ts"; | |
| 16 | 16 | import { clean, plain, terminal, wrap } from "./terminal.ts"; |
| 17 | 17 | import type { Who } from "./who.ts"; |
| 18 | 18 | |
| @@ -635,3 +635,31 @@ test("w on the desk shows the week report in the page's words; esc goes back; a | ||
| 635 | 635 | expect(refused).toContain("Sign in to open the report."); |
| 636 | 636 | expect(refused).not.toContain("Leads in"); |
| 637 | 637 | }); |
| 638 | ||
| 639 | test("I pastes a CSV, previews it, and y imports it; a guest is refused and nothing lands", () => { | |
| 640 | const csv = "name,email,source\r\nTerm Import,[email protected],website\r\n,[email protected],zillow\r\n"; | |
| 641 | const db = fresh(); | |
| 642 | const t = terminal(db, alice); | |
| 643 | t.key("I"); | |
| 644 | t.key(csv); | |
| 645 | t.key("\x04"); | |
| 646 | const screen = read(t); | |
| 647 | expect(screen).toContain("1 will be added, 0 already on the desk, 1 cannot be used."); | |
| 648 | expect(screen).toContain("cannot use"); | |
| 649 | expect(listLeads(db, alice).ok && listLeads(db, alice).value.leads.some((l) => l.name === "Term Import")).toBe(false); | |
| 650 | t.key("y"); | |
| 651 | expect(read(t)).toContain("1 imported with no message drafted"); | |
| 652 | const listed = listLeads(db, alice); | |
| 653 | const one = listed.ok && listed.value.leads.find((l) => l.name === "Term Import"); | |
| 654 | expect(one && one.imported).toBe(true); | |
| 655 | ||
| 656 | const gdb = fresh(); | |
| 657 | const had = desk(gdb, -1).length; | |
| 658 | const g = terminal(gdb, { role: "anon" }); | |
| 659 | g.key("I"); | |
| 660 | g.key(csv); | |
| 661 | g.key("\x04"); | |
| 662 | expect(read(g)).not.toContain("will be added"); | |
| 663 | expect(desk(gdb, -1).length).toBe(had); | |
| 664 | expect(desk(gdb, -1).some((l) => l.name === "Term Import")).toBe(false); | |
| 665 | }); |
src/terminal.ts +58 −1
| @@ -5,7 +5,7 @@ | ||
| 5 | 5 | * may — the hints only leave out keys the table would refuse. |
| 6 | 6 | * |
| 7 | 7 | * Screens: |
| 8 | * desk j/k move · enter open · * star · f view (needs/open/closed/all) · / find by name, email, phone or note · n log a lead · v paste a lead email · $ price change · w this week's report · d due follow-ups · o open house sign-ins · i new sample inquiry · p profile · a activity (admin) · R restart samples (admin) · esc quit | |
| 8 | * desk j/k move · enter open · * star · f view (needs/open/closed/all) · / find by name, email, phone or note · n log a lead · v paste a lead email · $ price change · w this week's report · d due follow-ups · o open house sign-ins · I import a spreadsheet (CSV) · i new sample inquiry · p profile · a activity (admin) · R restart samples (admin) · esc quit | |
| 9 | 9 | * due j/k move · space tick · s send the ticked ones (each shown in full) · esc back |
| 10 | 10 | |
| 11 | 11 | * lead s approve and send · e edit, then send · w they wrote back · ] next lead that needs you · |
| @@ -35,6 +35,9 @@ import { | ||
| 35 | 35 | addSignIns, |
| 36 | 36 | reportWeek, |
| 37 | 37 | previewSignIns, |
| 38 | previewImport, | |
| 39 | importLeads, | |
| 40 | type ImportPreview, | |
| 38 | 41 | addSampleInquiry, |
| 39 | 42 | deleteLead, |
| 40 | 43 | editLead, |
| @@ -205,6 +208,7 @@ type View = | ||
| 205 | 208 | | { name: "lost"; id: number } |
| 206 | 209 | | { name: "details"; id: number; step: number; typed: string; fields: Record<string, string> } |
| 207 | 210 | | { name: "paste"; text: string } |
| 211 | | { name: "import"; text: string; preview?: ImportPreview } | |
| 208 | 212 | | { name: "price"; step: "address" | "price"; address: string; price: string } |
| 209 | 213 | | { name: "signins"; step: "address" | "text" | "check"; address: string; text: string; preview?: SignInPreview } |
| 210 | 214 | | { name: "add"; step: number; typed: string; fields: Record<string, string>; found?: Record<string, string>; note?: string } |
| @@ -275,6 +279,7 @@ export function terminal( | ||
| 275 | 279 | : l.ready !== null && isAnswer(l.ready.day) |
| 276 | 280 | ? l.ready.since |
| 277 | 281 | : null; |
| 282 | if (l.imported && l.replied_at === null && owed === null) return `${dim("imported".padStart(7))} no clock`; | |
| 278 | 283 | const waiting = l.replied_at === null || owed !== null; |
| 279 | 284 | const ms = owed !== null |
| 280 | 285 | ? between(owed, now()) |
| @@ -670,6 +675,29 @@ export function terminal( | ||
| 670 | 675 | return lines; |
| 671 | 676 | } |
| 672 | 677 | |
| 678 | function importScreen(v: Extract<View, { name: "import" }>): string[] { | |
| 679 | const lines = [bold("Import leads from a spreadsheet"), ""]; | |
| 680 | if (v.preview === undefined) { | |
| 681 | const shown = clean(v.text).split("\n").slice(-12); | |
| 682 | lines.push(...shown.map((l, i) => (i === shown.length - 1 ? `${l}█` : l))); | |
| 683 | lines.push("", dim("paste the CSV, header row first (name, source, email and/or phone, message, state) · ctrl-d check it · esc cancel")); | |
| 684 | return lines; | |
| 685 | } | |
| 686 | const { counts, rows } = v.preview; | |
| 687 | lines.push(`${counts.new} will be added, ${counts.duplicate} already on the desk, ${counts.refused} cannot be used.`, ""); | |
| 688 | const unusable = rows.filter((r) => r.outcome !== "new"); | |
| 689 | const shown = [...unusable, ...rows.filter((r) => r.outcome === "new")].slice(0, 12); | |
| 690 | for (const r of shown) | |
| 691 | lines.push( | |
| 692 | r.outcome === "new" | |
| 693 | ? ` row ${r.line}: ${clean(r.name)} · ${r.email === "" ? "no email" : clean(r.email)} · ${r.phone === "" ? "no phone" : clean(r.phone)}` | |
| 694 | : ` ${bold(r.outcome === "duplicate" ? "already here" : "cannot use")} row ${r.line}: ${clean(r.name)} (${clean(r.reason ?? "")})`, | |
| 695 | ); | |
| 696 | if (rows.length > shown.length) lines.push(dim(` … and ${rows.length - shown.length} more`)); | |
| 697 | lines.push("", dim(counts.new === 0 ? "nothing can be added · esc back" : `y import ${counts.new} (no message is drafted or sent, no clock starts) · esc cancel`)); | |
| 698 | return lines; | |
| 699 | } | |
| 700 | ||
| 673 | 701 | function pasteScreen(v: Extract<View, { name: "paste" }>): string[] { |
| 674 | 702 | const lines = [bold("Paste a lead email"), ""]; |
| 675 | 703 | const shown = clean(v.text).split("\n").slice(-12); |
| @@ -809,6 +837,8 @@ export function terminal( | ||
| 809 | 837 | ? detailsScreen(view) |
| 810 | 838 | : view.name === "paste" |
| 811 | 839 | ? pasteScreen(view) |
| 840 | : view.name === "import" | |
| 841 | ? importScreen(view) | |
| 812 | 842 | : view.name === "price" |
| 813 | 843 | ? priceScreen(view) |
| 814 | 844 | : view.name === "signins" |
| @@ -851,6 +881,7 @@ export function terminal( | ||
| 851 | 881 | cursor = 0; |
| 852 | 882 | } else if (input === "/") view = { name: "search", typed: q }; |
| 853 | 883 | else if (input === "v") view = { name: "paste", text: "" }; |
| 884 | else if (input === "I") view = { name: "import", text: "" }; | |
| 854 | 885 | else if (input === "$") view = { name: "price", step: "address", address: "", price: "" }; |
| 855 | 886 | else if (input === "o") view = { name: "signins", step: "address", address: "", text: "" }; |
| 856 | 887 | else if (input === "n") |
| @@ -1236,6 +1267,32 @@ export function terminal( | ||
| 1236 | 1267 | } |
| 1237 | 1268 | return; |
| 1238 | 1269 | } |
| 1270 | case "import": { | |
| 1271 | if (input === "\x1b") { | |
| 1272 | said = "Nothing imported."; | |
| 1273 | view = { name: "desk" }; | |
| 1274 | } else if (view.preview === undefined) { | |
| 1275 | if (input === "\x04") { | |
| 1276 | const read = previewImport(db, me, { csv: view.text }, new Date(now())); | |
| 1277 | if (read.ok) view = { ...view, preview: read.value }; | |
| 1278 | else { | |
| 1279 | said = read.message; | |
| 1280 | view = { name: "desk" }; | |
| 1281 | } | |
| 1282 | } else if (input === "\r") view = { ...view, text: `${view.text}\n` }; | |
| 1283 | else { | |
| 1284 | const text = input.replace(/\r\n?/g, "\n").replace(/[\x00-\x09\x0b-\x1f\x7f]/g, ""); | |
| 1285 | view = input === "\x7f" || input === "\b" ? { ...view, text: view.text.slice(0, -1) } : { ...view, text: view.text + text }; | |
| 1286 | } | |
| 1287 | } else if (input === "y") { | |
| 1288 | const imported = importLeads(db, me, { csv: view.text }, new Date(now())); | |
| 1289 | said = imported.ok | |
| 1290 | ? `${imported.value.added.length} imported with no message drafted${imported.value.duplicates.length === 0 ? "" : `, ${imported.value.duplicates.length} already on the desk`}${imported.value.refused.length === 0 ? "" : `, ${imported.value.refused.length} left out`}. Nothing is sent.` | |
| 1291 | : imported.message; | |
| 1292 | view = { name: "desk" }; | |
| 1293 | } | |
| 1294 | return; | |
| 1295 | } | |
| 1239 | 1296 | case "signins": { |
| 1240 | 1297 | if (input === "\x1b") { |
| 1241 | 1298 | said = "Nothing added."; |
new above
offline