Skip to content

Commit 1339f43

Browse files
committed
fix: the scorer never knew the company sizes or the industries the customer chose
The wizard asks for headcount bands and sectors, stores both on the agent row, and until now only the yes/no fit judge in messages/generate.ts ever read them. scoreLead, which produces the number and the sentence a customer reads on their queue and which decides who is worth an invitation, was told the ICP, the countries and what the customer sells, and nothing about the two other fields they filled in. So an agent set to "11-50, marketing agencies" scored its leads with a model that had never been given either, and two of the four things the wizard collects were decorative. Both arrive as evidence rather than as a filter, and the wording is the whole of it. A headcount is printed on a company page and almost never on a person's profile, so a band list dropped in with no instruction reads as a filter and scores a good prospect at zero for a fact LinkedIn never showed us. The scorer is told to read the size off what the person and their company say about themselves, and to leave the score alone and say so when nothing does. The industry line carries the warning the fit judge already carries next door: it is the sector their company works in, not the words in their job title. targetingLines is pure and exported so the wording is testable without a model call. Also fixes two tests left red by ef1f12e: "Greater Paris Metropolitan Region" now resolves to France, so it is no longer an example of an unreadable place. Greater Cambridge Area takes its place, being one of the labels the table refuses to resolve on purpose, and the readable half of that change gets the test it did not have.
1 parent ef1f12e commit 1339f43

4 files changed

Lines changed: 142 additions & 6 deletions

File tree

‎worker/src/ai.ts‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -353,6 +353,47 @@ export async function classifyReply(
353353
return { handOver: verdict, why: why || answer.trim().slice(0, 120), intent };
354354
}
355355

356+
/**
357+
* The customer's own targeting, written for a model rather than for a form.
358+
*
359+
* Pure and exported so the wording is testable without a model call, because
360+
* the wording is the whole of it: a band list dropped into a prompt with no
361+
* instruction is read as a filter, and a scorer that treats an unprintable fact
362+
* as a filter scores good prospects at zero.
363+
*
364+
* "11-50" is a range LinkedIn prints on a company page and almost never on a
365+
* person's profile, so the scorer is told to read it off what the person and
366+
* their company say about themselves, and to leave the score alone when nothing
367+
* says either way.
368+
*/
369+
export function targetingLines(companySizes: string[], industries: string[]): string[] {
370+
const lines: string[] = [];
371+
const bands = companySizes.filter((b) => b.trim());
372+
const trades = industries.filter((t) => t.trim());
373+
374+
if (bands.length) {
375+
lines.push(
376+
`Company sizes the customer sells to, by headcount: ${bands.join(", ")}. ` +
377+
"Judge it from what the person and their company say about themselves, an agency of a few people, a team, a department, a group. " +
378+
"A company clearly bigger or smaller than every band listed is a weak match however good the title reads. " +
379+
"When nothing here says how big the company is, that is neither evidence for nor against them, so score the rest and say the size is unknown."
380+
);
381+
}
382+
383+
if (trades.length) {
384+
// The industry is the sector the prospect's company works in. Matching it
385+
// against a job title is how a supermarket planning assistant reached a
386+
// queue meant for website owners, and the fit judge next door carries the
387+
// same warning for the same reason.
388+
lines.push(
389+
`Industries the customer sells to: ${trades.join(", ")}. ` +
390+
"That is the sector their own company works in, not the words in their job title."
391+
);
392+
}
393+
394+
return lines;
395+
}
396+
356397
/** Full scoring, still on the fast model. The writer here is the biggest cost mistake available. */
357398
export async function scoreLead(
358399
ctx: AgentContext,
@@ -442,6 +483,26 @@ export async function scoreLead(
442483
? `The customer only sells to people in: ${places.map((c) => countryName(c)).join(", ")}. Anybody outside those scores 0, whatever their title reads.`
443484
: "";
444485

486+
/**
487+
* The headcount bands and the sectors, which this prompt never carried.
488+
*
489+
* The wizard asks for both, stores both on the agent row, and until now only
490+
* the yes/no fit judge in messages/generate.ts ever read them. So a customer
491+
* who said "11-50, marketing agencies" got a queue scored by a model that had
492+
* been told neither, and the number and the sentence they read on the
493+
* dashboard came from a judgement their own targeting never entered. Two of
494+
* the four things the wizard collects were decorative.
495+
*
496+
* Both are written as evidence to weigh rather than as a filter. The size of
497+
* a company is rarely printed on a person's profile and inferring it from a
498+
* headline is a guess; saying so keeps a good prospect from being scored down
499+
* for a fact LinkedIn never showed us, which is the failure mode a hard rule
500+
* here would create.
501+
*/
502+
const bands = ctx.companySizes ?? [];
503+
const trades = ctx.cfg.leads.industries ?? [];
504+
const wantedShape = targetingLines(bands, trades).join("\n");
505+
445506
const m = await models();
446507
const answer = await generate(
447508
ctx,
@@ -454,6 +515,7 @@ Headline: ${profile.headline}
454515
Company: ${profile.company ?? "unknown"}
455516
${profile.location ? `Where LinkedIn says they are: ${profile.location}` : ""}
456517
${wantedPlaces}
518+
${wantedShape}
457519
${profile.signal ? `How they were found: ${profile.signal}` : ""}${repeats}
458520
${profile.about ? `About: ${profile.about.slice(0, 600)}` : ""}
459521

‎worker/src/linkedin/sources.test.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,23 @@ test("a card with no place on it is unknown, not allowed", () => {
298298
assert.equal(placeOf(FRANCE, null), "unknown");
299299
assert.equal(placeOf(FRANCE, ""), "unknown");
300300
assert.equal(placeOf(FRANCE, " "), "unknown");
301-
assert.equal(placeOf(FRANCE, "Greater Paris Metropolitan Region"), "unknown");
301+
// A metro label built on a city that exists in two countries stays unread on
302+
// purpose, which is the rule the whole table runs on: there is a Cambridge in
303+
// England and one in Massachusetts, and LinkedIn prints both this way.
304+
assert.equal(placeOf(FRANCE, "Greater Cambridge Area"), "unknown");
305+
});
306+
307+
/**
308+
* The metro labels, which used to be unreadable and are now read from the city.
309+
*
310+
* LinkedIn labels most of the United States, and Paris, Zurich and Mumbai with
311+
* it, as a metro area with no country after it. Answering unknown for those
312+
* sent an agent aimed at the United States to a profile visit for most of its
313+
* own market and then closed the lead.
314+
*/
315+
test("a metro label with no country after it is read from its city", () => {
316+
assert.equal(placeOf(FRANCE, "Greater Paris Metropolitan Region"), "in");
317+
assert.equal(placeOf(FRANCE, "Greater Boston Area"), "out");
302318
});
303319

304320
test("the country is read whatever language LinkedIn printed it in", () => {

‎worker/src/safety/geo-fence.test.ts‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -194,21 +194,45 @@ test("the profile settling on the wrong country stops everything", async () => {
194194
/**
195195
* A country that still cannot be read is a refusal, not a shrug.
196196
*
197-
* "Greater Paris Metropolitan Region" names no country and neither does a blank
198-
* profile. Guessing at one is the whole shape of the bug, so a customer who
199-
* named their countries gets the strict reading and the row says why.
197+
* "Greater Cambridge Area" names no country and neither does a blank profile.
198+
* Guessing at one is the whole shape of the bug, so a customer who named their
199+
* countries gets the strict reading and the row says why. The city is the
200+
* example on purpose: there is a Cambridge in England and one in Massachusetts,
201+
* so it is one of the few metro labels the table refuses to resolve.
200202
*/
201203
test("a place the profile does not give either is refused and said so", async () => {
202204
await freshDb();
203205
const lead = await seed("Unreadable", null);
204206
const { actions, done } = spyActions();
205-
const fenced = onlyInCountries(actions, ctxWith(AMERICAS), page, async () => "Greater Paris Metropolitan Region");
207+
const fenced = onlyInCountries(actions, ctxWith(AMERICAS), page, async () => "Greater Cambridge Area");
206208

207209
assert.equal(await fenced.sendConnect(lead, ""), "failed");
208210
assert.deepEqual(done, []);
209211
assert.equal((await rowOf("Unreadable")).excluded_reason, UNREADABLE_REASON);
210212
});
211213

214+
/**
215+
* A metro label the table does resolve is decided on, not sent to a visit.
216+
*
217+
* This is the half of the same change that matters to the customer: an agent
218+
* aimed at the Americas keeps a lead labelled "Greater Boston Area" instead of
219+
* spending a profile read to find out what the string already said.
220+
*/
221+
test("a metro label naming a city we know is decided without a second look", async () => {
222+
await freshDb();
223+
const lead = await seed("Bostonian", "Greater Boston Area");
224+
const { actions, done } = spyActions();
225+
let visits = 0;
226+
const fenced = onlyInCountries(actions, ctxWith(AMERICAS), page, async () => {
227+
visits += 1;
228+
return null;
229+
});
230+
231+
assert.equal(await fenced.sendConnect(lead, ""), "sent");
232+
assert.deepEqual(done, ["invite:Bostonian"]);
233+
assert.equal(visits, 0);
234+
});
235+
212236
test("a profile that will not load is refused rather than allowed", async () => {
213237
await freshDb();
214238
const lead = await seed("Would Not Load", null);

‎worker/src/score.test.ts‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { test } from "node:test";
22
import assert from "node:assert/strict";
3-
import { cleanReason, parseScore } from "./ai.ts";
3+
import { cleanReason, parseScore, targetingLines } from "./ai.ts";
44

55
/**
66
* The shapes the scorer actually answered in.
@@ -101,3 +101,37 @@ test("the parser and the guard agree on a real answer", () => {
101101
assert.equal(parsed.score, 78);
102102
assert.equal(parsed.reason, "Co-founder with product architecture background.");
103103
});
104+
105+
/**
106+
* The targeting the wizard collects and the scorer used to ignore.
107+
*
108+
* Both fields were stored on the agent row from the first release and read only
109+
* by the yes/no fit judge, so the number and the sentence a customer reads on
110+
* the dashboard were produced by a model that had never been told the headcount
111+
* bands or the sectors they asked for.
112+
*/
113+
114+
test("the headcount bands reach the prompt, worded as evidence rather than as a filter", () => {
115+
const [sizes] = targetingLines(["1-10", "11-50"], []);
116+
assert.ok(sizes?.includes("1-10, 11-50"));
117+
assert.ok(/neither evidence for nor against/i.test(sizes ?? ""));
118+
});
119+
120+
test("the industries reach the prompt, and say they are a sector and not a job title", () => {
121+
const lines = targetingLines([], ["Marketing agencies", "Ecommerce"]);
122+
assert.equal(lines.length, 1);
123+
assert.ok(lines[0]?.includes("Marketing agencies, Ecommerce"));
124+
assert.ok(/not the words in their job title/i.test(lines[0] ?? ""));
125+
});
126+
127+
test("an agent that named neither adds nothing to the prompt", () => {
128+
assert.deepEqual(targetingLines([], []), []);
129+
assert.deepEqual(targetingLines(["", " "], [""]), []);
130+
});
131+
132+
test("both named produce both lines, in the order the prompt reads them", () => {
133+
const lines = targetingLines(["11-50"], ["SaaS"]);
134+
assert.equal(lines.length, 2);
135+
assert.ok(lines[0]?.startsWith("Company sizes"));
136+
assert.ok(lines[1]?.startsWith("Industries"));
137+
});

0 commit comments

Comments
 (0)