Case study · Debugging judgment

The scanner recommended a product-manager job to an engineer

I run an automated job scanner that reads ~700 postings a day and pushes the best ones to my phone. On August 9 it pushed a Lead Product Manager role, scored 89 out of a 45-point bar. This is what the fix actually took: one wrong theory caught before it shipped, three rules that a corpus of 800 real postings forced on me, and a regression suite that runs in 178 ms.

800real postings re-scored to validate the fix
21flagged, hand-checked, zero false positives
10regression tests, real titles, zero dependencies
0npm packages in the whole scanner
The system

What it is, in one paragraph

Job Radar is a zero-dependency, zero-LLM Node script. Every morning at 7:15 it pulls three sources — Hacker News "Who is hiring?" through the Algolia API, RemoteOK, and two We Work Remotely RSS feeds — scores roughly 700 postings against my profile in about 15 seconds, and writes a ranked report. Anything over 45 surfaces; the top 3 unseen hits over 60 are merged straight into my target list and synced to a card on my phone.

Scoring is weighted regex, not a model: Claude Code +30, MCP +25, agentic +20, and so on, minus penalties for US-only, onsite, internship, or stale. No LLM anywhere in the pipeline — it costs nothing to run, it can't hallucinate a company that doesn't exist, and every score decomposes into the exact signals that produced it. That last property is the only reason the bug below was diagnosable at all.

Why this matters for the bug

The scorer is a bag of keywords over the whole posting body. Which means a role that is about AI scores exactly like a role that builds it.

The bug

A posting I am not qualified for, scored 89

HN comment 48358657"Scribd | Lead Product Manager, AI Developer Platform | US, Canada | REMOTE". It tripped Claude Code, agentic, MCP, Claude, and TypeScript on the strength of the product description alone, scored 89, cleared the auto-merge bar of 60, and landed on my phone as a target with a pitch pre-written for it.

It is a real posting for a real job. It is simply a job for a product manager, and I am an engineer. The scanner had no concept of which seat a posting was for — only of which words it contained.

The part that actually matters

Three days earlier, the obvious theory was wrong

This was the second false positive of its kind, and the first one is the more useful story. On August 6, Radar merged a target named "Location: Dallas, TX Remote: Yes Willing to relocate: Yes" — comment 49185076. It had written me a pitch addressed to a fellow job applicant.

The obvious explanation was right there: HN runs a "Who wants to be hired?" thread alongside "Who is hiring?", and my Algolia query was clearly picking up both. An easy fix — tighten the query, ship it, move on.

I checked it first. It was wrong.

The query returns only "Who is hiring?" stories — verified live against the API. And 49185076's parent is 49156683: the genuine August hiring thread. A real job seeker had simply posted their résumé where the employers were.

Had I shipped the obvious fix, I'd have tightened a query that was already correct, watched the bug survive, and lost trust in the dedupe layer for no reason. The five minutes spent falsifying the theory were the only five minutes that mattered.

Because the source was legitimate, the fix had to detect the shape of a résumé instead: HN's candidate template is a fixed set of labelled fields, and three or more of them together is something no employer post produces.

// candidate-filter.mjs — detected by SHAPE, not keyword.
const CANDIDATE_FIELDS = [
  /(^|[\s|·>])location\s*:/i,
  /(^|[\s|·>])remote\s*:/i,
  /willing to relocate\s*:/i,
  /(^|[\s|·>])technologies\s*:/i,
  /r(é|e)sum(é|e)\s*(\/\s*cv)?\s*:/i,
  /* …cv, email, seeking */
];

export const CANDIDATE_MIN_FIELDS = 3;

The threshold is 3 and not 1, because 180 genuine postings in that single thread use "Location:" or "Remote:" as ordinary fields. Requiring one or two would have rejected the thread.

Validated against all 183 comments of the August thread: 3 flagged, all genuine job seekers; 180 kept; zero false positives. Rejection happens at fetch time, so a candidate post never enters the dedupe cache and can't be silently skipped later.

The fix

Three rules, each one forced by the corpus

Back to the product manager. The fix is a −60 penalty on postings whose title names a role I can't fill. Every part of that sentence was rewritten at least once by contact with real data.

Rule 1

Match the title, never the body.

My first attempt scanned the whole posting and rejected most of the thread. Genuine engineering postings are full of sentences like "partner closely with product managers." The role a posting is for lives in its header; the roles it merely mentions live in its prose.

Rule 2

Match role nouns, not topic words.

Every near miss was an engineering title containing a non-engineering word: Sales Engineer, Solutions Engineer, Support Engineer, Design Engineer, Data Engineer. So the pattern matches sales manager and account executive — never a bare sales.

Rule 3

An engineering seat in the title vetoes the penalty.

HN posts advertise whole req lists at once. 49207409 is "2 Full Stack AI Engineer, 1 GTM"; Sourcegraph's is "SWE, Tech Lead, Agent Engineer, Product Manager, Product Marketing." Both name a seat for me. Without the veto, the first of those — a live target already on my board — dropped from 84 to 24.

The subtlety that cost the most time

HN postings arrive as one flat string, so Radar's "title" is just the first 110 characters — header plus whatever prose fits. That bleed alone was enough to misread a posting: Ashby's ad for engineering leaders reads "… $200k–$275k Ashby is the all-in-one recruiting platform", and the trailing sentence made it a recruiting job.

// Cut at the first sentence end, then drop the trailing pipe-delimited
// field — which is where the prose starts once a header has 3+ fields.
export function titlePart(title) {
  const head = String(title || "")
    .split(/(?<=[.!?])\s|\s[-–—]\s(?=[A-Z])/)[0]
    .slice(0, 200);
  const fields = head.split("|");
  return (fields.length >= 3 ? fields.slice(0, -1).join(" | ") : head).slice(0, 140);
}

And one guard I'd never have predicted — PM is a useful role token and also a clock time. A lookbehind keeps "apply by 5 pm ET" from reading as a product manager:

/(?<!\d)(?<!\d )\bpms?\b(?!\s*[-–—]?\s*(?:e[sd]t|p[sd]t|c[sd]t|m[sd]t|utc|gmt))/i

The whole thing lives in its own module rather than inside the scraper, because the scraper does network I/O at import time and I wanted these rules testable without touching the network.

Validation

I don't trust a filter I haven't tried to break

A rule that rejects the wrong posting is easy. A rule that rejects the wrong posting and nothing else has to be measured. So I re-scored 800 real postings from the June, July and August HN threads.

The harness tests itself first

Before measuring anything, the harness reproduces production's own recorded scores — 89, 113, 84 — to prove the scoring table it copied hasn't drifted from the live one. A validation run against a stale table would have looked exactly like a passing one.

PostingBeforeAfterOutcome
Scribd — Lead PM, AI Developer Platform 8929 Dropped ✓
Portless — AI Engineer, NA remote 113113 Unchanged ✓
Interview Resources — Full Stack AI Eng 8484 Unchanged ✓
Surfacing band (≥ 45) 3734 3 removed, all correct
Auto-merge band (≥ 60) 2120 1 removed — the Scribd posting

Across all 800: 21 flagged, every one hand-checked, zero false positives. Then I ran it against 255 RemoteOK and We Work Remotely titles — a completely different title shape, since those sources supply real structured titles instead of body text. 37 flagged, all genuine.

The regression suite

Ten cases, all built from real postings rather than invented strings, no dependencies, run with the Node built-in test runner:

$ npm run test:job-radar

✔ the Scribd Lead PM that started this is rejected
✔ the two genuine engineering targets survive
✔ rejects non-engineering roles
✔ keeps engineering titles that contain a non-engineering word
✔ keeps multi-role postings that include an engineering seat
✔ engineering management is rejected, engineering is not
✔ developer-as-audience does not count as an engineering seat
✔ title is cut back to the header, so body prose cannot decide the verdict
✔ a clock time in a title is not a PM role
✔ empty and malformed input is safe

ℹ tests 10   ℹ pass 10   ℹ fail 0   ℹ duration_ms 178.26

Live output, not a mock-up. Every bug on this page has a test named after it.

Known limitations

What I deliberately left open

Both of these are written into the source as comments, because a limitation nobody can find is just a bug with better manners.

Why this one

What I'd want you to take from it

I could have shown you a bigger system. I picked this one because a screenshot of a working app proves almost nothing now — anyone can generate one in an afternoon. What's harder to fake is the part in between: falsifying the obvious theory before building on it, letting real data overrule the design three times, measuring the fix against a corpus instead of a hunch, and writing down what's still wrong with it.

I build with AI coding agents daily — Claude Code is my primary tool, and this scanner was written with it. The agent wrote most of these regexes. It did not decide to go check whether the Algolia query was really the culprit, it did not decide that 3 was the right field threshold, and it did not decide that Engineering Manager was out of scope. That judgment is the job, and it's what I'd bring to yours.

Next

The rest of the work, and how to reach me

Ten-plus live systems, built and operated solo: a life-OS PWA, an autonomous product-health monitor, a SaaS crawler, published MCP servers, and a CRM running a real business.

carson.roell@gmail.com See the live systems Resume Try a live tool