Typesafe AI (Jev) vs LLM agents for ICP qualification: 89% agreement on 3,141 companies for $0.13
We re-scored 3,141 company records with Typesafe's calibrated classifier and compared with verdicts previously produced by LLM agents reading the same records. Two-pass design reaches 89% agreement, 90% precision, 82% recall, in 5 minutes.
| Setup | Agreement with LLM verdicts | Precision | Recall | Cost | Time |
|---|---|---|---|---|---|
| 1 pass: 'choice' over 9 company types | 77% | ||||
| 2 passes: choice + 4 atomic yes/no questions with calibrated thresholds | 89% | 90% | 82% | $0.13 | about 5 minutes |
| Batch | Records kept |
|---|---|
| 1 | 133 |
| 2 | 20 |
| 3 | 112 |
| 4 | 36 |
What it is and is not
Typesafe's model answers closed questions with calibrated probabilities (yes/no, choice among options, ordered score). It generates no text: no extraction of free values, no arithmetic. Pricing at the time: $0.042 per million input tokens, output free; a record with three questions is about 620 tokens, so 5,000 records cost about $0.15. No batch endpoint, one request per record, about 6 concurrent workers under the 1,200 requests per minute limit.
Where it disagreed and who was right
- On the 'brand' class it over-included: 672 false positives in one pass, because it reads 'makes products' where the LLM judged made-to-order or trade (B2B) or a local shop. It also classifies an empty description as 'brand' with 98% confidence by guessing from the industry; a code guard (description under 60 characters = review) fixes that.
- On a second vertical (operators vs their software vendors, 400 records), it corrected the LLM agents in the right direction: it dropped four B2B platforms that the agents had kept and kept six real operators that the agents had dropped.
- The 227 brands it missed were mostly luxury and made-to-order, where the LLM reference was itself borderline.
How we use it now
- First filter on the whole raw pool: company type as 'choice', one yes/no per hard criterion, one fit score; combine in code; always include an 'other' option.
- Confidence below 0.7 goes to LLM review; 'drop' is reliable, 'keep' is right about 90% of the time. This divides the volume sent to LLM review by about 3.
- Never for writing: it chooses an angle well but the sentence it points at can be empty. Copy variables stay with a generative model.
- Do not put the offer's context inside the question: stating what we sell made it classify buyers of that product as its makers. Neutral 'choice', then separate yes/no questions.
What this measurement does not say
- The reference verdicts were produced by LLM agents, not by humans; 'agreement' measures consistency with that reference, and the variance test shows the reference itself is noisy.
- Thresholds were tuned on this pool; a new vertical needs a 40-record test before a full run.
- Prices and model version (jev-1.13) as of September 2026.
Questions
Is Typesafe AI better than an LLM for lead qualification?
It is far more stable and about 100 times cheaper for closed questions; it does not write anything. We use it as the first gate and keep LLM review for the gray zone.
Related measurements
- The 'companies that are hiring' signal selected 283 software vendors and 36 real clinics: measured on a healthcare outbound list
- GetLeads job_titles filter is fuzzy: 29% of 'Founder, CEO, Owner, President' results were not executives
Want this run on your pipeline?
We operate multichannel outbound (cold email, LinkedIn, paid retargeting, newsletters) for B2B companies, and we measure it the way these pages do.
Book a call