Home / Outbound Benchmarks / Typesafe Jev vs LLM agents for ICP

Typesafe AI (Jev) vs LLM agents for ICP qualification: 89% agreement on 3,141 companies for $0.13

We re-scored 3,141 company records with Typesafe's calibrated classifier and compared with verdicts previously produced by LLM agents reading the same records. Two-pass design reaches 89% agreement, 90% precision, 82% recall, in 5 minutes.

Key finding: One 'choice' question (9 company types) alone agreed with the LLM verdicts on 77% of 3,141 records; adding 4 atomic yes/no questions on the records classified 'brand' raised agreement to 89% with 90% precision and 82% recall. Total cost $0.13, about 5 minutes. The LLM agents themselves were far less stable: four batches of the same composition returned 133, 20, 112 and 36 'keep' decisions.
Measured onSeptember 28, 2026
Sample3,141 company records with prior LLM verdicts (benchmark); 400 records in a second vertical (calibration); 4 LLM batches of identical composition (variance test, September 30)
MethodRecords (name, description, industry, headcount, country) sent to Typesafe's systemone endpoint with typed questions: one 'choice' over 9 company types, then 4 'noul' (probability a statement is true) on informative description, trade or made-to-order, local-only, shippable to consumers. Thresholds found by grid search against the prior verdicts. Agreement, precision and recall computed against those verdicts as reference.
SourceVision Media internal measurement (campaign and API logs)
Benchmark on 3,141 consumer-brand company records
SetupAgreement with LLM verdictsPrecisionRecallCostTime
1 pass: 'choice' over 9 company types77%
2 passes: choice + 4 atomic yes/no questions with calibrated thresholds89%90%82%$0.13about 5 minutes
Variance of LLM agents on the same job (4 batches of identical composition, gray-zone records)
BatchRecords kept
1133
220
3112
436

What it is and is not

Typesafe's model answers closed questions with calibrated probabilities (yes/no, choice among options, ordered score). It generates no text: no extraction of free values, no arithmetic. Pricing at the time: $0.042 per million input tokens, output free; a record with three questions is about 620 tokens, so 5,000 records cost about $0.15. No batch endpoint, one request per record, about 6 concurrent workers under the 1,200 requests per minute limit.

Where it disagreed and who was right

How we use it now

  1. First filter on the whole raw pool: company type as 'choice', one yes/no per hard criterion, one fit score; combine in code; always include an 'other' option.
  2. Confidence below 0.7 goes to LLM review; 'drop' is reliable, 'keep' is right about 90% of the time. This divides the volume sent to LLM review by about 3.
  3. Never for writing: it chooses an angle well but the sentence it points at can be empty. Copy variables stay with a generative model.
  4. Do not put the offer's context inside the question: stating what we sell made it classify buyers of that product as its makers. Neutral 'choice', then separate yes/no questions.

What this measurement does not say

Questions

Is Typesafe AI better than an LLM for lead qualification?

It is far more stable and about 100 times cheaper for closed questions; it does not write anything. We use it as the first gate and keep LLM review for the gray zone.

Related measurements

Want this run on your pipeline?

We operate multichannel outbound (cold email, LinkedIn, paid retargeting, newsletters) for B2B companies, and we measure it the way these pages do.

Book a call