A reproducible benchmark rather than a leaderboard screenshot. Each model gets the production prompt and a fictional company's website carrying one trap: a Careers link that is not a hiring signal, a 1919 product line, a certificate that implies a market, an instruction addressed to AI assistants. Scoring is string comparison against an answer key, with no judge model, so the same outputs always give the same numbers. Every raw answer is stored in the repository and a test recomputes the page from them. The same outputs are run through the pipeline's own verifier, so the page also reports what the guard let through.
The AI SDR pipeline reads a company's website and extracts what it makes, who it serves and how big it is, and every field has to come with the sentence that proves it. This runs that exact step, with the production prompt, on 17 pages written to tempt a model into saying something the page does not: 14 traps and 3 controls. Scoring is string comparison against an answer key, so the same outputs always produce the same numbers.
What this run changed
90% → 100%
The pipeline defaulted to GPT-4o-mini, which ranked 10 of 12 here with 30 bad citations and 6 unsupported claims across 51 answers. Its default is now GPT-6 Luna, with 0 and 0.
Cheapest model with a clean sheet
$0.28 / 1k pages
GPT-6 Luna. 6 of 12 models made no unsupported claim, fell for no trap and lost no answer. They cost from $0.28 to $15.31 per thousand pages.
The pipeline's own guard
16 of 19
unsupported claims got past verifyFacts, the check that deletes a field whose quote is not on the page. It also dropped 85 of 3173 true ones.
| Model | F1 (95% interval) | Precision | Recall | Claims made | Bad cit. | Unsupp. | Traps fallen | Cost / 1k | Median |
|---|---|---|---|---|---|---|---|---|---|
GPT-6 Luna OpenAIthe pipeline's default | 100.0%100%–100% | 100.0% | 100.0% | 321 | 0 | 0 | 0 / 42 | $0.28 | 3.7s |
Gemini 4 Argon Google | 100.0%100%–100% | 100.0% | 100.0% | 318 | 0 | 0 | 0 / 42 | $3.89 | 6.4s |
Claude Opus 5.5 Anthropic | 99.8%99%–100% | 100.0% | 99.6% | 298 | 0 | 0 | 0 / 42 | $15.31 | 5.0s |
Claude Haiku 5.5 Anthropic | 99.7%99%–100% | 99.7% | 99.6% | 312 | 0 | 1 | 0 / 42 | $0.57 | 4.0s |
GPT-6.1 Sol OpenAI | 99.5%98%–100% | 100.0% | 98.9% | 302 | 0 | 0 | 0 / 42 | $4.25 | 3.7s |
Gemini 3.8 Flash Google | 99.5%98%–100% | 100.0% | 98.9% | 301 | 0 | 0 | 0 / 42 | $4.63 | 7.3s |
Claude Sonnet 5.5 Anthropic | 98.9%97%–100% | 100.0% | 97.9% | 290 | 0 | 0 | 0 / 42 | $5.95 | 2.1s |
Llama 4 Maverick Meta | 96.9%92%–100% | 94.9% | 98.9% | 316 | 6 | 10 | 6 / 42 | $0.39 | 2.6s |
DeepSeek V4.1 Flash DeepSeek8 of 51 answers cut off | 90.2%82%–97% | 99.6% | 82.4% | 271 | 0 | 1 | 0 / 42 | $1.75 | 12.4s |
GPT-4o-mini OpenAIthe default before this run | 89.9%81%–96% | 88.4% | 91.4% | 311 | 30 | 6 | 6 / 42 | $0.23 | 3.7s |
Qwen3.8 Flash Alibaba24 of 51 answers too slow | 60.7%42%–77% | 99.3% | 43.7% | 145 | 0 | 1 | 0 / 42 | $1.18 | 35.0s |
Mistral Large 4 Mistral42 of 51 answers cut off | 25.1%4%–51% | 100.0% | 14.3% | 43 | 0 | 0 | 0 / 42 | $8.86 | 23.2s |
A cut-off answer hit the pipeline's 4,000-token cap, which covers a reasoning model's thinking as well as its answer. It scores as an empty answer, because that is what the pipeline would have received. It says the model does not fit this configuration, not that it reads pages badly.
A too-slow answer took longer than the 40 seconds the pipeline waits. The benchmark waits longer so it can record how long the answer took, then scores it as no answer, because the pipeline would never have read it. The median time is over every answer that came back, slow ones included.
Each cell is how many of 3 runs fell for that trap. 6 of 12 models never fell for any, so this set separates models that are safe for the pipeline from ones that are not, and does not rank the safe ones. Pick a cell to read what the model actually said.
| Model | furniture | history | name-inference | list-overreach | empty-page | wrong-number | parent-company | third-party-news | negation | injection | quote-fidelity | long-sentence | certification-inference | error-page |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna | ||||||||||||||
| Gemini 4 Argon | ||||||||||||||
| Claude Opus 5.5 | ||||||||||||||
| Claude Haiku 5.5 | ||||||||||||||
| GPT-6.1 Sol | ||||||||||||||
| Gemini 3.8 Flash | ||||||||||||||
| Claude Sonnet 5.5 | ||||||||||||||
| Llama 4 Maverick | ||||||||||||||
| DeepSeek V4.1 Flash | ||||||||||||||
| GPT-4o-mini | ||||||||||||||
| Qwen3.8 Flash | ||||||||||||||
| Mistral Large 4 |
A Careers link in the navigation is not a hiring signal
Loading the page and the answer…
The pipeline does not trust any of these models. verifyFacts deletes a field unless its quote is on the fetched page and names the thing claimed. Every answer in this run went through it unchanged, which makes the benchmark a test of the guard too.
Where trap claims got through
The other 7 it kept were not the trap a page was built around. They are real phrases from the page filed under the wrong heading, such as a product listed as a process.
A quote check proves a sentence is on the page and names the thing claimed. It cannot tell whose news the sentence is, whether it affirms or denies, or whether the thing named belongs in that field. That is a gap in the pipeline, found by this run, and it stays listed here until it is closed.
run 2026-11-11-5f7d1ede · commit 7133a96 · input 5f7d1edeeaf7c24a · total spend $2.41