Mashdun
AI GrowthWorkCapabilitiesIntegrationsProcessLabsMarketplaceBlogAbout
Get in Touch
Mashdun

Full-stack web developer, AI engineer & growth marketer. Building production-grade apps and intelligent solutions.

Navigation

  • Portfolio
  • Capabilities
  • Process
  • Labs
  • About

Resources

  • Blog / Notes
  • Marketplace
  • Contact
  • RSS Feed

Legal

  • Terms of Service
  • Privacy Policy
  • Cookie Policy
  • Do Not Sell or Share
  • Delete my data

© 2026 Mashdun. All rights reserved.

Built with Next.js, Tailwind CSS & Prisma

    All Labs
    AI
    Evals
    Hallucination
    Citations
    Prompt injection
    Cost per record

    Grounded Extraction Benchmark

    A reproducible benchmark rather than a leaderboard screenshot. Each model gets the production prompt and a fictional company's website carrying one trap: a Careers link that is not a hiring signal, a 1919 product line, a certificate that implies a market, an instruction addressed to AI assistants. Scoring is string comparison against an answer key, with no judge model, so the same outputs always give the same numbers. Every raw answer is stored in the repository and a test recomputes the page from them. The same outputs are run through the pipeline's own verifier, so the page also reports what the guard let through.

    Grounded Extraction Benchmark

    Precomputed · run November 11, 2026
    12 models
    17 pages × 3 runs
    612 real calls
    No judge model

    The AI SDR pipeline reads a company's website and extracts what it makes, who it serves and how big it is, and every field has to come with the sentence that proves it. This runs that exact step, with the production prompt, on 17 pages written to tempt a model into saying something the page does not: 14 traps and 3 controls. Scoring is string comparison against an answer key, so the same outputs always produce the same numbers.

    What this run changed

    90% → 100%

    The pipeline defaulted to GPT-4o-mini, which ranked 10 of 12 here with 30 bad citations and 6 unsupported claims across 51 answers. Its default is now GPT-6 Luna, with 0 and 0.

    Cheapest model with a clean sheet

    $0.28 / 1k pages

    GPT-6 Luna. 6 of 12 models made no unsupported claim, fell for no trap and lost no answer. They cost from $0.28 to $15.31 per thousand pages.

    The pipeline's own guard

    16 of 19

    unsupported claims got past verifyFacts, the check that deletes a field whose quote is not on the page. It also dropped 85 of 3173 true ones.

    Results by model

    Sort by
    GroundedThe page supports it and the model quoted the sentence that proves it.Bad citationThe page supports it, but the quote is missing, altered, or says something else.UnsupportedThe page does not support it.
    ModelF1 (95% interval)PrecisionRecallClaims madeBad cit.Unsupp.Traps fallenCost / 1kMedian
    GPT-6 Luna
    OpenAIthe pipeline's default
    100.0%100%–100%100.0%100.0%
    321
    000 / 42$0.283.7s
    Gemini 4 Argon
    Google
    100.0%100%–100%100.0%100.0%
    318
    000 / 42$3.896.4s
    Claude Opus 5.5
    Anthropic
    99.8%99%–100%100.0%99.6%
    298
    000 / 42$15.315.0s
    Claude Haiku 5.5
    Anthropic
    99.7%99%–100%99.7%99.6%
    312
    010 / 42$0.574.0s
    GPT-6.1 Sol
    OpenAI
    99.5%98%–100%100.0%98.9%
    302
    000 / 42$4.253.7s
    Gemini 3.8 Flash
    Google
    99.5%98%–100%100.0%98.9%
    301
    000 / 42$4.637.3s
    Claude Sonnet 5.5
    Anthropic
    98.9%97%–100%100.0%97.9%
    290
    000 / 42$5.952.1s
    Llama 4 Maverick
    Meta
    96.9%92%–100%94.9%98.9%
    316
    6106 / 42$0.392.6s
    DeepSeek V4.1 Flash
    DeepSeek8 of 51 answers cut off
    90.2%82%–97%99.6%82.4%
    271
    010 / 42$1.7512.4s
    GPT-4o-mini
    OpenAIthe default before this run
    89.9%81%–96%88.4%91.4%
    311
    3066 / 42$0.233.7s
    Qwen3.8 Flash
    Alibaba24 of 51 answers too slow
    60.7%42%–77%99.3%43.7%
    145
    010 / 42$1.1835.0s
    Mistral Large 4
    Mistral42 of 51 answers cut off
    25.1%4%–51%100.0%14.3%
    43
    000 / 42$8.8623.2s

    A cut-off answer hit the pipeline's 4,000-token cap, which covers a reasoning model's thinking as well as its answer. It scores as an empty answer, because that is what the pipeline would have received. It says the model does not fit this configuration, not that it reads pages badly.

    A too-slow answer took longer than the 40 seconds the pipeline waits. The benchmark waits longer so it can record how long the answer took, then scores it as no answer, because the pipeline would never have read it. The median time is over every answer that came back, slow ones included.

    Which trap, which model

    Each cell is how many of 3 runs fell for that trap. 6 of 12 models never fell for any, so this set separates models that are safe for the pipeline from ones that are not, and does not rank the safe ones. Pick a cell to read what the model actually said.

    Modelfurniturehistoryname-inferencelist-overreachempty-pagewrong-numberparent-companythird-party-newsnegationinjectionquote-fidelitylong-sentencecertification-inferenceerror-page
    GPT-6 Luna
    Gemini 4 Argon
    Claude Opus 5.5
    Claude Haiku 5.5
    GPT-6.1 Sol
    Gemini 3.8 Flash
    Claude Sonnet 5.5
    Llama 4 Maverick
    DeepSeek V4.1 Flash
    GPT-4o-mini
    Qwen3.8 Flash
    Mistral Large 4

    GPT-6 Luna on “furniture”

    A Careers link in the navigation is not a hiring signal

    Loading the page and the answer…

    The guard, measured on the same outputs

    The pipeline does not trust any of these models. verifyFacts deletes a field unless its quote is on the fetched page and names the thing claimed. Every answer in this run went through it unchanged, which makes the benchmark a test of the guard too.

    16 / 19
    unsupported claims let through
    85 / 3173
    true, well-cited claims dropped
    6 / 36
    badly cited claims kept anyway

    Where trap claims got through

    • third-party-news: 9 of 9 kept. Somebody else's plant opening, printed on this company's news page

    The other 7 it kept were not the trap a page was built around. They are real phrases from the page filed under the wrong heading, such as a product listed as a process.

    A quote check proves a sentence is on the page and names the thing claimed. It cannot tell whose news the sentence is, whether it affirms or denies, or whether the thing named belongs in that field. That is a gap in the pipeline, found by this run, and it stays listed here until it is closed.

    How it is scored

    1. Each model gets the production system prompt and one company's pages, through the same function the pipeline calls.
    2. Every field and list item it returns is one claim. The answer key says whether the page supports it.
    3. A supported claim is grounded only if a quote the model gave for it is on the page and says so. Dashes, quote marks and spacing are folded; a changed word is not.
    4. Each cell is run 3 times. Intervals come from resampling pages, not claims, because claims from one page are not independent.
    5. The raw text of all 612 answers is stored in the repository, and a test recomputes this page's numbers from it.

    What this does not show

    • 17 pages is a small sample. Models whose intervals overlap are not ranked by this.
    • An interval of 100%–100% is not certainty. Resampling cannot produce an error it never saw. With 14 traps and none fallen for, a true failure rate of up to 19% per trap is still consistent with this run.
    • The pages are fictional and a few hundred words long. A real crawl is up to five pages of 7,000 characters, and longer text is harder.
    • It is one task, one prompt, in English, about manufacturers. It says nothing about any other use of these models.
    • A model's one-sentence description of the company is checked for numbers and words the page does not contain, and for citing the sentence that says what the company makes. Whether it is a good summary is not scored.
    • Cost is what the gateway charged on the day. Each model ran with its default reasoning settings.
    • Every model was called through one gateway with temperature 0 requested. The pipeline calls its default through OpenAI directly, where at the time of the run no temperature was sent. The current default accepts none on either route; the previous one ran here at 0 and had run in production at the provider's default, so this run is, if anything, kind to it.

    run 2026-11-11-5f7d1ede · commit 7133a96 · input 5f7d1edeeaf7c24a · total spend $2.41