· 7 min read · benchmarks · search api · methodology
Six Web Search Configurations, 300 Questions: The Honest Scorecard
Why we publish our own scores
Every search API vendor publishes numbers that put themselves first. We built a public benchmark page so that anyone can check our claims with a fixed question set, a shared answer model and confidence intervals — and this post walks through the first full run, including the rows where SearchPipe does not win.
The full scorecard lives at /benchmarks with the protocol, raw exports and download links. This article is the story of the run: what we measured, how, and what the numbers do and do not mean.
The setup: 300 questions, one reader, one judge
The dataset is a proportional, topic-stratified sample of 300 questions from SimpleQA Verified (MIT licensed, revision pinned by SHA-256). Each question goes to every search configuration exactly once — one search call, top 10 results, at most 2,400 UTF-8 bytes per document and 18,000 bytes of context in total.
A shared reader model (DeepSeek Flash) turns each configuration's results into a final answer, and a shared judge (the same model) grades the answer semantically against the golden answer. Because every configuration sees the identical reader and judge, the comparison isolates retrieval quality: the search layer is the only thing that changes between rows.
Latency is end-to-end from the caller's perspective, including retries and backoff on the public endpoints, measured on successful queries only. Prices are the public list prices as of 2026-10-24, normalized to dollars per 1,000 calls.
The results
The table below is rendered from the same validated export that powers the /benchmarks page — we do not maintain a second copy of these numbers. Confidence intervals are Wilson intervals at 95%.
| Configuration | Accuracy | 95% CI | p50 latency | List price / 1k |
|---|---|---|---|---|
| SearchPipe Basic | 32.3% | 27.3–37.8% | 2.37s | $5.0 |
| SearchPipe Advanced | 32.7% | 27.6–38.2% | 2.39s | $10.0 |
| Tavily Basic | 84.0% | 79.4–87.7% | 2.19s | $8.0 |
| Tavily Advanced | 87.3% | 83.1–90.6% | 4.75s | $16.0 |
| Exa Auto | 97.0% | 94.4–98.4% | 2.23s | $7.0 |
| Parallel Fast | 93.7% | 90.3–95.9% | 1.38s | $1.0 |
Full protocol, per-question results and downloads →
Two rows are ours: searchpipe-basic and searchpipe-advanced. On this question set, with this reader, they land well behind the leaders. We are publishing that anyway, for three reasons: the methodology rewards configurations that return enough relevant text in the first page of results; our default content allowance per result was tuned smaller than that; and the advanced tier's wider recall did not compensate on factual single-hop questions the way it does on research-style queries. The recall-quality release we shipped on 2026-10-24 targets exactly these gaps, and the next run will be published against the same frozen question set.
What this benchmark is not
This is not the official closed-book SimpleQA Verified evaluation, and it is not comparable with scores other publishers have posted for their own runs. The returned text is byte-bounded: there are no extra fetches, no multi-turn agent loop and no browsing. And because SearchPipe organizes and participates in this evaluation, treat our rows with the skepticism you would give any self-reported number — the protection is the frozen question set and the shared reader, not our goodwill.
Latency numbers reflect the observed cache state of public endpoints on the day of the run; they are a snapshot, not a service-level agreement.
Reproduce or extend it
The protocol, the sampled question list with its fixed seed, and the raw per-question results are all downloadable from the benchmarks page, and the export carries the dataset revision hash so the question set can be re-derived independently. If you run a configuration we do not cover, the comparison table is the contract: same questions, same reader, same judge.
Questions about the methodology: support@searchpipe.tech.