One search per question
Each configuration receives the original question through its public endpoint. We use the first 10 returned results. Optional direct answers are disabled; there are no extra fetches or agent search loops.
SEARCH API EVALUATION
One question set.
Every search API.
A shared reader, a fixed text budget, and the same questions. Compare answer accuracy, observed search latency, and public list prices.
Organized by SearchPipe, which also participates. These are our measurements, not an official benchmark leaderboard.
DatasetSimpleQA Verified
Sample300 fixed questions
Run20261024-v4
Reader / judgedeepseek-flash / deepseek-flash
Answer accuracy is graded semantically. Literal quote matching is a separate diagnostic, not a pass/fail gate. The protocol records the pre-publication correction and its unchanged search snapshots.
SAME-RUN RESULTS
Answer accuracy
Correct answers divided by all frozen questions. Whiskers show 95% Wilson confidence intervals.
Only complete configurations enter the chart. Small score gaps may be statistical noise; paired comparisons are included in the JSON export. List prices verified 2026-10-24.
The numbers behind the chart
| Configuration | Accuracy | 95% interval | Search p50 / p95 | List price / 1k | Status |
|---|---|---|---|---|---|
| SearchPipe Basic | 32.3% 97/300 | 27.3–37.8% | 2371 / 5871 ms | $5.00 | 0 search errors |
| SearchPipe Advanced | 32.7% 98/300 | 27.6–38.2% | 2389 / 5565 ms | $10.00 | 0 search errors |
| Tavily Basic | 84.0% 252/300 | 79.4–87.7% | 2186 / 3998 ms | $8.00 | 18 search errors |
| Tavily Advanced | 87.3% 262/300 | 83.1–90.6% | 4754 / 6602 ms | $16.00 | 18 search errors |
| Exa Auto | 97.0% 291/300 | 94.4–98.4% | 2232 / 3476 ms | $7.00 | 0 search errors |
| Parallel Fast | 93.7% 281/300 | 90.3–95.9% | 1384 / 2081 ms | $1.00 | 0 search errors |
Provider-blind AI spot-check: 180 answers reviewed by Codex. Agreement with the primary grader: 100.0%. This is a reference-consistency check, not independent human verification.
METHOD, BEFORE MARKETING
What this comparison measures
A shared reader
Returned text is capped at 2,400 UTF-8 bytes per document and 18,000 content bytes per question. One reader answers from those documents and is asked for a supporting quote. Literal quote matching is recorded separately; formatting differences never override the answer grade.
A fixed denominator
A separate grading call checks the answer against the reference. Search errors and abstentions remain in the denominator. Evaluator failures stay pending until resolved. We never select the best of multiple successful runs.
Costs you can compare
Charts use public list prices, without free credits or promotions. Search latency includes retries and backoff, excludes reader and judge time, and describes successful queries in this run. Provider cache states may differ.
The fixed sample is proportionally stratified by topic from Google's SimpleQA Verified. This search-assisted protocol differs from its original closed-book evaluation. Results do not transfer to other publishers' model setups, samples, or historical runs. The reader and judge use separate calls to the same model family; grading is not an independent human assessment. Public model aliases can change; returned model identifiers are recorded.
Public benchmark mirrors appeared in retrieved results, including reference-answer wording in a reviewed item. This can inflate public-dataset scores. We retained every frozen question and made no post-hoc exclusions; this is not a contamination-free measure of business performance.
The quote check verifies literal provenance, not logical entailment. Full raw responses remain in the reproducibility archive. The public per-question export contains answers, quotes, verdicts, timing and source URLs. SearchPipe on GitHub ↗