Keenable AI Launches NEEDLE, a Continuously Updating Search Benchmark

Keenable has introduced NEEDLE, a live open-source benchmark that rebuilds its query set from fresh public sources instead of freezing a static dataset. [1] This approach is designed to address issues where a search agent might download gold labels mid-evaluation or rely on answers encoded in a model's parametric memory. [1] The benchmark regenerates news queries hourly using RSS feeds and Google Trends. [1] Queries for finance, scholar, legal, and rare-entity verticals are regenerated daily from sources including SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. [1] Fifteen search APIs are run against the same query text, and scores are compared against a pooled oracle engine. [1]

NEEDLE is implemented as an open-source evaluation harness rather than a commercial product. [1] It operates as a Python CLI installed via uv sync, utilizing subcommands for generating and running benchmarks. [1] The system requires an OpenRouter key for judging and individual API keys for each tested engine. [1] It allows users to recreate all used query streams alongside the ranking quality judgements. [1]

By continuously updating its evaluation data, NEEDLE directly addresses the persistent problem of data contamination in AI benchmarking. If a model has already memorized the answers in a static dataset, it is impossible to evaluate its actual web retrieval capabilities accurately. NEEDLE's approach of generating queries from live sources like SEC filings and RSS feeds ensures that search APIs are tested on genuinely new information. What the sources don't address: How the computational cost of running continuous, dynamically generated evaluations compares to traditional static benchmarking.