Evergreen: Efficient Claim Verification for Semantic Aggregates
Evergreen checks claims in LLM-written summaries of tables by compiling each claim into a semantic verification query with citations. It keeps F1 at 0.94 while cutting cost 3.1x through early stopping, relevance sorting and prompt caching.
Paper (from PDF)
Read full textEvergreen: Efficient Claim Verification for Semantic Aggregates AlexanderW.Lee BenjaminHan ShayakSen BrownUniversityandSnowflakeInc. SnowflakeInc. SnowflakeInc. alexander_w_lee@brown.edu benjamin.han@snowflake.com shayak.sen@snowflake.com SamYeom UğurÇetintemel AnupamDatta SnowflakeInc. BrownUniversityandSnowflakeInc. SnowflakeInc. sam.yeom@snowflake.com ugur_cetintemel@brown.edu anupam.datta@snowflake.com ABSTRACT 1.00 Withrecentsemanticqueryprocessingengines,semanticaggrega- 0.95 tionhasbecomeaprimitiveoperator,enablingthereductionofa 0.90 relationintoanaturallanguageaggregateusinganLLM.However, theresultingsemanticaggregatemaycontainclaimsthatarenot 0.85 groundedintheunderlyingrelation.Verifyingsuchclaimsischal- 0.80 lenging:theyofteninvolvequantifiers,groupings,andcomparisons 0.75 overrelationsthatfarexceedLLMcontextwindowsandrequire 0.70 anexpensivecombinationofsemanticandsymbolicprocessing. WepresentEvergreen,asystemthatrecastsclaimverification 10−1 100 101 asasemanticqueryprocessingtaskwithtailoredoptimizationsand Cost ($) provenancecapture.Evergreencompileseachclaimintoadeclar- ative semantic verification query that can execute on the …
Explained
What happened
Evergreen (Brown and Snowflake; PVLDB 2026) checks whether an LLM's summary of a table is actually true. Engines like Snowflake Cortex AISQL let you run a semantic aggregate: "summarize these 5,000 reviews". The output can contain false claims ("most reviews are positive" when they aren't). Evergreen verifies each claim and cites the rows that prove it.
On a benchmark of 32 claims over Yelp and Twitter customer-support data:
- With a strong model, F1 0.94 at 3.1x lower cost than unoptimized verification.
- With a much weaker model, F1 0.87, beating the best external baseline (0.83) at 7x lower cost.
- Its configurations occupy the entire cost/quality Pareto frontier vs LLM-as-judge, RAG agents and recursive LMs.
Why it works
Claims from aggregates are usually logic over many rows: quantifiers ("all locations have complaints"), groupings and comparisons ("location A is top-ranked"). LLMs are bad at counting over huge inputs that don't fit the context window, but good at judging one row ("does this review complain about service?").
So Evergreen splits the work:
- An LLM breaks the summary into claims and turns each into a declarative verification query (relational operators plus per-row semantic filters).
- The query engine does the counting, grouping and comparing symbolically.
- LLM calls happen only per row, and are cut by verification-aware optimizations:
- Early stopping: "all X" fails at the first counterexample.
- Relevance sorting: check likely counterexamples first.
- Confidence sequences: for "most reviews...", sample rows until the estimate is statistically settled.
- Plus generic tricks: operator fusion, similarity filtering, prompt caching.
- Each verdict comes with a minimal set of supporting rows, based on semiring provenance.
The bigger pattern
This is neuro-symbolic design: the LLM handles fuzzy per-item judgments, and deterministic code handles logic and aggregation. It is the same principle as "don't ask the LLM to do arithmetic; give it a calculator", applied with database optimizer techniques.
What it means in practice
- If you build LLM features over large datasets (summaries of transactions, support tickets, reviews), don't trust an aggregate summary. Verify claims with SQL plus per-row LLM checks.
- Early stopping and sampling with confidence bounds are general tools for cutting LLM cost in any batch-classification pipeline (for example background jobs calling an LLM per record).
- Citations to source rows make AI outputs auditable, which matters in regulated domains.