Jev-as-a-Judge for Portuguese IR: Relevance, Reranking

The paper evaluates Jev, a non-generative model, in relevance assessment and reranking tasks for Portuguese IR, using Brazilian Portuguese datasets. Exact agreement: Jev qrels matched 64.9% to 52.9% of individual relevance decisions but preserved systemic-level similarity with high Kendall’s τ (0.90–0.93). BM25 + Jev reranking substantially improved nDCG@10 (90.2% improvement) and MRR (relative 0.69%) in Quati, proving Jev operational effectiveness. Study objectives and positioning The paper evaluates Jev, a System One model designed for structured, non-generative decision-making, in two Brazilian Portuguese information-retrieval settings: automatic relevance assessment and candidate reranking. Its central premise is that both tasks can be expressed as bounded decisions rather than open-ended text generation. For relevance assessment, Jev selects a predefined relevance level for a query–document pair. For reranking, it estimates the probability that a document directly answers a query and uses that probability as a pointwise ranking score. This framing addresses two practical problems in IR. First, manually constructing qrels is expensive, particularly for specialized collections with long documents and domain-specific relevance criteria. Second, generative LLM rerankers can require repeated pairwise, setwise, or listwise inference, creating substantial computational and operational overhead. The paper therefore asks whether a structured decision model can produce qrels that preserve retrieval conclusions and whether it can improve a conventional BM25 ranking. The evaluation is conducted exclusively on Brazilian Portuguese resources, which is methodologically important because much of the LLM-based IR literature is centered on English. The four collections span web passages, jurisprudence, normative acts, and legislative proposals. They also differ in document length, relevance scale, and source of reference judgments. Quati and JurisTCU use four-level labels, with LLM-generated references and expert review in the latter; NormasTCU and Ulysses-RFCorpus use three-level labels derived from domain-expert annotation. This diversity permits an assessment of Jev under heterogeneous annotation conditions rather than on a single uniform benchmark. The paper’s motivation is consistent with prior work showing that LLM-generated judgments can produce high system-level agreement despite imperfect query–document-level agreement (Upadhyay et al., 2024). It also …