Using JEV for Steerable Reranking in RAG Pipelines

The Breakthrough

JEV provides a steerable, low-latency classification and ranking primitive that allows RAG pipelines to enforce dynamic business policies on retrieved documents, outperforming traditional cross-encoders that lack instruction-following capabilities.

What Actually Worked

• Policy-Driven Reranking: Instead of relying on semantic proximity via cosine similarity, JEV uses defined criteria as executable policy boundaries to rank chunks based on specific organizational requirements.
• Primitive Selection: The system utilizes three primary modes: binary decision-making (true/false classification), choice ranking (selecting from multiple options), and score ladders (assigning probabilities to options).
• Concurrency Optimization: By utilizing a 64-request pool, the system processes 17 decisions per second, reducing total latency for batch operations from 40 seconds to 7.6 seconds compared to serial execution.
• Instruction-Based Filtering: Users can define strict criteria, such as distinguishing between chunks that provide a direct answer versus those that merely share keywords or topics, allowing for the rejection of low-confidence retrievals based on a set threshold.

Before / After

• Baseline Accuracy: Using BM25 full-text search alone yielded 21% top-one accuracy.
• Improved Accuracy: Integrating JEV as a reranker increased top-one accuracy to 54%.

Context

Traditional RAG pipelines often rely on vector search and cross-encoders to rank documents. Vector search is limited to semantic proximity, failing when business logic or policy requirements change, while cross-encoders are frozen in time and cannot follow instructions. LLM-based rerankers offer instruction following but are prohibitively expensive and slow at scale. JEV serves as a drop-in replacement for these decision points, offering the steerability of an LLM with the cost-efficiency and speed required for production workflows.

Content References

{"type": "tool", "title": "JEV", "url": "https://typesafe.ai/blog/introducing-system-one-models-and-jev", "context": "reviewed"}, {"type": "tool", "title": "localGPT", "url": "https://bit.ly/localGPT", "context": "mentioned"}