Introducing RubriX: Fine-Tunable Rubric Scoring at Constant Latency as Rubrics Grow
Highlights
- 75.4% exact accuracy on held-out rubrics (new criteria, familiar domains), against 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge.
- About 28 ms per judgment, flat: from 2 to 100 levels, and from 10 to 3,000 tokens per level, once a rubric's levels are cached.
- 7.7× faster than Jev and 18.8× faster than a Qwen3-8B judge at 3,000 tokens per level (Jev's approximate network time subtracted).
- Beats G-Eval with GPT-3.5 on every SummEval dimension and both QAGS subsets, and matches G-Eval with GPT-4 on SummEval coherence (against the results in the G-Eval paper).
- Fine-tuned in 2.8 hours on a single H100.
- Publicly available on Hugging Face: huggingface.co/CobuildX-AI/RubriX-8B.
Every team that evaluates AI with an LLM judge pays a hidden tax. Each time a response is scored against a rubric, the judge reads the entire rubric again: every level description, on every single call. Want more levels? Sharper, more detailed descriptions? Every judgment gets slower and more expensive, and long rubrics eventually stop fitting in the judge's context at all.
Today we're introducing RubriX, a non-autoregressive rubric judge that turns that cost curve flat. RubriX reads each level description once per rubric, then scores every response against it in a single pass, with no text generation. On new rubrics in the domains it was trained on, it is the most accurate of the judges we tested by a wide margin, and its latency stays at about 28 ms whether a rubric has 2 levels or 100. RubriX-8B is publicly available on Hugging Face at huggingface.co/CobuildX-AI/RubriX-8B.
The hidden tax in LLM judges
A rubric question has a fixed set of answers: which of these levels does this response meet? A generative judge answers it the long way round. It reads the conversation, the response and every level description, writes out a verdict token by token, and you parse the result. In our Jev red-teaming study we saw the same thing for guardrails: when the output is a choice, a model that scores the options directly is faster and returns probabilities instead of text.
RubriX takes that idea one step further for rubrics. A rubric's level descriptions don't change from one response to the next, so there's no reason to read them again for every response.
Meet RubriX: read the rubric once, judge every response in one pass
RubriX's architecture is inspired by contrastive language models (CLM) [7]. It is a bi-encoder: one encoder (Qwen3-8B [9] with LoRA adapters [11]) embeds the input, meaning the conversation, the response and the criterion, and separately embeds each level description. Two small projection heads map both into a shared space, and the score of each level is a temperature-scaled cosine similarity. A softmax over the rubric's own levels turns those scores into a probability for every level.

Here x is the input, ℓₖ the k-th of the rubric's K level descriptions, f and g the two projection heads and τ a temperature. pₖ is the probability of level k.
Because the level embeddings never depend on the input, they can be computed once per rubric and cached. After that, each new response costs a single encoder pass, however many levels the rubric has and however long they are.
See RubriX in action
A few tricky cases, picked to show what RubriX catches: responses that look fine at a glance but miss on one detail. Each tab shows the rubric question, its levels, the content RubriX judged, and the probability it gives every level.
Example rubric questions
Does the code correctly solve the task, including edge cases?
- 0Broken. The code does not run or does not address the task.
- 1Often wrong. The code gives wrong results for common inputs.
- 2Several gaps. The code works for typical inputs but fails several edge cases.
- 3One edge case. The code is correct except for one edge case.
- 4Correct. The code is correct for all inputs, including edge cases.
Content to evaluate
Write a Python function is_prime(n) that returns True if n is a prime number.
def is_prime(n): for i in range(2, int(n ** 0.5) + 1): if n % i == 0: return False return True
RubriX: probability of each level
Most likely: One edge case (32%)
Broken
Often wrong
Several gaps
One edge case
Correct
Broken: 11%, Often wrong: 24%, Several gaps: 23%, One edge case: 32%, Correct: 9%
Why it's tricky: The loop is right, but there is no check for n < 2, so 0 and 1 are reported as prime.
Teaching a contrastive model that levels have an order
The original CLM treats options as an unordered set, which is fine for picking a category but not for rubrics: predicting level 2 when the answer is 5 should cost more than predicting 4. We train with cross-entropy over each rubric's own levels, plus the ranked probability score (RPS) [12], which compares the predicted cumulative distribution with the true one, so near misses are penalised less than far ones.
![L_RPS = 1/(K−1) Σ_k (P_k − 1[y ≤ k])², P_k = Σ_{j≤k} p_j; L = L_CE + λ_t L_RPS, λ_t = α · mean(L_CE) / mean(L_RPS)](/blog/rubrix-rubric-judge/rps_equations.png)
Pₖ is the predicted probability that the level is at most k, y is the gold level, and 1[y ≤ k] is 1 if the gold level is at most k and 0 otherwise. ℒCE = −log pᵧ is the cross-entropy, and the bars are running averages of the two losses.
RPS is much smaller than cross-entropy in magnitude, so a fixed weight lets it fade into the background. Instead, we weight it adaptively: λₜ is the ratio of the two losses' running averages (with α = 1), so on average the RPS term contributes about as much as cross-entropy throughout training. This is a simple instance of adaptive loss balancing, which is well established in multi-task learning [13–14].
Trained in under three hours
- Data: 34,261 examples covering 2,284 distinct criteria, a mix of public rubric datasets (Feedback Collection, HelpSteer, HelpSteer2, HelpSteer3, UltraFeedback, ASAP 2.0) and a proprietary dataset.
- Capped per criterion: at most 25 examples per criterion, balanced across levels, so a few large sources do not dominate.
- Compute: one epoch on a single H100 in 2.8 hours, training 62.5M parameters (43.6M adapter and 18.9M head parameters).
Results
Far ahead on new rubrics in familiar domains
First, the test set held out from the training mix: 6,749 judgments on rubrics whose criteria never appear in training. Here RubriX is well ahead, with 75.4% exact accuracy against 57.1% for Jev and 49.8% for the Qwen3-8B judge, and 97.6% within one level against 89.6% and 88.4%.

Competitive on human-labelled benchmarks
Next, eight human-labelled benchmark sets the model never trained on: HealthBench [1], BiGGen [2], ELLIPSE [3], SummEval [4], JudgmentBench [5] (split into its pass/fail checklist items and its multi-level error counts) and the two subsets of QAGS [6].

Here the picture is more balanced. RubriX is within 6 percentage points of Jev on QAGS (1.7 and 3.0 points behind), ELLIPSE (1.2), HealthBench (4.8) and BiGGen (5.2), and ahead of it on JudgmentBench's multi-level items. Jev keeps a larger lead on SummEval and JudgmentBench's checklist items. The Qwen3-8B judge is the most accurate on ELLIPSE and on JudgmentBench's multi-level items.

Within one level, RubriX reaches 96.4% on JudgmentBench's multi-level items, where Jev reaches 42.4%.
Ranking responses the way people do
Accuracy asks whether a judge lands on the exact level. A second question matters just as much for evaluation: given several responses to the same input, does the judge put them in the same order as people do? We measure this with summary-level Spearman ρ and Kendall τ [15]: responses are grouped by input and criterion, the judge's scores are correlated with the human scores within each group, and the group correlations are averaged.
| Benchmark | Category | RubriX ρ | RubriX τ | Jev ρ | Jev τ | Qwen3-8B judge ρ | Qwen3-8B judge τ | G-Eval-4 ρ | G-Eval-4 τ | G-Eval-3.5 ρ | G-Eval-3.5 τ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SummEval | Coherence | 0.581 | 0.450 | 0.649 | 0.510 | 0.432 | 0.330 | 0.582 | 0.457 | 0.440 | 0.335 |
| Consistency | 0.496 | 0.413 | 0.559 | 0.473 | 0.458 | 0.377 | 0.507 | 0.425 | 0.386 | 0.318 | |
| Fluency | 0.435 | 0.360 | 0.484 | 0.402 | 0.461 | 0.381 | 0.455 | 0.378 | 0.424 | 0.347 | |
| Relevance | 0.422 | 0.328 | 0.474 | 0.370 | 0.360 | 0.280 | 0.547 | 0.433 | 0.385 | 0.293 | |
| AVG | 0.483 | 0.388 | 0.541 | 0.439 | 0.428 | 0.342 | 0.514 | 0.418 | 0.401 | 0.320 | |
| QAGS | QAGS-CNN | 0.719 | 0.583 | 0.715 | 0.579 | 0.656 | 0.518 | 0.685 | 0.591 | 0.516 | 0.410 |
| QAGS-XSUM | 0.521 | 0.427 | 0.653 | 0.547 | 0.576 | 0.472 | 0.537 | 0.472 | 0.406 | 0.343 | |
| AVG | 0.620 | 0.505 | 0.684 | 0.563 | 0.616 | 0.495 | 0.611 | 0.525 | 0.461 | 0.377 | |
| ELLIPSE | Overall | 0.589 | 0.463 | 0.651 | 0.520 | 0.520 | 0.403 | – | – | – | – |
| Cohesion | 0.510 | 0.395 | 0.579 | 0.453 | 0.475 | 0.364 | – | – | – | – | |
| Syntax | 0.516 | 0.399 | 0.590 | 0.462 | 0.494 | 0.379 | – | – | – | – | |
| Vocabulary | 0.516 | 0.402 | 0.569 | 0.447 | 0.520 | 0.406 | – | – | – | – | |
| Phraseology | 0.472 | 0.362 | 0.566 | 0.441 | 0.467 | 0.358 | – | – | – | – | |
| Grammar | 0.532 | 0.410 | 0.615 | 0.482 | 0.424 | 0.324 | – | – | – | – | |
| Conventions | 0.470 | 0.360 | 0.636 | 0.500 | 0.497 | 0.380 | – | – | – | – | |
| AVG | 0.515 | 0.399 | 0.601 | 0.472 | 0.485 | 0.373 | – | – | – | – | |
| HealthBench | Communication | 0.286 | 0.259 | 0.361 | 0.332 | 0.226 | 0.204 | – | – | – | – |
| Complex responses | 0.131 | 0.120 | 0.162 | 0.149 | 0.155 | 0.139 | – | – | – | – | |
| Context seeking | 0.484 | 0.437 | 0.565 | 0.518 | 0.317 | 0.285 | – | – | – | – | |
| Emergency referrals | 0.429 | 0.388 | 0.493 | 0.453 | 0.357 | 0.322 | – | – | – | – | |
| Global health | 0.438 | 0.398 | 0.436 | 0.398 | 0.340 | 0.306 | – | – | – | – | |
| Health data tasks | 0.314 | 0.286 | 0.584 | 0.535 | 0.355 | 0.323 | – | – | – | – | |
| Hedging | 0.154 | 0.140 | 0.483 | 0.447 | 0.160 | 0.147 | – | – | – | – | |
| AVG | 0.320 | 0.290 | 0.441 | 0.405 | 0.273 | 0.246 | – | – | – | – | |
| BiGGen | Grounding | 0.549 | 0.513 | 0.643 | 0.598 | 0.632 | 0.577 | – | – | – | – |
| Instruction following | 0.405 | 0.359 | 0.560 | 0.507 | 0.550 | 0.492 | – | – | – | – | |
| Multilingual | 0.418 | 0.342 | 0.427 | 0.365 | 0.474 | 0.408 | – | – | – | – | |
| Planning | 0.507 | 0.444 | 0.539 | 0.489 | 0.580 | 0.512 | – | – | – | – | |
| Reasoning | 0.453 | 0.408 | 0.533 | 0.496 | 0.542 | 0.498 | – | – | – | – | |
| Refinement | 0.363 | 0.320 | 0.577 | 0.513 | 0.514 | 0.468 | – | – | – | – | |
| Safety | 0.362 | 0.325 | 0.527 | 0.476 | 0.545 | 0.478 | – | – | – | – | |
| Theory of mind | 0.404 | 0.362 | 0.458 | 0.415 | 0.438 | 0.390 | – | – | – | – | |
| Tool usage | 0.442 | 0.394 | 0.465 | 0.414 | 0.466 | 0.424 | – | – | – | – | |
| AVG | 0.434 | 0.385 | 0.526 | 0.475 | 0.527 | 0.472 | – | – | – | – | |
| JudgmentBench | Substance | 0.082 | 0.067 | 0.114 | 0.099 | 0.073 | 0.059 | – | – | – | – |
| Structure | -0.002 | -0.001 | 0.098 | 0.091 | -0.037 | -0.030 | – | – | – | – | |
| Style | 0.047 | 0.037 | 0.010 | 0.001 | -0.030 | -0.025 | – | – | – | – | |
| Negative points | 0.051 | 0.041 | 0.048 | 0.039 | 0.003 | 0.003 | – | – | – | – | |
| AVG | 0.045 | 0.036 | 0.068 | 0.058 | 0.002 | 0.002 | – | – | – | – |
Spearman ρ and Kendall τ within groups of responses to the same input, averaged per category; AVG is the mean over a benchmark's categories. Bold marks the best judge in each row and underline the second best. G-Eval values are the ones reported in its paper, which covers SummEval and QAGS only.
Jev ranks best on average on five of the six benchmarks. RubriX stands out on two of the benchmarks G-Eval reports [10]: it has the highest Spearman correlation on QAGS-CNN (0.719, against 0.715 for Jev and 0.685 for G-Eval with GPT-4), matches G-Eval with GPT-4 on SummEval coherence (0.581 against 0.582), and is above G-Eval with GPT-3.5 on every SummEval dimension and both QAGS subsets, in both ρ and τ. (G-Eval numbers are the ones reported in its paper.)
JudgmentBench is hard for everyone: every judge's average correlation is close to zero (at most 0.068). Ordering the answers to the same legal task by their checklist and error items is not something any of these judges does reliably yet.
Latency that doesn't care how big your rubric is
To see how cost scales with the rubric, we timed one request at a time on dummy text with exact token counts: an 800-token response, a 30-token criterion, and level descriptions of a fixed length, with 110 judgments per setting. RubriX and the Qwen3-8B judge ran on one H100 with vLLM; Jev was called through its API, with an approximate network time of 232.6 ms subtracted.

With its level embeddings cached, RubriX stays at about 28 ms per judgment (27.5–28.3 ms) from 2 to 100 levels of 1,500 tokens each, a 150,000-token rubric at the top end. The Qwen3-8B judge grows from 108 ms to 1,241 ms at 20 levels, the most its 32k context can hold. Jev grows from 73 ms to 200 ms at 10 levels, the most its score questions accept.

The same holds for longer descriptions. As each of five levels grows from 10 to 3,000 tokens, RubriX stays at 28 ms while Jev slows from 44 ms to 211 ms and the Qwen3-8B judge from 32 ms to 516 ms. At 3,000 tokens per level that makes RubriX 7.7× faster than Jev and 18.8× faster than the Qwen3-8B judge. The first time RubriX meets a new rubric it has to embed the levels (the dashed line), which costs about as much as the Qwen3-8B judge, but that happens once per rubric.
Where RubriX shines, and where it's still growing
- Best with some data of your own. RubriX responds strongly to fine-tuning. On held-out rubrics from its training sources it is far ahead, while on unfamiliar benchmark formats it trails Jev on most sets. If you have a few labelled examples per criterion from your domain, fine-tuning takes a few hours on a single GPU (2.8 hours on one H100 for our model), and it is very cheap to run.
- Long or many-level rubrics. Because the levels are cached, adding levels or making them more precise costs nothing per judgment.
- It reads the response and the levels separately. The two meet only at the final similarity, so verdicts that depend on checking one specific fact against a level are harder for it than for a judge that reads both together.
- Capacity. RubriX uses an 8B encoder with small heads, which likely limits how far it generalises beyond the rubric styles it was trained on. A second epoch of training did not improve validation loss.
What's next
- Bigger base models, to see how much of the gap on unfamiliar rubric styles is a matter of size.
- More training data, and a closer look at how accuracy and ranking change as it grows.
- Domain-specific fine-tuning, comparing how it does on a domain's own rubrics against general ones.
- Rubrics with hundreds of levels, where an LLM judge needs a long-context model to even fit the rubric, while RubriX's cost stays flat.
Explore OptiX
OptiX is our platform for testing, evaluating and monitoring AI systems. It brings together:
- Security scans for jailbreaks, prompt injection and other adversarial inputs
- Multi-agent evaluation across complex, multi-step workflows
- Observability for AI systems in production
- Use-case evaluation for RAG, text-to-SQL, document parsing, entity extraction, summarisation and more
Want to try RubriX for evaluating your agents? Or have a use case where RubriX could fit and would benefit from fine-tuning? Check out optix.cobuildx.ai and get in touch with our team.
References
Benchmarks
- R. K. Arora et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775, 2025.
- S. Kim et al. The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. NAACL 2025. arXiv:2406.05761.
- S. A. Crossley et al. The English Language Learner Insight, Proficiency and Skills Evaluation (ELLIPSE) Corpus. International Journal of Learner Corpus Research, 9(2):248–269, 2023.
- A. R. Fabbri et al. SummEval: Re-evaluating Summarization Evaluation. TACL 9:391–409, 2021.
- JudgmentBench. https://huggingface.co/datasets/judgmentbench/JudgmentBench, 2026.
- A. Wang, K. Cho, M. Lewis. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. ACL 2020.
Models and methods
- Contrastive-LM. CLM: Contrastive language models, and the CLM-v0.1-8B checkpoint. https://github.com/Contrastive-LM/CLM
- TypeSafe AI. Jev model documentation. https://docs.typesafe.ai/models
- Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025.
- Y. Liu et al. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. EMNLP 2023.
- E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
- E. S. Epstein. A Scoring System for Probability Forecasts of Ranked Categories. Journal of Applied Meteorology 8(6):985–987, 1969.
- Z. Chen et al. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. ICML 2018.
- R. Groenendijk et al. Multi-Loss Weighting with Coefficient of Variations. WACV 2021.
- M. Bhandari et al. Re-evaluating Evaluation in Text Summarization. EMNLP 2020.