Quick, reliable decision AI goes open-source: benchmarking Clef on our serving stack
Jev made fast multiple-choice decisions with a confidence score the talk of the industry, but it can only be rented. Cloudflare's Clef is an open alternative. We ran it on a single consumer GPU and measured how fast it is, how often it is right, and whether its confidence means anything.
October 7, 2026
The short answer
A new kind of model has taken over the conversation in the last few weeks. Instead of writing text, it answers a multiple-choice question and tells you how confident it is: which team should handle this ticket? is this clause a termination right? how sure are you? TypeSafe's Jev set the standard in September. Clones followed within days; the public Decision Index leaderboard already lists 70 entries.
Jev is excellent, but it is a hosted service: you send your documents to someone else's servers. For confidential legal work, that is often unacceptable. So we asked the practical question: can you get Jev-style decisions on your own hardware?
Here is what we found, in four lines.
- Accuracy: on the kind of task these models are built for (sorting text into categories), the open model we tested, Cloudflare's Clef, matched or beat Jev on the same requests. On tasks that need reasoning or judgement, it did worse than Jev.
- Speed and context length: on one RTX 4090 a document of about 1,900 tokens takes 321 ms on our Clef and 113 ms on our Clef-flash, against 307 ms for Jev, 706 ms for hosted Clef and 933 ms for hosted Clef-flash (the hosted figures are round trips over the internet). Jev then stays nearly flat as prompts grow while ours grow steadily: at 32,000 tokens ours take 5.7 s (Clef) and 1.8 s (Clef-flash) against Jev's 559 ms. Jev stops at 32,000 tokens and the hosted Clef services at 65,536; our compressed KV cache holds 441,600 tokens, and we timed our Clef to 136,000 tokens, where it takes 33 s, and our Clef-flash to 131,000 tokens, where it takes 10 s.
- Confidence: decision models provide a confidence score along with their answer and this is a key differentiator with traditional LLMs. In our comparison Clef's confidence scores were useful, with a better Brier score than Jev's, but not better calibrated than Jev's. We say what that does and does not show below.
- Cost: compressed to about 21 GB (9 GB for Clef-flash), the models run on one 24 GB consumer GPU, with no per-request fee and no document leaving your premises. Against Cloudflare's own hosted originals, our compressed models gave the same top answer on 98 per cent of requests for Clef and 98 per cent for Clef-flash, with accuracy within 0.01 on most datasets. Our server's default 4-bit KV cache, calibrated on the model, changed 0.9 per cent of Clef's answers against an uncompressed cache, and our Clef costs an estimated $0.036 per million input tokens (Clef-flash $0.012) against $0.09 to $0.24 hosted.
The four numbers behind those lines, from our own measurements:
98%
of answers identical to Cloudflare’s originals after compressing the weights to 4.5 bits and the KV cache to 4, for both Clef and Clef-flash
8×
faster than the hosted version on a 1.9k-token document (113 ms against 933 ms)
9 ms
spread between our slowest and fastest repeat at 32k tokens; hosted Clef ranged over 13 s
$0.012
per million input tokens on our card, against $0.09 to $0.24 hosted (an estimate; see the cost chart)
What "decision AI" actually is
A chatbot writes an answer. A decision model picks one, from a list you give it, and says how likely each option is. You send it some text (the "state") and some questions with fixed answers. It reads everything once and returns a probability for every option of every question: relevant, 97%; not relevant, 3%. No generated sentence, nothing to parse, and nothing it can invent.
Think of a thermostat rather than an essayist. A thermostat cannot write you a poem, but you can wire it to a heating system and trust it to answer one question quickly, thousands of times a day. That is the appeal for software that has to make many small judgements: route a support ticket, flag a clause, approve or escalate a request.
Why the confidence score is the whole point
Ordinary language models are poor at this. You can read a probability off the model's word choices (its "logprobs"), but those numbers are not trained to mean "how likely am I to be right". Jev's pitch is that its training teaches the model to report an honest number. Published descriptions call the method RLCD (reinforcement learning for calibrated decisions) and say it optimises against scoring rules that reward honest probabilities. The reward, the architecture and any calibration curves have not been released, so we can say what it is for but not how it works.
Cloudflare's Clef does something simpler to describe: it takes an existing open model, Qwen3.8-27B, post-trains it, and adds a small "head" that reads the model's internal state at each question and each option and turns them into one probability per option. Its model card does not describe the calibration training in detail. So, rather than guess, we measured it in two separate exercises: our own legal document review (next), and a public third-party benchmark (in the deep dive).
A worked example: reviewing a case's documents
Law firms already use this shape of tool for tabular review: documents are the rows, questions are the columns, and the model fills in each cell. The attraction is obvious: one document, many questions, one quick call. The risk is just as clear: a confident wrong cell can send a privileged document to the other side.
We built a test around the two questions every disclosure exercise asks of every document: is it relevant, and is it privileged?
Fifty synthetic documents, from one-line emails to a long operations log. We included some hard calls on purpose: legal advice copied to an outside PR firm, a lawyer copied on a routine note, drafts and memos written by counsel after the dispute began. Reasonable lawyers will disagree about several of these. The labels we used are ours, not a practising lawyer's, and we do not claim to settle the contested ones. We report what each model said and how sure it was. The review at the top of this page lets you step through every document with both models' answers.
What happened
Each document went in on its own with the same instructions and the same three questions: relevant?, privileged? (attorney-client, work product, or neither) and does it contain an admission that the equipment was run beyond its rating. We ran Clef on our serving stack on a single consumer GPU and Jev through OpenRouter, three rounds each.
The two models gave the same answers on 47 of the 50 documents. Against our working labels each matched on 46 (a slightly different four for each). The documents where they differ from each other, or from our label, are the ones we are least sure of ourselves.
Three documents where the models disagree with each other:
| Document | Question | Clef | Jev |
|---|---|---|---|
| Litigation hold notice from in-house counsel | privilege | work product (0.77) | not privileged (0.40) |
| A lawyer's commercial advice on replacing the winches | relevance | not relevant (0.54) | relevant (0.77) |
| Expert report commissioned by counsel | admission | no (0.57) | yes (0.63) |
Two documents where the models agree with each other but not with our label: a draft defence the firm sent to the client's in-house lawyer, and the firm's advice memo to the board. We had called both attorney-client. Both models said work product, which a lawyer could argue, since each is counsel's communication written for the litigation.
| Document | Clef | Jev |
|---|---|---|
| Draft defence email | work product 0.85, attorney-client 0.13 | work product 0.50, attorney-client 0.49 |
| Board advice memo | work product 0.83, attorney-client 0.15 | work product 0.62, attorney-client 0.34 |
We cannot tell you which answer is right. What the numbers show is a difference in manner. On the draft defence email Jev put 0.50 on work product and 0.49 on attorney-client, which reads as "this could be either". Clef put 0.85 on work product. If you think the document is genuinely both, Jev's answer is the more faithful one; if you think it is plainly one, Clef's may be. In a real review, a document a model is unsure about is the one you want a person to look at.
How many documents would go to a person? Suppose a document goes to a person whenever the model's weakest answer is under a confidence threshold. This needs no labels, only each model's own numbers.
| Threshold | Clef: documents to a person | Jev: documents to a person | The three disagreement documents flagged (Clef, Jev) |
|---|---|---|---|
| under 0.70 | 8 of 50 | 5 of 50 | 3 of 3, 2 of 3 |
| under 0.80 | 14 of 50 | 9 of 50 | 3 of 3, 3 of 3 |
| under 0.90 | 23 of 50 | 13 of 50 | 3 of 3, 3 of 3 |
At the same threshold Jev sends fewer documents to a person, so it clears more of them on its own. Whether that is a virtue depends on whether you agree with what it cleared, and we have not scored that.
Reading the claims properly
“Clef matches Jev.”
“Decision models are fast.”
“They cannot hallucinate.”
“The confidence is reliable.”
Questions people actually ask
▸What is Jev?
Jev is a "System One" model from TypeSafe: a model that takes some text and a set of multiple-choice questions and returns a probability for every option. It is offered as a hosted API, and as far as we know its weights are not available.
▸Can I self-host Jev?
Not today. TypeSafe sells it as a service. Open alternatives exist, and Cloudflare's Clef is one of the first to approach Jev's results on classification tasks.
▸Is Clef better than Jev?
It depends on the job. On intent classification and common-sense tasks in our runs it matched or beat Jev. On toxicity and sentence entailment it did not. For inputs longer than about 2,000 tokens, a hosted Jev is faster.
▸Is decision AI just a classifier?
Close. A classifier is trained for one fixed set of labels. A decision model takes the labels as part of the request, so the same model can answer a different set of questions on every call, and it also tells you how sure it is.
▸Can a decision model do multi-step reasoning?
No, and it is not designed to. In our tests it did worse than Jev on the tasks that need judgement (toxicity and entailment). Use a decision model for the quick first pass, and a reasoning model or a person for the hard cases.
▸What does it need to run?
One GPU with about 24 GB of memory for the compressed Clef we tested.
Deep dive
The first half of this article needs no technical background. From here on, we show the machinery.
The three sections that follow on speed, accuracy and confidence come from a separate, third-party benchmark (the one Jev's own evaluators published). They are not the legal exercise above, and the two sets of results are not mixed.
The field in brief
Jev (TypeSafe) is hosted only: a 32,000-token limit and $0.042 per million input tokens, also sold through OpenRouter. Clef (Cloudflare, Apache-2.0) retrofits an open 27-billion-parameter model to answer in Jev's format; a smaller Clef-flash exists. The public Decision Index lists 70 entries, and OpenAI's Decisions API is in public beta with no published accuracy or calibration figures. All of them suit single-step classification and routing, not multi-step reasoning.
Five contenders, five questions
Two models of ours, running on arbi-serve on one RTX 4090, against the three hosted services we could reach through OpenRouter. Everything below is measured on the same requests; the one estimate, our cost, is labelled where it appears. The last row of the table is not a contender but a baseline: the plain Qwen3.8-27B with no decision head, which has an accuracy result and nothing else.
| Model | Where it runs | Longest prompt | What it is |
|---|---|---|---|
| Our compressed Clef | One RTX 4090, on arbi-serve | Timed to 136,000 tokens; its KV cache holds 441,600 | 4.5-bit weights (21 GB), 4-bit KV cache |
| Our compressed Clef-flash | One RTX 4090, on arbi-serve | Timed to 131,000 tokens | 4.5-bit weights (9 GB), 4-bit KV cache |
| Clef (hosted) | Cloudflare, via OpenRouter | 65,536 tokens | The original model; Apache-2.0 weights |
| Clef-flash (hosted) | Cloudflare, via OpenRouter | 65,536 tokens | Smaller model, built on Qwen3.5 9B |
| Jev 1.13 (hosted) | TypeSafe, via OpenRouter | 32,000 tokens | Closed model, rented only |
| Qwen3.8-27B, untrained (prompted, same quantisation) | On arbi-serve, in compatibility mode | Accuracy only, not timed | The plain open model Clef was built from, with no decision head: 4.5-bit weights, uncompressed KV cache. Answers one question per request |
1. Does compression change the answers?
Compressing both models (weights to 4.5 bits, KV cache to 4) changed 2 in 100 answers, and accuracy stayed within 0.01 of the hosted original on every dataset.
- Our compressed Clef
- Clef (hosted)
- Our compressed Clef-flash
- Clef-flash (hosted)
The same 200 requests per dataset went to each. Each dark dot sits on its lighter twin because the scores are nearly equal. The figures on the right are how often the two gave the same top answer, request by request: first for Clef (1,564 of 1,600 overall), then for Clef-flash (1,544 of 1,572; hosted Clef-flash answered only 172 of the 200 HellaSwag requests before its rate limit). The least agreement is ToxiGen, where the models differ on cases near their decision boundary and accuracy barely moves.
2. How fast is it?
On the short prompts most documents are, our own server answers 8 times faster than the hosted version of the same model. Only Jev stays quick as prompts grow, and only ours takes prompts beyond 65,000 tokens.
Time for one decision, from about 1,900 to 136,000 tokens, with the vertical axis on a log scale so a tenth of a second and half a minute fit on one chart. Median of three requests per point, five seconds apart, each starting with a unique line so no prefix cache can help; a faint bar marks a point whose repeats differed widely. Ours is the server's own time on one RTX 4090; the others are round trips over the internet through OpenRouter, so they include the network. Jev stops at its 32,000-token limit; ours are timed to 136,000 tokens, past the 65,536 the hosted services accept. The plain Qwen3.8-27B baseline is not here: it answers one question per request, so its time would not be comparable.
3. Is it the same every time?
Run the same request again on our server and it takes the same time to within 9 ms. On the hosted service the same request took anywhere from 7.5 s to 20 s.
Three repeats of one prompt of about 32,000 tokens on each deployment: the dot is the median and the bar runs from the fastest to the slowest repeat. A shared hosted service has queues and replicas we cannot see, and we do not know which caused the spread; a server you run yourself has neither. On a hosted Clef request of about 1,900 tokens we also saw a single cold start take 79 seconds.
4. What does it cost?
Running it ourselves costs about 8 times less per token than renting the same small model, and about the same as Jev’s list price for the large one.
- Our compressed Clef-flash$0.012
- Our compressed Clef$0.036
- Jev 1.13 (hosted)$0.042
- Clef-flash (hosted)$0.090
- Clef (hosted)$0.24
Dollars per million input tokens. Hosted prices are OpenRouter's list prices. Ours are estimates for the graphics card alone: a $1,800 card written off over three years ($0.07 an hour), 450 W at $0.15 per kWh while busy ($0.07 an hour), and the throughput we measured at 22,000 tokens (about 5,800 tokens a second for Clef and 18,000 for Clef-flash), assuming the card is busy one hour in ten. If it were never idle the figures would be about $0.007 and $0.002. They leave out the rest of the server, staff and cooling. The plain Qwen3.8-27B baseline has no price here: we ran it as an accuracy baseline and did not cost it.
5. Who is right where?
The open models beat Jev at sorting text into categories (Banking77: 16 points ahead) and fall behind where judgement matters (ToxiGen: 9 points behind).
- Jev 1.13 (hosted)
- Qwen3.8-27B, untrained (prompted, same quantisation)
- Our compressed Clef
- Clef (hosted)
- Our compressed Clef-flash
- Clef-flash (hosted)
Accuracy on 200 requests per dataset from the benchmark Jev's own evaluators published, sorted by how far the better of our two models is ahead of Jev (the figure on the right). Each hosted original sits almost exactly on its compressed twin. The grey dot is the plain Qwen3.8-27B with no decision head: it answers by reading next-token probabilities over the option labels, one request per question, so it appears here and not in the speed or cost charts. Intervals on 200 requests are roughly plus or minus 0.05 to 0.07, so only the larger gaps mean much.
Speed against input length
Published latencies rarely say for what input size: Cloudflare's model card gives 209 ms for Clef, 38.8 ms for Clef-flash and 524 ms for Jev, the Decision Index 102 ms for Clef, and none states a prompt length. We timed all five on the same prompts, from about 1,900 tokens up to each service's limit (ours to 136,000 tokens, the hosted Clef services to about 62,000, Jev to 32,000): median of three requests, five seconds apart, each starting with a unique line so no prefix cache can answer part of it.
| Prompt (tokens) | Ours: Clef | Ours: Clef-flash | Hosted Clef | Hosted Clef-flash | Jev |
|---|---|---|---|---|---|
| about 1,900 | 321 ms | 113 ms | 706 ms | 933 ms | 307 ms |
| about 7,300 | 1.2 s | 396 ms | 1.8 s | 1.2 s | 317 ms |
| about 12,100 | 2.0 s | 652 ms | 2.2 s | 1.1 s | 367 ms |
| about 20,800 | 3.6 s | 1.2 s | 4.4 s | 1.3 s | 433 ms |
| about 22,450 | 3.9 s | 1.2 s | 4.4 s | 2.1 s | 444 ms |
| about 26,000 | 4.6 s | 1.5 s | 4.8 s | 2.1 s | 454 ms |
| about 32,000 | 5.7 s | 1.8 s | 19 s | 2.0 s | 559 ms |
| about 36,800 | 6.7 s | 2.1 s | 7.2 s | 2.3 s | over its limit |
| about 45,500 | 8.5 s | 2.8 s | 23 s | 3.3 s | over its limit |
| about 62,200 | 12 s | 4.0 s | 28 s | 4.9 s | over its limit |
| about 70,300 | 14 s | 4.5 s | over its limit | over its limit | over its limit |
| about 95,000 | 21 s | 6.6 s | over its limit | over its limit | over its limit |
| about 131,000 | 32 s | 10 s | over its limit | over its limit | over its limit |
The untrained Qwen3.8-27B baseline of the next section is not timed here: it answers one question per request, so its time would not be comparable. Ours are the server's own time on one RTX 4090 with arbi-serve's default compressed KV cache; the hosted ones are round trips over the internet through OpenRouter and include the network. Jev stops at its 32,000-token limit. Read the table as shapes. Jev is nearly flat, adding about 10 ms per thousand tokens between 12,000 and 32,000. Our Clef-flash adds about 60 ms, hosted Clef-flash about 42 ms, our Clef about 186 ms and hosted Clef about 827 ms over the same range, and hosted Clef then jumps and turns erratic beyond about 26,000 tokens. At short lengths a model on your own card has no network to cross, which is most of why ours is faster there; Jev's per-token cost is a fraction of ours, and we do not know how, and we will not guess.
Accuracy on the same requests
We used the public benchmark Jev's own evaluators released: it rebuilds identical requests from public datasets and ships Jev's recorded answers. We ran 200 requests from each of eight datasets through all five models, and through a sixth row that is not a service: the plain Qwen3.8-27B, with no decision head, as a baseline.
| Dataset | Ours: Clef | Ours: Clef-flash | Hosted Clef | Hosted Clef-flash | Jev | Qwen3.8-27B, untrained (prompted) |
|---|---|---|---|---|---|---|
| Banking77 (77 intents) | 0.940 | 0.955 | 0.945 | 0.955 | 0.795 | 0.800 |
| CLINC150 (150 intents) | 0.965 | 0.995 | 0.960 | 0.995 | 0.875 | 0.845 |
| HellaSwag (common sense) | 0.995 | 0.990 | 0.995 | 0.994 | 0.965 | 0.935 |
| BoolQ (yes or no over a passage) | 0.940 | 0.945 | 0.945 | 0.935 | 0.935 | 0.925 |
| MMLU (general knowledge, multiple choice) | 0.910 | 0.945 | 0.920 | 0.950 | 0.900 | 0.850 |
| ANLI (does one sentence follow from another) | 0.620 | 0.530 | 0.610 | 0.535 | 0.685 | 0.650 |
| ToxiGen (toxicity) | 0.745 | 0.830 | 0.755 | 0.820 | 0.915 | 0.865 |
| SST-5 (sentiment, five levels) | 0.530 | 0.565 | 0.540 | 0.560 | 0.575 | 0.495 |
Intervals on 200 requests are roughly plus or minus 0.05 to 0.07, so only the larger gaps mean much. The pattern: both open models are well ahead of Jev at sorting text into categories (Banking77, CLINC150) and behind it where judgement is needed (ToxiGen, ANLI), the same boundary that applies to Jev itself. The smaller Clef-flash scores as well as our Clef or better on 6 of the eight datasets and loses clearly only on ANLI, so on this benchmark the extra 18 billion parameters buy little. Each hosted original is within a few hundredths of its compressed twin on every dataset, which the next section checks request by request.
The untrained baseline. The last column is Qwen3.8-27B, the open model Clef was built from, with no decision head and no training for this format, on the same 4.5-bit weights and the same 200 requests. We ran it through arbi-serve's compatibility mode, which answers each question by reading the model's next-token probabilities over the option labels, one request per question; its speed is therefore not comparable with the decision models' single pass, and it appears in the accuracy and confidence results only. Its mean accuracy is 0.796, against 0.831 for Jev, 0.831 for our Clef and 0.844 for our Clef-flash. It trails most where the trained head has the most to do: the 77- and 150-way intent sets (Banking77 0.800 against 0.940 and 0.955 for ours, level with Jev's 0.795; CLINC150 0.845 against 0.965 and 0.995), and on HellaSwag and MMLU. It is ahead of both Clef models where they trail Jev: ToxiGen (0.865 against 0.745 and 0.830) and ANLI (0.650 against 0.620, within the noise, and 0.530). So the training and the head bring clear gains on classification, and on those two datasets we see none; we have not tested why.
Does our compressed Clef match Cloudflare's hosted one?
Compression is only worth having if the answers survive it. We sent the same benchmark requests to each of our compressed models (EXL3 weights at about 4.5 bits, 4-bit KV cache, on our serving stack) and to Cloudflare's hosted original through OpenRouter, and compared the answers request by request.
| Dataset | Clef: same top answer | Clef accuracy, ours / hosted | Clef-flash: same top answer | Clef-flash accuracy, ours / hosted |
|---|---|---|---|---|
| Banking77 | 99.5% | 0.940 / 0.945 | 100.0% | 0.955 / 0.955 |
| CLINC150 | 98.5% | 0.965 / 0.960 | 100.0% | 0.995 / 0.995 |
| HellaSwag | 100.0% | 0.995 / 0.995 | 100.0% | 0.990 / 0.994 |
| BoolQ | 99.5% | 0.940 / 0.945 | 99.0% | 0.945 / 0.935 |
| MMLU | 98.5% | 0.910 / 0.920 | 98.5% | 0.945 / 0.950 |
| ANLI | 97.0% | 0.620 / 0.610 | 96.0% | 0.530 / 0.535 |
| ToxiGen | 91.0% | 0.745 / 0.755 | 96.0% | 0.830 / 0.820 |
| SST-5 | 98.0% | 0.530 / 0.540 | 96.5% | 0.565 / 0.560 |
Over all 1,600 requests our Clef gave the same top answer as the hosted Clef 97.8 per cent of the time (1,564 of 1,600), and our Clef-flash matched the hosted Clef-flash 98.2 per cent (1,544 of 1,572; the hosted Clef-flash answered only 172 of the 200 HellaSwag requests before its rate limit, so that row compares 172). The least agreement is ToxiGen, where the models differ on cases near their decision boundary and accuracy barely moves. Our models also run with arbi-serve's default compressed KV cache (4-bit keys and values, calibrated on the model) in place of an uncompressed one: for Clef that gave the same top answer as the uncompressed cache on 1,585 of 1,600 requests (99.1 per cent), with mean accuracy 0.831 against 0.830 and the same calibration. For Clef-flash it was 1,596 of 1,600 (99.8 per cent). Compression, in our measurements, costs nothing we can detect on this benchmark.
What Cloudflare changed
Clef is not simply a head on an untouched model. Comparing its checkpoint with Qwen3.8-27B tensor by tensor: the shapes are identical, the multi-token-prediction layers are dropped, and 186 of the 1,184 remaining tensors changed, all of them projection matrices (the attention and MLP weights). Embeddings, the output layer and every normalisation tensor are byte-identical. Where a tensor changed, it moved by 4 to 22 per cent of its size, 8 per cent at the median, so the backbone was fine-tuned in earnest and not lightly touched. And the starting point, Qwen3.8-27B, is already Qwen's post-trained chat model, so this is a second round of tuning on top of a tuned model.
Does the confidence mean anything?
| Model | Accuracy (mean of eight) | Calibration error (lower is better) | Brier score (lower is better) | Ranks right above wrong (AUROC, higher is better) | Accuracy on its 80 per cent most confident (higher is better) |
|---|---|---|---|---|---|
| Jev | 0.831 | 0.080 | 0.175 | 0.852 | 0.934 |
| Ours: Clef | 0.831 | 0.089 | 0.151 | 0.861 | 0.919 |
| Hosted Clef | 0.834 | 0.086 | 0.150 | 0.861 | 0.918 |
| Ours: Clef-flash | 0.844 | 0.083 | 0.148 | 0.827 | 0.911 |
| Hosted Clef-flash | 0.843 | 0.080 | 0.145 | 0.823 | 0.914 |
| Qwen3.8-27B, untrained (prompted) | 0.796 | 0.091 | 0.218 | 0.824 | 0.900 |
Same 200-request samples. Calibration error and Brier score are averaged over the seven datasets where the harness computes them (SST-5 is scored on accuracy only), and the last two columns use the same seven datasets and each model's top probability. The Qwen row is the untrained Qwen3.8-27B in arbi-serve's compatibility mode: its confidence is its next-token probability over the permitted options, not trained for calibration. For Banking77 and CLINC150 that probability is read in two steps and drops the last 1 per cent of the first step's probability, and we did not keep how much probability it put outside the options. What follows is modest:
- A chat model's own next-token probabilities are a usable but weaker confidence. On the same requests the untrained Qwen's top probability ranks right answers above wrong ones with an AUROC of 0.82, level with the two Clef-flash heads (0.82 to 0.83) and a little below Clef and Jev (0.85 to 0.86). Answering only its 80 per cent most confident requests lifts its accuracy to 0.900, against 0.934 for Jev. Its calibration error is the same size as the others' (0.091 against 0.080 to 0.089), but its Brier score is the worst of the six (0.218), mostly because it is wrong more often. At the top it is slightly overconfident: of its 412 answers at 99 per cent or more, 97.3 per cent were right (Jev: 522 answers, 98.1 per cent).
- Both Clef models have a better (lower) Brier score than Jev, 0.150 and 0.145 for the hosted ones, 0.151 and 0.148 for ours, against 0.175, mostly because they are more accurate. Their calibration error, which isolates confidence from accuracy, is no better than Jev's (0.089 and 0.083 for ours, 0.080 for Jev). On ANLI our Clef was overconfident (average confidence 0.79 against 0.62 accuracy), and so was our Clef-flash (0.73 against 0.53).
- Compressing the models did not change their calibration in any way we can see: each hosted original is within 0.003 of its compressed twin on both measures.
- 200 requests is too few to measure calibration well (the error is biased upward at this size), and several datasets have fewer than ten mistakes for the Clef models (HellaSwag, CLINC150), so their per-dataset AUROC is noisy. Treat the table as direction, not a ruling, and gaps under about 0.03 as noise.
- The
confidencefield is not the same number on every service. On our server, and in compatibility mode, it is the probability of the winning option. The hosted Clef and Clef-flash return a different number there on every multiple-choice answer in these runs, and Jev's matches the winning probability on 45 per cent of them. Every confidence measure above therefore uses the winning probability; ranking by each service's own field instead changes the AUROC by at most 0.001.
How a decision comes out of one forward pass
The prompt. A request is a state and a list of questions, each with a list of options. We lay these out as a single text, with each question and each option in its own marked span of tokens. The model reads the whole thing once.
The head. At the end of that pass we take the model's final hidden state at every question span and every option span. A small trained module, the "joint head", combines the question vector, the option vectors, and the model's own word-embedding rows for each option's tokens into one score per option. Scores within a question go through a softmax, so each question's probabilities add to one. Nothing is generated and nothing is sampled.
What this means for speed. Because everything is read in one pass, the cost is the cost of reading the prompt (prefill), not of writing an answer. That is why latency in our measurements grows with input length and barely with the number of questions: the extra questions only add a few tokens and a little head arithmetic. For tabular review, asking ten questions of a document costs little more than asking one.
How we make it fast on one card. The weights are compressed to roughly 4.5 bits each using an EXL3 quantisation, which brings the 27-billion-parameter model down to about 21 GB, and the KV cache (what the model remembers of the prompt while it reads it) is stored at 4 bits as well, so each 256-token page takes 4 MiB instead of 16 and the same card holds 441,600 tokens of context instead of about 47,000. Our serving engine, arbi-serve, runs the prompt through an integer-8 matrix-multiply path specialised for the weight format, together with the turbo-attn attention kernels. On an RTX 4090 that gives roughly 186 ms for every thousand tokens of input once prompts are long, about 5,379 tokens per second, and the compressed cache is as fast as the uncompressed one at every length we tried.
Does compression change the answers? On a fixed set of 54 hard questions with known answers (arithmetic, dates, choices, scores), the compressed model scored 94.4 per cent, and the uncompressed model split across four GPUs scored exactly the same, with the same three mistakes, all arithmetic or date reasoning where the model was confidently wrong.
Probabilities with known answers. For ten questions where the true probability is exact (a fair die, a card drawn from a deck), the model's answers were off by 0.07 to 0.08 on average, against 0.26 if it had said 50 per cent to everything. For questions where the text does not contain the answer, it leaned toward "no" rather than splitting its confidence.
Honest limits. One implementation, one consumer GPU, 200 requests per dataset, and a small hand-built hard set. Latency is the server's own time and excludes the network. Our own latency is measured up to 136,000 tokens for Clef and 131,000 for Clef-flash, well inside the 441,600 tokens the compressed KV cache holds; we make no claim beyond what we timed. We have not confirmed whether post-training gave Clef calibration comparable to Jev's.
Run it yourself
The comparison harness we used is the one Jev's evaluators published, plus a small script that sends the same requests to a server and records speed. The script is in the repository (scripts/bench/jev_suite) so anyone with access can repeat the comparison.
arbi-serve is in limited preview. To get access to the repository, request access and we will add your GitHub account.
Sources & further reading: the Jev benchmark and its published responses (arXiv 2609.37647 and its Zenodo data) · Cloudflare's Clef model card (Hugging Face, Cloudflare/clef) · the Decision Index leaderboard (Hugging Face Spaces, multimodalart/jev-decision-index) · OpenRouter's Jev 1.13 listing (32,000-token context, $0.042 per million input tokens) · OpenAI Decisions API documentation (public beta). © 2026 Arbitration City Ltd.