index / evals / jev
jev vs llms, closed and open
sep 2026 · eval
A government RFP hides a few hundred must-do rules in a hundred pages of legal text. Miss one and your bid can be tossed. I marked every rule in three real RFPs by hand, then asked five AI models to find them. The winner can't write a sentence. It reads and picks, for two cents a document.
What Jev is. Jev is a sorter: you hand it some text and a fixed set of slots, and it drops the text in one slot and tells you how sure it is. It can't write a sentence. the write-up.
macro-F1 on requirements found · cost to read one RFP
| model | kind | F1 | per RFP |
|---|
| Jev 1.13 TypeSafe AI | decision model | 0.64 | $0.02 |
|---|
| Kimi K2.6, fine-tuned | open | 0.57 | $3.30 |
|---|
| Qwen3.5-4B, fine-tuned | open | 0.55 | $0.22 |
|---|
| Inkling, fine-tuned | open | 0.53 | ~$1.20 |
|---|
| Claude Opus 4.8, few-shot | closed | 0.52 | $4.70 |
|---|
higher is better. 1.0 would mean every rule found and nothing wrongly flagged.
open a past run
the scored RFPs, plus one more from the july run (no answer key, so it gets a bid read but no score). each opens a saved Jev read against a sample firm's documents.
what was the job
the job
When a state agency buys something, it publishes an RFP: a long document listing everything a vendor must do, have, or send. I labeled three Texas RFPs by hand and marked all 438 rules. That list is the answer key.
the score
Each model read the same three documents and listed the rules it found. The score goes up for every real rule it catches and down for every sentence it flags that isn't one. Nobody came close to 1.0.
what happened
Jev caught 354 of the 438 rules, far more than anyone else, and won on all three RFPs. It also over-flags: about half of what it picks isn't a rule. The LLMs were pickier but missed a lot more.
the catch
I built the answer key one sentence at a time, and Jev picks one sentence at a time, so the test suits it. When it said 50 to 80% sure, it was right 18% of the time.
model by model
- Jev 1.13 0.64 · $0.02 a document
- Doesn't write at all. It reads each sentence and answers a fixed question: is this a rule, and what kind? Found 81% of the rules.
- Kimi K2.6, fine-tuned 0.57 · $3.30
- A big open model trained on 11 practice RFPs. The best of the LLMs.
- Qwen3.5-4B, fine-tuned 0.55 · $0.22
- The small one. After the same training it nearly matched Kimi at a fifteenth of the price.
- Inkling, fine-tuned 0.53 · ~$1.20
- Trained the same way. Uneven across the three RFPs.
- Claude Opus 4.8, few-shot 0.52 · $4.70
- No training, just a few worked examples. The most careful, and the most expensive.
the rules
Jev answers. Code decides.
a missing qualification is a no
Years in business, past contracts, a certification you don't hold. One gap at 70% confidence ends it.
paperwork is a checklist
You can order a certificate by Friday. You can't order five years of history.
coverage decides the rest
Half the required work covered is a bid. Less is a closer look.
where Jev breaks: its confidence. Answers it gave 50 to 80% were right 18% of the time.
← all evals · the write-up