Jev vs. Laya: The Rematch — the Commercial "System One" Model Scored 70% Against My Family's Finances (and What That Win Costs)
The hype, again
The story of the month in decision-AI has two protagonists. On one side, Laya, the 421M open-source model I deployed on my own server and that lost 18 to 2 against my family's finances. On the other, Jev, the commercial TypeSafe AI model Laya compared itself against at launch — 7.8× slower and paid, according to the benchmark that made the underdog go viral.
When TypeSafe opened a free-access window to the Jev API, I didn't hesitate: same set of 20 real transactions, same rules, zero cherry-picking. A rematch gets measured on the same court.
For context on the system — the Family Wealth Command Center, NocoDB with my family's finances, the Hermes agent on Telegram, the custom 56-category taxonomy in Spanish — read the Laya article first. Everything that follows uses exactly the same methodology.
The experiment
Laya drowned in my taxonomy: 56 options exceeded its practical limit (~20 per question), so I had to implement the hierarchical dimension → category pattern, with one call per level. Jev doesn't have that problem: its choice primitive accepts up to 255 options. That made it possible to test something impossible with Laya: a single flat question with all 57 real categories, in Spanish, with a minimal state ("Family expense transaction in Peru. Concept/Amount/Currency/Account/Notes").
The rules didn't change: the same 20 "Expense" transactions I had already categorized myself (ground truth), without revealing the real category, measuring exact category match, latency and reported confidence. And since Jev also supports the hierarchical pattern, I ran it in both modes to compare apples to apples.
One rigor note before the numbers: two rows of the ground truth had inherited mojibake (the replacement character �, residue of a broken encoding) in the category name. I normalized them against the clean category table before measuring; without that correction the score would have been 12/20, corrected and fair it is 14/20. Uncomfortable moral: Jev returned the correct accented characters even when my own database no longer had them. And that mojibake turned out to be the symptom of an encoding bug in my write pipeline to NocoDB — now on the investigation list. Cheap benchmarks always pay interest.
The results
| Configuration | Exact-category accuracy | Dimension accuracy | Avg. latency | Mean confidence on hits | Mean confidence on misses | Cost per decision |
|---|---|---|---|---|---|---|
| Laya zero-shot | 10% (2/20) | n/a | ~900 ms (local CPU) | 0.72–0.78 | ~0.3 | $0 |
| Laya few-shot | 5% (1/20) | n/a | ~1000 ms (local CPU) | n/a | n/a | $0 |
| Jev flat | 70% (14/20) | 80% (16/20) | ~599 ms (547–718) | 0.90 | 0.34 | ~$0.0004 |
| Jev hierarchical | 50% (10/20) | 60% (12/20) | ~1169 ms | 0.96 | 0.53 | ~$0.0008 |
Seventy percent. Seven times Laya's accuracy on the same task, at two thirds of the latency, with a trivially simpler architecture: one call instead of two.
And look at the row that's easy to skip: the hierarchical mode — the one Laya forced me to use — costs Jev 20 accuracy points, doubles its latency and, worse, produces high-confidence errors (misses reported at 0.91 and 0.97 confidence). If anyone ever doubts that structured-question design matters as much as or more than the model, this natural experiment is the evidence.
The actionable finding: a calibration you can actually operate
With Laya I had already been impressed by one thing: the model missed a lot, but it admitted it (~0.3 confidence on misses). Jev takes that idea into productive territory. With a confidence threshold at 0.7, flat mode splits into two clean zones:
- 12 decisions auto-accept → 100% correct.
- 8 decisions escalate to a human → that's where the 6 real errors live, plus 2 low-confidence hits.
In production terms: today, with no fine-tuning, with not a single training example, I can automate 60% of categorization with perfect precision and route the rest to Hermes or to myself. That is exactly the act-or-escalate pattern these models are designed for. The contrast with hierarchical mode is abysmal: there the calibration breaks, and a miss at 0.97 confidence is the most dangerous pattern in automation — it degrades your data convinced it's right.
The 6 misses: data ambiguity, not model weakness
| Concept | Real category | Jev predicted | Conf. |
|---|---|---|---|
| Boss's birthday lunch | Individual Outings / Dad's Friends | Restaurants & Family Outings | 0.63 |
| Phone screen protector | Home Goods & Furniture | Home Maintenance & Repairs | 0.14 |
| Breakfast at the home market | Bakery & Daily Shopping | Fresh Market | 0.49 |
| Plin to Ruth | Fresh Market | Mom's Pension / Support | 0.40 |
| Web/app transfer | Home Maintenance & Repairs | Emergency Fund | 0.11 |
| Plin to Gustavo | Snacks / Home Treats | Individual Outings / Dad's Friends | 0.26 |
All six errors share a signature: P2P transfers ("Plin to Ruth", "Plin to Gustavo") and vague concepts ("web/app transfer"). Without context, "Plin to Ruth" could be groceries, a pension or a loan — I know which because I know Ruth; the model only sees the string. None of these gets fixed by a bigger model: they get fixed with more signal in the data (notes, counterparty, per-account rules). Hermes, my current LLM, faces exactly the same challenge with these cases.
And the hits at 1.00 confidence tell the other half of the story: "Compra de fruta" (fruit purchase) → Fresh Market; "Compra de abarrotes" (staples) → Weekly Supermarket; "Compra en Pastipan" → Bakery & Daily Shopping (local Peruvian shop, correct); "Compra de polos" (T-shirts) → Dad's Clothing & Shoes. That last one is my favorite: it understands that the T-shirts are dad's, not the whole family's — the model captured the personal scope of a taxonomy it had never seen.
The real trade-off: privacy, not price
Here's the part the "70%" headline doesn't tell. Jev is a closed, hosted API: every transaction of my family's finances leaves my LAN for TypeSafe AI's servers. Laya, for all its 10%, never took a single byte out of my house. That is the real decision in this article, and it has nothing to do with accuracy.
Cost, on the other hand, is almost a red herring: ~$0.0004 per decision. At five expenses a day — my real volume — that's ~$0.06 a month. Price is not what tips the scale.
What TypeSafe AI sells — founded by Diogo Almeida (ex-OpenAI), launched on September 15 with a $40M seed led by DCVC — is an API with an SLA and clean primitives: choice (up to 255 options), score (2–10 scale), noul (yes/no probability), 64k tokens per request, 1,200 req/min, at $0.042 per million input tokens with output free. Its flagship claim — 67.8% agreement with GPT-6/Claude — runs on its own evaluation set, with vendor methodology. That's why this blog always measures on its own data, even if it's 20 rows.
The architecture I'm taking with me
- Flat Jev with a 0.7 threshold auto-accepts; below the threshold, it escalates to Hermes or a human. 60% automation with perfect precision from day one.
- The 6 failure patterns get attacked with signal, not model: mandatory notes on P2P transfers, per-account rules, and eventually enough history that vague concepts stop being vague.
- Laya stays deployed on my server. When the history is enough for fine-tuning, I re-evaluate with the same script and the same set. The rematch of the rematch, with a date.
The lessons
- The viral benchmark measures the benchmark's distribution; your real problem is a different distribution. Yes, again.
- Sometimes what unlocks the simple architecture isn't the model but the product's capacity: "up to 255 options" wiped out in one stroke the hierarchy that cost Jev 20 accuracy points and Laya its entire life.
- Calibration is the feature. Again. A model that misses at 0.14 confidence is operable; one that misses at 0.97 is a silent incident.
- The real trade-off wasn't in the results table: it was in where your data sleeps.
- Cheap experiments pay real bugs: this benchmark exposed an encoding bug in my own pipeline, before it corrupted more data.
The full stack — NocoDB, FastAPI, Hermes, and now the Jev classifier — runs in (and partly from) my homelab in Peru. The benchmark script and the results are available for anyone who wants to replicate the experiment. If you also measured Jev or Laya against a real use case, I'd love to read your numbers.
