docs(jev): record probe results — score/noul confirmed, noul has no confidence (#7)
This commit is contained in:
1 parent
ac95ab1480
commit
2637b87ac8
1 file changed
+2
@@ -12,6 +12,8 @@ _Research 2026-10-27. Sources at the bottom. Vendor numbers (latency, price, cal
|
||||
- **Reported cost and speed (vendor):** about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free.
|
||||
- **Limits:** text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in `state`, so secrets must stay out.
|
||||
|
||||
**Verified 2026-10-27** (`TestProbeScoreAndNoulAgainstLiveAPI`, jev-1.14.0, 0.17 s, 466 tokens, one mixed call): `score` returns score/confidence/legend/probabilities as documented (severity 1.95 of 2, conf 0.93); `noul` returns only `noul` (0.94), **no confidence** — strategies 3 and 4 must threshold noul directly (docs suggest yes 0.7 / no 0.3, middle to review). Confidence is distribution concentration as warned: the split doc example (0.35) matches our probe's concentration reading.
|
||||
|
||||
**Crows today** uses one call with two `choice` questions (mode, task type) and a single global threshold of 0.7 (`jev.go`). So **custom labels are supported**, which answers spec open question 2: expectation checks and orchestration checks are just more questions.
|
||||
|
||||
## Strategies for accuracy
|
||||
|
||||
Reference in new issue
Block a user