jac/so-many-crows
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(jev): record probe results — score/noul confirmed, noul has no confidence (#7)

Browse Source
This commit is contained in:
Smokey committed 2026-10-27 23:42:40 -06:00
1 parent ac95ab1480
commit 2637b87ac8
1 file changed
+2
Download Patch File Download Diff File Expand all files Collapse all files
JEV_STRATEGIES.md
+2
View File
@@ -12,6 +12,8 @@ _Research 2026-10-27. Sources at the bottom. Vendor numbers (latency, price, cal
- **Reported cost and speed (vendor):** about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free. - **Reported cost and speed (vendor):** about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free.
- **Limits:** text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in `state`, so secrets must stay out. - **Limits:** text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in `state`, so secrets must stay out.
**Verified 2026-10-27** (`TestProbeScoreAndNoulAgainstLiveAPI`, jev-1.14.0, 0.17 s, 466 tokens, one mixed call): `score` returns score/confidence/legend/probabilities as documented (severity 1.95 of 2, conf 0.93); `noul` returns only `noul` (0.94), **no confidence** — strategies 3 and 4 must threshold noul directly (docs suggest yes 0.7 / no 0.3, middle to review). Confidence is distribution concentration as warned: the split doc example (0.35) matches our probe's concentration reading.
**Crows today** uses one call with two `choice` questions (mode, task type) and a single global threshold of 0.7 (`jev.go`). So **custom labels are supported**, which answers spec open question 2: expectation checks and orchestration checks are just more questions. **Crows today** uses one call with two `choice` questions (mode, task type) and a single global threshold of 0.7 (`jev.go`). So **custom labels are supported**, which answers spec open question 2: expectation checks and orchestration checks are just more questions.
## Strategies for accuracy ## Strategies for accuracy