From 2637b87ac837e6f667e6736af1f08e17b8b65f5e Mon Sep 17 00:00:00 2001 From: Smokey <150986568+jacisjake@users.noreply.github.com> Date: Tue, 27 Oct 2026 23:42:40 -0600 Subject: [PATCH] =?UTF-8?q?docs(jev):=20record=20probe=20results=20?= =?UTF-8?q?=E2=80=94=20score/noul=20confirmed,=20noul=20has=20no=20confide?= =?UTF-8?q?nce=20(#7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- JEV_STRATEGIES.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/JEV_STRATEGIES.md b/JEV_STRATEGIES.md index 721c8cb..ca1f2df 100644 --- a/JEV_STRATEGIES.md +++ b/JEV_STRATEGIES.md @@ -12,6 +12,8 @@ _Research 2026-10-27. Sources at the bottom. Vendor numbers (latency, price, cal - **Reported cost and speed (vendor):** about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free. - **Limits:** text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in `state`, so secrets must stay out. +**Verified 2026-10-27** (`TestProbeScoreAndNoulAgainstLiveAPI`, jev-1.14.0, 0.17 s, 466 tokens, one mixed call): `score` returns score/confidence/legend/probabilities as documented (severity 1.95 of 2, conf 0.93); `noul` returns only `noul` (0.94), **no confidence** — strategies 3 and 4 must threshold noul directly (docs suggest yes 0.7 / no 0.3, middle to review). Confidence is distribution concentration as warned: the split doc example (0.35) matches our probe's concentration reading. + **Crows today** uses one call with two `choice` questions (mode, task type) and a single global threshold of 0.7 (`jev.go`). So **custom labels are supported**, which answers spec open question 2: expectation checks and orchestration checks are just more questions. ## Strategies for accuracy