6.7 KiB
6.7 KiB
Jev: Strategies for More Accurate and Engaging Crows
Research 2026-10-27. Sources at the bottom. Vendor numbers (latency, price, calibration) come from TypeSafe via secondary write-ups and haven't been checked against crows' own workload. Related: CHATROOM_SPEC.md, GROKBOT_NOTES.md.
What Jev can do (facts)
- Endpoint:
POST https://api.typesafe.ai/v1/systemone. A request hasmodel, astate(plain text or JSON), and namedquestions. All questions in a request are evaluated independently, in parallel, in one call. The context budget is reported as 64,000 tokens per request. - Three question types:
choice: one option from up to 255.criteriamaps option names to descriptions. Returnschoice,probabilitiesandconfidence.score: a position on an ordered scale of 2 to 10 levels (criteriais an array of level descriptions). Can land between levels. Returnsscore,legend,probabilitiesandconfidence.noul: yes/no. Returns the probability of yes, with noconfidence.
confidenceisn't the probability of being right. It measures how concentrated the probabilities are. Accuracy has to be measured on our own labeled data.- Reported cost and speed (vendor): about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free.
- Limits: text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in
state, so secrets must stay out.
Verified 2026-10-27 (TestProbeScoreAndNoulAgainstLiveAPI, jev-1.14.0, 0.17 s, 466 tokens, one mixed call): score returns score/confidence/legend/probabilities as documented (severity 1.95 of 2, conf 0.93); noul returns only noul (0.94), no confidence — strategies 3 and 4 must threshold noul directly (docs suggest yes 0.7 / no 0.3, middle to review). Confidence is distribution concentration as warned: the split doc example (0.35) matches our probe's concentration reading.
Crows today uses one call with two choice questions (mode, task type) and a single global threshold of 0.7 (jev.go). So custom labels are supported, which answers spec open question 2: expectation checks and orchestration checks are just more questions.
Strategies for accuracy
- Many narrow questions per call, not one big one. For each new message, send one state and ask several questions together: message kind (
choice), "does this create an expectation?" (noul), who it's addressed to (choiceover crows present, plusnone). One call, parallel answers. This also answers spec open question 5: Jev can tag the message kind. - Always include an escape option (
other,none,unclear) so Jev is never forced into a wrong label. Today'stask_typehasother;modedoesn't. - Use
scorefor anything with degrees. Usenoulonly for crisp yes/no conditions (for example, "states a fact without a source"). Anoulof 0.5 means a coin flip, not "medium." For "how badly is this expectation missed," use ascorewith written levels. - Three bands per question, not one global threshold. High: the harness acts (sends the nudge, routes the message). Medium: log it and let the crow decide. Low: do nothing, or show the owner. Use stricter bands for anything that will become irreversible once real capabilities exist.
- Calibrate on our own labels. Log every verdict with its probabilities, then record what actually happened (owner approved or rejected, expectation later met or not). Use that to set each question's bands. The single 0.7 threshold today is a guess.
- Structure the state as JSON, trimmed. Send the expectation, the responsible crow, and only the relevant slice of the room, not the whole log. Never include
.envvalues or anything private. - Composite scores for orchestration (spec 7a). Jev can't explain itself, so break crow and model choice into narrow questions (task complexity
score, riskscore, persona fitchoice), then weight them in Go. Weights are visible and tunable, and the log shows exactly which answer drove a decision. - Jev checks crow output before it's merged. A few
noulquestions on eachdeliver: does it answer the request, does it state facts without sources, does it contradict an earlier message. This is the "claims sourced" check, and it would have caught the Wandering Inn narrator error before the owner did.
Strategies for engagement
- Who speaks next in the open flock. For an unaddressed message, ask Jev to pick which crow (or
none) should respond, with criteria written from each persona's backstory and strengths. This prevents the three-identical-answers problem from the Lab test and keeps the room lively without spam. Trust still gates who may speak unprompted. - Banter moments chosen by Jev instead of scripted triggers. Score "how worth reacting to is this moment" (1 to 5) and pick which persona has the most to say about it. Generate banter only above a threshold, still within the 2-per-list cap.
- Read the owner's intent, not their mood. A
choiceon each owner message: wants a result, wants to chat, wants to think out loud, wants a decision. Crows adapt length and banter to match. (Avoid scoring emotions; stick to what the owner seems to want from the room.) - Debate quality, only in opt-in debates. Score each turn: new point vs. restatement, concedes vs. holds, evidence offered. Restaters get a nudge; the judge crow gets the scores.
- Celebrate real wins. A
noulon deliveries: "closes something the owner asked for." That drives the Duolingo-style moment (sparkles, level-up) only when it's earned.
Guardrails for using Jev
- Every Jev call goes through
Flock.classify, so budgets, the crow cap, timeouts and the flight log apply (existing rule). - Log question, answer, probabilities and the action taken. Jev can't explain, so the log is the explanation.
- Batch questions per message to keep calls and latency down. Measure real latency from Utah.
- Jev never blocks work: if it errors or is unsure, crows carry on (existing rule).
Suggested first experiment (RITE)
Add two questions to every message in one room: message kind and who should respond (crows present plus none). Don't act on them yet. Log the answers next to what actually happened for a few sessions, then decide on bands.