so-many-crows/JEV_STRATEGIES.md

# Jev: Strategies for More Accurate and Engaging Crows

_Research 2026-10-27. Sources at the bottom. Vendor numbers (latency, price, calibration) come from TypeSafe via secondary write-ups and haven't been checked against crows' own workload. Related: `CHATROOM_SPEC.md`, `GROKBOT_NOTES.md`._

## What Jev can do (facts)
- **Endpoint:** `POST https://api.typesafe.ai/v1/systemone`. A request has `model`, a `state` (plain text or JSON), and named `questions`. All questions in a request are evaluated independently, in parallel, in one call. The context budget is reported as 64,000 tokens per request.
- **Three question types:**
  - `choice`: one option from up to 255. `criteria` maps option names to descriptions. Returns `choice`, `probabilities` and `confidence`.
  - `score`: a position on an ordered scale of 2 to 10 levels (`criteria` is an array of level descriptions). Can land between levels. Returns `score`, `legend`, `probabilities` and `confidence`.
  - `noul`: yes/no. Returns the probability of yes, with no `confidence`.
- **`confidence` isn't the probability of being right.** It measures how concentrated the probabilities are. Accuracy has to be measured on our own labeled data.
- **Reported cost and speed (vendor):** about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free.
- **Limits:** text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in `state`, so secrets must stay out.

**Verified 2026-10-27** (`TestProbeScoreAndNoulAgainstLiveAPI`, jev-1.14.0, 0.17 s, 466 tokens, one mixed call): `score` returns score/confidence/legend/probabilities as documented (severity 1.95 of 2, conf 0.93); `noul` returns only `noul` (0.94), **no confidence** — strategies 3 and 4 must threshold noul directly (docs suggest yes 0.7 / no 0.3, middle to review). Confidence is distribution concentration as warned: the split doc example (0.35) matches our probe's concentration reading.

**Crows today** uses one call with two `choice` questions (mode, task type) and a single global threshold of 0.7 (`jev.go`). So **custom labels are supported**, which answers spec open question 2: expectation checks and orchestration checks are just more questions.

## Strategies for accuracy
1. **Many narrow questions per call, not one big one.** For each new message, send one state and ask several questions together: message kind (`choice`), "does this create an expectation?" (`noul`), who it's addressed to (`choice` over crows present, plus `none`). One call, parallel answers. This also answers spec open question 5: Jev can tag the message kind.
2. **Always include an escape option** (`other`, `none`, `unclear`) so Jev is never forced into a wrong label. Today's `task_type` has `other`; `mode` doesn't.
3. **Use `score` for anything with degrees.** Use `noul` only for crisp yes/no conditions (for example, "states a fact without a source"). A `noul` of 0.5 means a coin flip, not "medium." For "how badly is this expectation missed," use a `score` with written levels.
4. **Three bands per question, not one global threshold.** High: the harness acts (sends the nudge, routes the message). Medium: log it and let the crow decide. Low: do nothing, or show the owner. Use stricter bands for anything that will become irreversible once real capabilities exist.
5. **Calibrate on our own labels.** Log every verdict with its probabilities, then record what actually happened (owner approved or rejected, expectation later met or not). Use that to set each question's bands. The single 0.7 threshold today is a guess.
6. **Structure the state as JSON, trimmed.** Send the expectation, the responsible crow, and only the relevant slice of the room, not the whole log. Never include `.env` values or anything private.
7. **Composite scores for orchestration (spec 7a).** Jev can't explain itself, so break crow and model choice into narrow questions (task complexity `score`, risk `score`, persona fit `choice`), then weight them in Go. Weights are visible and tunable, and the log shows exactly which answer drove a decision.
8. **Jev checks crow output before it's merged.** A few `noul` questions on each `deliver`: does it answer the request, does it state facts without sources, does it contradict an earlier message. This is the "claims sourced" check, and it would have caught the Wandering Inn narrator error before the owner did.

## Strategies for engagement
9. **Who speaks next in the open flock.** For an unaddressed message, ask Jev to pick which crow (or `none`) should respond, with criteria written from each persona's backstory and strengths. This prevents the three-identical-answers problem from the Lab test and keeps the room lively without spam. Trust still gates who may speak unprompted.
10. **Banter moments chosen by Jev instead of scripted triggers.** Score "how worth reacting to is this moment" (1 to 5) and pick which persona has the most to say about it. Generate banter only above a threshold, still within the 2-per-list cap.
11. **Read the owner's intent, not their mood.** A `choice` on each owner message: wants a result, wants to chat, wants to think out loud, wants a decision. Crows adapt length and banter to match. (Avoid scoring emotions; stick to what the owner seems to want from the room.)
12. **Debate quality, only in opt-in debates.** Score each turn: new point vs. restatement, concedes vs. holds, evidence offered. Restaters get a nudge; the judge crow gets the scores.
13. **Celebrate real wins.** A `noul` on deliveries: "closes something the owner asked for." That drives the Duolingo-style moment (sparkles, level-up) only when it's earned.

## Guardrails for using Jev
- Every Jev call goes through `Flock.classify`, so budgets, the crow cap, timeouts and the flight log apply (existing rule).
- Log question, answer, probabilities and the action taken. Jev can't explain, so the log is the explanation.
- Batch questions per message to keep calls and latency down. Measure real latency from Utah.
- Jev never blocks work: if it errors or is unsure, crows carry on (existing rule).

## Suggested first experiment (RITE)
Add two questions to every message in one room: **message kind** and **who should respond** (crows present plus `none`). Don't act on them yet. Log the answers next to what actually happened for a few sessions, then decide on bands.

## Sources
- [Jev and System One models: typed decisions (Noze)](https://www.noze.it/en/insights/jev-system-one-typed-decisions/)
- [TypeSafe integration (LangChain docs)](https://docs.langchain.com/oss/javascript/integrations/providers/typesafe)
- [Jev, a System One model: classifier, not LLM (innFactory)](https://innfactory.ai/en/blog/jev-system-one-model-classifier-not-llm/)
- Crows source: `jev.go`