From 5ea1a7a3a26811031b288f0745f36b8c43be2ef1 Mon Sep 17 00:00:00 2001 From: Smokey <150986568+jacisjake@users.noreply.github.com> Date: Tue, 27 Oct 2026 23:07:33 -0600 Subject: [PATCH] Chatroom pivot: spec, black-box notes, Jev strategies; roadmap rewritten --- CHATROOM_SPEC.md | 130 ++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 36 +++++++------ GROKBOT_NOTES.md | 124 +++++++++++++++++++++++++++++++++++++++++++ JEV_STRATEGIES.md | 47 +++++++++++++++++ 4 files changed, 320 insertions(+), 17 deletions(-) create mode 100644 CHATROOM_SPEC.md create mode 100644 GROKBOT_NOTES.md create mode 100644 JEV_STRATEGIES.md diff --git a/CHATROOM_SPEC.md b/CHATROOM_SPEC.md new file mode 100644 index 0000000..7e40e19 --- /dev/null +++ b/CHATROOM_SPEC.md @@ -0,0 +1,130 @@ +# Crows Chatroom: Clean-Room Spec (Draft 1) + +_Draft 2026-10-27. Built from the Grok Bot black-box findings (`GROKBOT_NOTES.md`; Jev research in `JEV_STRATEGIES.md`) and the founding rules in `CLAUDE.md`. It doesn't copy Grok Bot, and it doesn't carry over the current pipeline (Sentry → crows → Elder) by default. Marked **[decided]** = Smokey's call; **[proposed]** = draft, open to veto._ + +## 1. Purpose +An **agentic social construct with a human in the flock.** The owner is a member of the flock, not an operator outside it. Crows talk to the owner and to each other, take on work, argue, check each other, and earn autonomy over time. [decided] It's less deterministic than today's pipeline: **conversation decides the next step**, inside visible guardrails. + +**Not in scope for this draft:** native apps, login, real capabilities (web, files, APIs), hosting. + +## 2. Principles +1. **Guardrails before capability.** [decided, founding rule] +2. **Autonomy is earned** through the trust ladder. [decided, founding rule] +3. **The protocol is built in; roles and formats are configuration** shaped with the owner through conversation. [decided, from Grok Bot finding] +4. **Free-form by default.** Structured debate is opt-in, only when the user asks. [decided] +5. **The classifier judges whether expectations were carried out.** On a miss, its verdict is handed to the responsible crow as a trigger to proceed. Jev scores; crows act. [decided] +6. **One point of contact when swarming,** but the owner can talk to any crow directly. [decided] +7. **Budgets are visible** to the moderator and the owner. [proposed] + +## 3. Concepts +| Term | Meaning | +|---|---| +| **Owner** | Smokey. The only human for now, and a member of the flock. | +| **Flock (open space)** | Where every crow exists by default. Crows can live in the open without belonging to any nest. [decided] | +| **Crow** | An agent with a name, persona (prompt and output policies), trust record and memory. | +| **Nest** | A small room dedicated to one task or project. It has members and a message log. **A crow can belong to many nests at once.** The owner can create a nest or have a crow create one. [decided] | +| **Direct chat** | The owner and one crow, one-to-one. The owner may stay with a single personality for as long as they like. [decided] | +| **Assistant crow** | The owner's default point of contact and liaison when crows are swarming. [decided role; name to be decided later] | +| **Chief of Staff (COS) crow** | Spins up and regulates crows. [decided role; name to be decided later] | +| **Classifier** | Jev (TypeSafe). **Not an agent and not a chat participant.** It classifies and scores. The harness hands its verdicts to a crow as a trigger (a nudge). Also validates orchestration decisions (section 7a) and keeps its current triage job. [decided] | +| **Expectation** | Something a crow said it would do or was asked to do, tracked until it's met or dropped. | + +## 4. Message protocol [proposed] +Every message in a nest has a **kind**. Crows write in plain language, and the harness tags the kind (the model returns a small structured header, or the classifier labels it). + +| Kind | What it does | Example | +|---|---|---| +| `say` | Ordinary talk | "Good find, @Lab B." | +| `request` | Asks a specific crow (or the nest) for work | "@Signals, classify these drops." | +| `claim` | Takes ownership of a request; first claim wins | "Taking it, @Research." | +| `deliver` | Result, addressed back to the requester | "@Research, all three are passes…" | +| `refer` | Points to a file or artifact instead of pasting it | "See research/2026-10-27-watchlist.md" | +| `ask_owner` | A decision for the owner, as an exact approval phrase or A/B choices with a recommendation | "Reply 'approve plan' or pick A/B." | +| `nudge` | A trigger from the harness to one crow, built from a Jev verdict on a missed expectation. Not a chat message from Jev. | "You said three rounds; open round 2." | + +**Rules** +- **First claim wins.** Once a request is claimed, other crows may review but not redo it. +- **Deliver goes back to whoever requested.** +- **Corrections are acknowledged once,** by the crow that was corrected. Others stay quiet unless they add something. +- **Room membership is a boundary.** A crow that isn't a member relays through a member or through the assistant. + +## 5. Who speaks, and when [proposed] +- **Three places to talk:** the open flock, nests, and direct chats. The same rules apply in each. +- **Event-driven.** A crow is only invoked when something new is posted where it's present (the open flock, a nest it belongs to, or its direct chat). No idle loops. +- **One crow, many rooms.** A crow keeps one identity and memory across every nest and direct chat it's in; what it says in one room can inform another (evidence carries over, so do assumptions). +- **Addressed crows must respond** (claim, deliver, or decline with a reason). +- **Unaddressed messages:** whether a crow may speak up unprompted depends on its trust level. + - Stranger: speaks only when addressed. + - Regular: may claim open requests and add review comments. + - Trusted: may start work and open requests to other crows. +- **A nudge (a Jev verdict handed to a crow) counts as a new event for that crow,** so stalled work restarts without the owner asking. + +## 6. Roles [decided roles, proposed details] +**Assistant crow (point of contact)** +- The owner's default chat when swarming. (The owner can still chat directly with any crow, for as long as they like.) It takes requests, relays them, and answers status questions by **reading the nest first** and showing what it's doing. +- To get new crows, it asks the COS. +- Brings decisions to the owner as `ask_owner`. + +**Chief of Staff crow (orchestration)** +- Spins up crows (name, persona, starting trust), creates nests, and assigns members. +- Regulates: retires idle crows, enforces budgets, moderates when asked. +- When the owner requests a debate, the COS (or the moderator it appoints) sets stances, a judge, rules, rounds and, if requested, turn order. + +## 7. Classifier: expectation ledger and nudges [decided behavior, proposed mechanics] +Jev classifies and scores; it isn't an agent. The harness owns the ledger and the triggers. [decided] +- **Extract:** when a message creates an expectation (a promise, a claim, a round plan, a request), Jev classifies it, and the harness records it with an owner crow and a done-condition. +- **Check:** on each new event and on a short timer, Jev scores open expectations against the room log (met, missed, or pending, with confidence). +- **Trigger on a miss:** the harness hands the verdict to the responsible crow as a `nudge` trigger. The crow's model decides how to proceed. +- **Budget:** each nudge counts against the nest's exchange budget. When the budget runs out, open expectations go to the owner as a summary instead of more nudges. + +Initial checks (from testing): promise kept, rounds completed, turn order (debate only), single owner, claims sourced, correction handled once, role adherence, requester gets an answer. + +## 7a. Classifier: validating orchestration [decided role, proposed mechanics] +Jev also scores the **rationale behind orchestration decisions**, so the COS and assistant don't pick crows or models on vibes. [decided] +- **Which crow for which task:** when the COS (or assistant) proposes a crow for a request, Jev scores the fit between the task's classification and the crow's persona, trust record for that task type and recent results. +- **Which model a crow runs on:** when a crow is spun up or reassigned, Jev scores the proposed model against the task type, using the flight log (cost, speed, approval and rejection rates per model). This ties into the planned model comparison (free model, Claude through a gateway, Claude direct). +- **Low score:** handled like any other miss. The harness hands the verdict back to the deciding crow (usually the COS) as a nudge to reconsider; the crow decides. [consistent with section 7] +- Every decision and its score are logged, so the owner can see why a crow or model was chosen. + +## 8. Opt-in debate mode [decided opt-in, proposed shape] +Triggered only by the owner ("debate this," "have them argue"). The moderator posts a kickoff with the question, stances, a judge, rules (respond to specific points, concede when a point lands, demand evidence), a round count and the final-position format. The moderator opens each round, and the classifier enforces round completion and, if requested, turn order. + +## 9. Budgets and guardrails [proposed] +- Each nest shows **remaining exchanges and tokens** to the owner and the moderator. Nothing is hidden. +- Wind-down is announced in the room, not just injected silently. +- Existing guardrails stay: crows in flight, tokens per item and per list, timeouts, fail-safe approvals, prank cooldowns. + +## 10. Trust ladder in the chatroom [proposed] +- Trust is still earned per crow and per task type. +- Trust decides **when a crow may speak unprompted** (section 5) and **what needs owner approval.** +- A missed expectation is a nudge, not a penalty. [decided] Whether repeated misses should affect trust is open. + +## 11. What changes from today's prototype +| Today | Chatroom | +|---|---| +| Sentry triages every item into solo / split / murder | Crows claim work in conversation; the COS assigns when asked | +| Fixed team sizes | Team shape comes from who's in the nest and who claims | +| Elder merges as a fixed step | Synthesis emerges, or the requester asks for it | +| Banter on scripted moments | All talk is in the nest; personas shape the voice | +| Human approves everything until trust is earned | Owner approves by trust level, plus `ask_owner` for decisions | +| Jev classifies triage only | Jev also scores expectations and validates orchestration choices (crow and model); the harness turns misses into nudges | + +Reusable as is: the Go server, SSE event stream, trust ledger, guardrails, personas, nests, flight log, and the UI art direction. + +## 12. Build plan (RITE, one step at a time) [proposed] +1. **One nest, owner plus two crows, event-driven.** `say`, `request`, `claim`, `deliver` with @-addressing. Mock brain first. +2. **The assistant crow** as the owner's default chat, relaying into the nest. +3. **The expectation ledger plus nudges** for two checks first: promise kept and single owner. +4. **The COS crow** spinning up a crow on request. +5. **Opt-in debate mode.** +6. **Visible budgets** in the UI. + +Each step: show it, adjust, then the next. + +## 13. Open questions +1. Names for the assistant and COS crows (to be decided later). +2. ~~Can Jev score custom expectation labels?~~ Answered: yes, Jev takes custom questions (see `JEV_STRATEGIES.md`). +3. Should repeated misses ever affect trust? +4. Does the Sentry survive as a role, or fold into the COS? +5. How does the message kind get tagged: the model's own header, the classifier, or both? +6. What data does Jev need to score model choice well (flight-log stats per model and task type, owner approvals), and how much history before its scores are trusted? diff --git a/CLAUDE.md b/CLAUDE.md index ea55e9a..21681d1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,11 +1,13 @@ # So Many Crows: handoff for the next agent -Read this whole file before changing anything. It is the single source of truth for where the project stands, why it is built this way, and what comes next. Last updated 2026-10-27 by the agent that built v1 through personas. +Read this whole file before changing anything. It is the single source of truth for where the project stands, why it is built this way, and what comes next. Last updated 2026-10-27: same day, the roadmap pivoted toward the chatroom spec (section 8) — the pivot came from `CHATROOM_SPEC.md`, `GROKBOT_NOTES.md` and `JEV_STRATEGIES.md`. ## 1. What this is So Many Crows is a multi-agent app for one owner, Smokey (Jacob Madsen). You hand a "flock" of AI agents ("crows") a todo list in a chat-room web UI. A **Sentry** crow triages each item, worker crows do the work, an **Elder** crow merges multi-crow results, and **the human approves or rejects** until crows earn trust. It belongs to So Many Crows LLC, a Utah LLC Smokey is registering that builds marketing communications (marcomm) tools. +**That pipeline is what runs today, but the direction has changed** (2026-10-27): the target is a conversation-driven chatroom where crows claim work in a nest, two new roles appear (assistant crow, Chief of Staff crow), and Jev audits expectations and nudges stalled work. Read `CHATROOM_SPEC.md` (the design, with `[decided]`/`[proposed]` marks), `GROKBOT_NOTES.md` (its evidence base) and `JEV_STRATEGIES.md` (what the classifier can do). The pipeline isn't deleted — it stays the working app until chatroom work replaces it, and nests, the trust ledger, guardrails, SSE, personas and the flight log all carry over. + Working copy: `~/Projects/so-many-crows` on Smokey's Mac (Apple silicon, macOS). There is **no Git remote yet** (see section 8). Do not confuse it with `~/Projects/crows`, an unrelated older folder: never touch it. ## 2. How to work with Smokey @@ -35,7 +37,7 @@ These are his stated preferences. Follow them every time. - **The main LLM is switchable** via `CROWS_PROVIDER`: `openrouter` (default model `openrouter/free`, a router that picks a different free model per call) or `anthropic` (default `claude-sonnet-5-5`). Every flight logs which model actually answered. - **Jev (TypeSafe) is a classifier, run in tandem with the LLM** for the Sentry's triage, called **directly from TypeSafe** (`https://api.typesafe.ai/v1/systemone`), not via OpenRouter. Jev answers mode and task type with calibrated confidence. Its call stands when both confidences are at least 0.7 and the mode is not `split`. Otherwise the LLM Sentry decides: when Jev is unsure, when it calls a split (Jev can't write subtasks), or when Jev errors. Jev never blocks a list. - **Smokey's comparison plan:** a free model first, then Claude over OpenCode Zen, then Claude direct from Anthropic. Status: OpenRouter free works (live-tested). Zen is **not built**: Zen's own docs say its free models only work inside the OpenCode app, so only paid Zen models can be used from this app. Claude direct is **built but never live-tested** (no Anthropic key has been provided). -- **OpenRouter free tier limits** (key is on the free tier): 50 free-model requests per day, 20 per minute. Each todo item costs about 2 to 6 requests (Sentry, crows, Elder, banter). Jev triage saves one request per confidently-triaged item. +- **OpenRouter limits** (corrected 2026-10-27: the account is paid, $50 in credits bought, `is_free_tier: false`): 1,000 free-model requests per day, 20 per minute (https://openrouter.ai/docs/api-reference/limits). Check live with `GET /api/v1/key` (never print the label or key). **Pinned model: `CROWS_MODEL=google/gemma-4-26b-a4b-it:free` (switched from gemma-4-31b-it:free after upstream 429s)**; keep it until Smokey says otherwise. Each todo item costs about 2 to 6 requests (Sentry, crows, Elder, banter). Jev triage saves one request per confidently-triaged item. ### Stack - **Go backend, standard library only**, Go 1.24 (`go.mod`, module `somanycrows`). Smokey's team prefers Go or Rust, hosted on Firebase or Google Cloud (GCP) later. @@ -73,8 +75,11 @@ These are his stated preferences. Follow them every time. | `jev.go` | Jev classifier client and the guarded `classify` call | | `personas.go`, `banter.go` | Personas and policies; retrieval-then-generation banter | | `ledger.go`, `guardrails.go`, `nests.go`, `flightlog.go` | Trust ledger and requester strikes; limits; nests; JSON Lines flight log | -| `*_test.go` | 26 tests; run with `-race` | +| `*_test.go` | 31 tests; run with `-race` | | `EVENTS.md` | **The server and UI contract**: every event, payload and endpoint. Change it first, then the code. | +| `CHATROOM_SPEC.md` | **The pivot target**: the chatroom design (message kinds, roles, expectation ledger, build plan). | +| `GROKBOT_NOTES.md` | Black-box observations of Grok Bot that the spec is built from. | +| `JEV_STRATEGIES.md` | What Jev can do (question types, bands, calibration) and strategies for using it. | | `RIVE_CONTRACT.md` | Spec for the animator's `crows.riv` (data binding, one `Crow` artboard for v1) | | `PERSONAS.md` | Generated persona sheet. Regenerate with `./crows personas > PERSONAS.md` after editing `personas.go` | | `ui/` | Vite + TypeScript source (`src/main.ts`, `crow.ts`, `api.ts`, `style.css`). Builds into `web/` (git-ignored) | @@ -105,10 +110,10 @@ Guardrail defaults (`guardrails.go`): 5 crows in flight, murder of 3, 20 items p ## 6. Verified state (2026-10-27) -- All 26 tests pass with `-race`. +- All 31 tests pass with `-race`. - Live test with real keys: Jev triaged "Should we launch a CLI or a web app first" as murder (91%) and analysis (100%) in 195 ms; Hugin, Munin and Corvina answered on three different free models; the Elder merged in character; after a rejection, Soot generated: *"Hugin, another grand scheme foiled? Perhaps the CLI is the tiny bird worth chasing."* Total 6,257 tokens. - UI end-to-end tested headless at desktop and phone widths: nests, borrowing, approvals, reactions, prank cooldown, banter asides. No console errors. -- **Not yet done:** Smokey has not yet used the app himself through `start.command` with real crows. That is the next thing to support. +- **Not yet done:** Smokey has not yet used the app himself through `start.command` with real crows. It still matters (section 8, step 6), but the chatroom work leads now. ## 7. Known gaps and gotchas @@ -123,19 +128,16 @@ Guardrail defaults (`guardrails.go`): 5 crows in flight, murder of 3, 20 items p ## 8. What happens next (in order) -Do one at a time. Show the result. Ask before starting the next. +The roadmap pivoted (2026-10-27): the chatroom spec is now the direction. Do one step at a time. Show the result. Ask before starting the next. **Contracts first** still applies: change `EVENTS.md` before UI or server code for any chatroom message kind or event. -1. **Support Smokey's first hands-on session.** He starts `start.command` and uses the app with real crows. Watch the Terminal window and `./crows log` for errors, and fix what he hits. Keep an eye on the 50-a-day free limit. -2. **Print banter on the command line.** Small: handle `crow.banter` in the command line's printing emitter (`main.go`). -3. **Provider comparison.** Smokey wants to compare (a) a free model, (b) Claude over OpenCode Zen, (c) Claude direct from Anthropic. Steps: pin one model per run with `CROWS_MODEL` for fair comparisons (the free router randomizes); add an OpenCode Zen provider to `NewBrain` (base `https://opencode.ai/inference/`, Claude models via `anthropic/v1/messages`, open models via `openai/v1/chat/completions`; confirm the auth header in Zen's docs first, since they didn't state it for these endpoints); get keys from Smokey; then add a small `crows stats` summary of the flight log (per model: calls, average tokens, average milliseconds, approval and rejection rates). Ask Smokey which model to pin before running. -4. **Review queue for "regular" trust:** a way to review outputs that were auto-accepted while unreviewed. -5. **Rive.** An animator builds `crows.riv` per `RIVE_CONTRACT.md`; then add a Rive-backed `CrowView` in `ui/src/crow.ts` (the interface is already shaped for it). Smokey's crow art is the character. -6. **Threads per item** (Slack-style): asked of Smokey, not answered yet. Ask before building. -7. **Persona strengths in crow picking** (for example, Munin for retrieval). Not requested; propose only. -8. **First real capability (likely web search):** write its guardrail spec first (scope, limits, what needs approval), get Smokey's approval, then build. -9. **Login and real requester identity** before the server ever leaves localhost. -10. **GitHub:** Smokey deferred it. When he's ready: he creates an empty repo `jacisjake/so-many-crows`, then push. Commit email must be `150986568+jacisjake@users.noreply.github.com` (pushes with his real email are rejected). -11. **Hosting on Google Cloud** later; not started. +1. **Resolve the spec's open blockers before code** (Smokey decides, not the agent): message-kind tagging (Jev can tag it — see `JEV_STRATEGIES.md` strategy 1), the nudge loop guard (nudges count against the nest budget; a miss must not restart work forever), first-claim-wins arbitration when two crows claim at once, and Sentry/Elder's fate (spec open question 4). +2. **Chatroom build plan step 1** (`CHATROOM_SPEC.md` section 12): one nest, owner plus two crows, event-driven; `say`, `request`, `claim`, `deliver` with @-addressing; mock brain first, live models after it works. +3. **The assistant crow** as the owner's default chat, relaying into the nest. +4. **The expectation ledger plus nudges**, two checks first (promise kept, single owner), Jev questions per `JEV_STRATEGIES.md`. The "log-only experiment" there (kind + who-responds, no acting) can run beside current pipeline traffic to collect calibration data. +5. **The COS crow** spinning up crows on request. Then opt-in debate mode, then visible budgets. +6. **Smokey's first hands-on session** with the current pipeline stays worth supporting — if he runs `start.command`, watch the Terminal and `./crows log` and fix what he hits (1,000-a-day free-model limit). + +Paused by the pivot (resume only if Smokey asks): provider comparison beyond pinning a model (the pinned `CROWS_MODEL` stays); CLI banter printing; review queue for "regular" trust; threads; Rive; web-search capability; login; GitHub; GCP. GitHub note for when it happens: Smokey creates an empty repo `jacisjake/so-many-crows`, commit email must be `150986568+jacisjake@users.noreply.github.com`. ## 9. Conventions diff --git a/GROKBOT_NOTES.md b/GROKBOT_NOTES.md new file mode 100644 index 0000000..ea5968e --- /dev/null +++ b/GROKBOT_NOTES.md @@ -0,0 +1,124 @@ +# Grok Bot Black-Box Notes + +_Started 2026-10-27. These are observations of Smokey's own Grok Bot account (desktop app v0.77) plus public docs. Research spreadsheet: [Grok Bot black-box research](https://docs.google.com/spreadsheets/d/14T4vVvI6r0DNEMxPNDbHlzpaGmYGD6KLJHAurXWCrHM/edit). The design that came out of this is `CHATROOM_SPEC.md`; Jev research is `JEV_STRATEGIES.md`._ + +_Later decision (see the spec): Jev is not an agent. It classifies and scores; the harness hands its verdicts to a crow as a nudge. Where these notes say "the classifier nudges," read "the harness nudges, based on Jev's verdict."_ + +## Goal (Smokey, 2026-10-27) +Not 1:1 parity with Grok Bot. The target is **a chatroom of agents with varying levels of autonomy, and rhetoric and debate among them.** Clicking around to find features is part of the experiment. + +## Decisions (Smokey, 2026-10-27) +- **Debate rules are enforced only when the end user asks for them.** By default the room is free-form; structured debate (stances, judge, rounds, turn order) is opt-in. +- **The classifier (Jev) judges whether expectations were carried out.** The failures seen in these tests are what it should check for (see "Expectation checks for the classifier" below). +- **On a missed expectation, the classifier nudges the LLM to proceed.** The response to a miss is to prompt the crow to carry on and finish, not to flag the user or penalize trust. + +## Two key roles crows is missing (Smokey, 2026-10-27) +- **Chief of Staff (COS): the orchestration Bot.** It spins up new Bots and regulates them. +- **Grok Bot: the personal assistant.** It's the user's direct chat, and the liaison between the user, the COS and the other Bots. +- The user can talk to any Bot directly. When Bots are swarming, though, one focused point of contact works better for the user. +- To spin up Bots, the user asks Grok Bot, with a task in mind. + +## Expectation checks for the classifier (draft, from observations) +Each check is something a crow said or was asked to do, compared with what actually happened in the room. **On a miss, the classifier nudges the responsible crow's LLM to proceed.** + +| Expectation | Source of the expectation | Missed in testing | Nudge | +|---|---|---|---| +| **Promise kept:** a stated commitment ("I'll report after three rounds") gets fulfilled | Crow's own message | Grok Bot promised a report and never sent it unprompted | "You said you'd report after round 3. Proceed." | +| **Rounds completed:** a structured task reaches its stated end (rounds, final positions) | Moderator's kickoff | Debate stopped after round 1; no final positions | Moderator: "Open round 2." | +| **Turn order followed** (only when the user asked for structure) | Moderator's kickoff | Lab B spoke before the Advocate | Out-of-turn crow: hold until called | +| **Single owner:** one crow claims a task before working; no duplicate work | Claim rule | Three Labs did the same search in parallel | Second crow: "X is claimed; pick another part or review." | +| **Claims sourced** before they're merged or agreed to | Room rules / Skeptic | Narrator claim got through all three peers | Claiming crow: "Add a source or mark it unverified." | +| **Correction handled once:** one acknowledgment, not N | Harness norm | Three "you're right" replies to one correction | Extra repliers: suppress | +| **Role adherence:** a crow argues or acts within its assigned role | Moderator's role assignments | (Held in the debate test) | | +| **Requester gets an answer:** the reply goes back to whoever asked | Handoff protocol | (Held in the Investment Team) | | + +Answered later: Jev supports custom questions, so these checks are just more Jev questions (see `JEV_STRATEGIES.md`). The nudge wording above is a placeholder. + +## Product behavior vs. what Smokey configured +- **Out of the box (product):** how Bots interact with each other. They @-address teammates, acknowledge requests, hand off work and reply back to whoever asked. +- **Shaped by Smokey through conversation:** what each Bot does, meaning its role, its output format and its domain. + +## Teams in the account (2026-10-27) +- **Management / Investment Team:** Grok Research, Signals, Risk, Strategy, Execution +- **HSA/FSA:** Competitors, Consumer, TPA Voice, Pulse, Critic, Marketing +- **Benefit X Watch:** a group chat +- **Lab A, Lab B, Lab C:** test Bots plus a group chat (created 2026-10-27) +- **Advocate, Skeptic:** test Bots created by Grok Bot, plus the group "Debate test: orchestrator vs self-organize" (2026-10-27) + +## Observation 1: directed handoffs (Investment Team, established Bots) +1. **Directed request:** one Bot @-addresses one teammate with one specific job. +2. **Public claim:** the recipient acknowledges ("On it, @Grok Research… will post shortly"). That makes it the single owner of the stage. +3. **Delivery addressed back:** the result starts with an @-mention of the requester, closing the loop. +4. **File handoff:** Bots point to files on the shared computer instead of pasting contents. +5. **Structured output** came from Smokey's conversations with the Bots, not from the product. + +## Observation 2: undirected request to new Bots (Lab group) +Task: "Look for new RPG epic fantasy novels and compare them to The Traveling Inn or Mark of the Fool." No @-mention, no roles. + +- **Setup:** creating a Bot is just name, shape and color. A new Bot opens its own chat by asking what its job is, and offers starting jobs drawn from account context. New Bots aren't blank. +- **Round 1, parallel duplication:** all three answered on their own, with heavy overlap. Nobody claimed the task or split it up. +- **Cross-talk after round 1:** praise, a factual correction (@-mentioned), a concession. +- **Emergent synthesizer:** Lab B posted, unprompted, "The three of us all land on the same pick…" +- **Peer review missed a factual error:** Smokey caught it; then all three acknowledged, which made three replies to one correction. +- **Debate tone:** polite, quick to agree, concedes easily. + +## Observation 3: Chief of Staff and Grok Bot in practice (read only) +**Chief of Staff** +- Restates the user's direction and delegates by name ("I've asked Research to rewrite…"). +- Brings the specialist's proposal back for approval, using an exact approval phrase. +- States guardrails inline, and relays another Bot's dissent as its own post with an alternate approval phrase. + +**Grok Bot (assistant)** +- Relays decisions to specialists with a visible "Messaged → [Bot]" marker. +- **Respects room membership:** when it isn't a member of a room, it relays through a member Bot. +- Asks the user for decisions as A/B cards with its own recommendation; brings threshold alerts unprompted; corrects itself in the open; doesn't re-act on instructions it has already handled. + +## Observation 4: Grok Bot spins up a debate (2026-10-27) +Smokey's suggestion: ask Grok Bot to spin up Bots with a task in mind. Request sent to Grok Bot: create an Advocate and a Skeptic, add them to a group with Lab A–C, and debate "one orchestrator vs. self-organizing in a shared chat." + +**How Grok Bot handled it** +- Acknowledged at once ("On it. I'm setting up the Advocate and Skeptic bots first…"). +- **Created the Bots itself and didn't route to the COS** (in this case). It created the group, named it, and **joined the room itself as moderator**. +- Reported back in its own chat: "The debate has started… I'm in there too." It said it would report results after about three rounds. +- **Filled in what the request left out:** "You didn't give the Labs roles, so I picked some." That's a three-way debate (Lab A self-organize, Lab B hybrid) plus Lab C as judge. + +**The kickoff post (moderator framing)** +- Restated the question, assigned a role to each participant, and set the rules: keep turns short, "respond to specific points instead of restating, concede when a point lands," about three rounds, then a one-line final position plus the one remaining disagreement. It set a speaking order ("Advocate, you're up first"). + +**What happened** +- **Speaking order wasn't enforced.** Lab B posted its opening before the Advocate. Turn order was a request, not a lock. +- **The Skeptic did its job.** Before arguments began, it demanded definitions ("What does 'better' mean: lower cost, faster completion, fewer errors, or less work for Smokey?"), a concrete task, and "a real failure… not one that's just possible in principle." +- **The Advocate defined its terms** in answer ("fewer errors that reach Smokey… cost second"), conceded a point, and **moved to a partial position** (an orchestrator assigns and merges, and checking stays open to everyone). +- **Evidence from earlier work:** Bots cited the Lab book thread from earlier tonight as data. **Memory of another room carried over.** +- **Lab A argued self-organizing** and proposed a **claim-before-you-work rule** ("the first agent to post 'taking X' owns X"). It conceded that cost grows past about five agents. +- **Lab C's judge scorecard** listed which claims survived and which didn't, and called out the Advocate's Investment Team example as "a pipeline, not evidence about an orchestrator." +- **Owning a mistake:** Lab B owned its earlier narrator error and proposed that an orchestrator "refuses to merge any factual claim that doesn't have a source attached." +- **Debate quality:** far stronger than Observation 2. Given roles and rules, the Bots held positions, demanded evidence and conceded specific points. + +**The report (Smokey asked for it about 8 minutes after kickoff)** +- **The debate stalled after round 1.** Grok Bot: "They only got through one round, not three. The room closed before rounds 2 and 3, so nobody posted a final one-line position." **The moderator promised three rounds and a report, but didn't push the room forward or report back on its own.** It acted only when asked. +- **The report itself was good:** it read the room first (status "Reading the debate transcript," then "Analyzing the debate stances"), then gave the consensus, how each side moved, three points still in dispute, the judge's verdict, and an offer: "Want me to start rounds 2 and 3…?" +- **The room's consensus answer to our design question:** a hybrid. One orchestrator owns assignment and the final merge, and checking and objecting stay open to everyone. + +**Why it stopped (Smokey asked; Grok Bot investigated, with status "Investigating the room closure")** +- **The platform winds rooms down.** "In the last turn each bot got, the group chat added a note telling them it was wrapping up and to reply only if it was essential." So **there's a harness-level exchange budget** that injects a wrap-up instruction into each Bot's turn. +- **Grok Bot can't see that rule:** "I can't see what decides when a room wraps up, or how many exchanges it allows first… I won't guess at the exact rule." +- **Bots only speak in response to something.** "A bot in a room only speaks when something new is posted." No idle loop, no timers. +- **The moderator took responsibility:** "That gap is on me. I posted the kickoff once… when I should have opened each round." It offered to run rounds 2 and 3, then ask for final positions. + +## Implications for crows (draft) +- Built into the harness: request, claim, deliver, file reference, plus @-addressing and replies to the requester. +- Roles and formats are configuration, developed with the owner through conversation. +- **Two standing roles instead of Sentry and Elder:** a COS crow that spawns and regulates crows, and an assistant crow that's the owner's single point of contact. The assistant can also spin up crows and moderate a room for a task. +- **The classifier audits expectations against outcomes and nudges the LLM to proceed on a miss** (table above). That's what Grok Bot lacks: nothing noticed the moderator's broken promise until the user asked. The nudge is also the room's heartbeat. Event-driven crows go quiet, and a classifier nudge supplies the "something new is posted" that restarts them. +- **Crows speak only when something new is posted** (event-driven). That's cheap and prevents runaway chatter; the classifier nudge is what keeps unfinished work moving. +- **The exchange budget should be visible.** Grok Bot's budget is real but hidden, even from its own moderator. Show the room's remaining budget to the moderator and the user, and let a user-requested debate extend it. Nudges count against the budget, so the classifier can't create a loop. +- **The assistant reports by reading the room, not from memory,** and shows what it's doing. Keep that. +- **Structured debate is opt-in.** The default room is free-form. +- **Claim-before-you-work** (proposed by Lab A) is a cheap fix for broadcast duplication. +- **Sourced-claims-only merging** (proposed by Lab B) is a candidate guardrail for synthesis. +- **Room membership is a real boundary.** Crows that aren't members relay through members. +- **Approvals are conversational:** an exact phrase or an A/B card, with the recommending crow named. +- **Acknowledgments should collapse.** One correction shouldn't produce N "you're right" replies. +- **Autonomy varies by crow** (the trust ladder) and decides who may speak unprompted vs. only when addressed. +- **Shared memory across rooms** has a cost and a benefit: evidence carries over, but so do assumptions. diff --git a/JEV_STRATEGIES.md b/JEV_STRATEGIES.md new file mode 100644 index 0000000..721c8cb --- /dev/null +++ b/JEV_STRATEGIES.md @@ -0,0 +1,47 @@ +# Jev: Strategies for More Accurate and Engaging Crows + +_Research 2026-10-27. Sources at the bottom. Vendor numbers (latency, price, calibration) come from TypeSafe via secondary write-ups and haven't been checked against crows' own workload. Related: `CHATROOM_SPEC.md`, `GROKBOT_NOTES.md`._ + +## What Jev can do (facts) +- **Endpoint:** `POST https://api.typesafe.ai/v1/systemone`. A request has `model`, a `state` (plain text or JSON), and named `questions`. All questions in a request are evaluated independently, in parallel, in one call. The context budget is reported as 64,000 tokens per request. +- **Three question types:** + - `choice`: one option from up to 255. `criteria` maps option names to descriptions. Returns `choice`, `probabilities` and `confidence`. + - `score`: a position on an ordered scale of 2 to 10 levels (`criteria` is an array of level descriptions). Can land between levels. Returns `score`, `legend`, `probabilities` and `confidence`. + - `noul`: yes/no. Returns the probability of yes, with no `confidence`. +- **`confidence` isn't the probability of being right.** It measures how concentrated the probabilities are. Accuracy has to be measured on our own labeled data. +- **Reported cost and speed (vendor):** about 70–500 ms per call; input billed (about $0.042 per million input tokens for Jev 1.13), output free. +- **Limits:** text only, English-first, no explanations, no generated text. Early access with changing rate limits. It receives whatever is in `state`, so secrets must stay out. + +**Crows today** uses one call with two `choice` questions (mode, task type) and a single global threshold of 0.7 (`jev.go`). So **custom labels are supported**, which answers spec open question 2: expectation checks and orchestration checks are just more questions. + +## Strategies for accuracy +1. **Many narrow questions per call, not one big one.** For each new message, send one state and ask several questions together: message kind (`choice`), "does this create an expectation?" (`noul`), who it's addressed to (`choice` over crows present, plus `none`). One call, parallel answers. This also answers spec open question 5: Jev can tag the message kind. +2. **Always include an escape option** (`other`, `none`, `unclear`) so Jev is never forced into a wrong label. Today's `task_type` has `other`; `mode` doesn't. +3. **Use `score` for anything with degrees.** Use `noul` only for crisp yes/no conditions (for example, "states a fact without a source"). A `noul` of 0.5 means a coin flip, not "medium." For "how badly is this expectation missed," use a `score` with written levels. +4. **Three bands per question, not one global threshold.** High: the harness acts (sends the nudge, routes the message). Medium: log it and let the crow decide. Low: do nothing, or show the owner. Use stricter bands for anything that will become irreversible once real capabilities exist. +5. **Calibrate on our own labels.** Log every verdict with its probabilities, then record what actually happened (owner approved or rejected, expectation later met or not). Use that to set each question's bands. The single 0.7 threshold today is a guess. +6. **Structure the state as JSON, trimmed.** Send the expectation, the responsible crow, and only the relevant slice of the room, not the whole log. Never include `.env` values or anything private. +7. **Composite scores for orchestration (spec 7a).** Jev can't explain itself, so break crow and model choice into narrow questions (task complexity `score`, risk `score`, persona fit `choice`), then weight them in Go. Weights are visible and tunable, and the log shows exactly which answer drove a decision. +8. **Jev checks crow output before it's merged.** A few `noul` questions on each `deliver`: does it answer the request, does it state facts without sources, does it contradict an earlier message. This is the "claims sourced" check, and it would have caught the Wandering Inn narrator error before the owner did. + +## Strategies for engagement +9. **Who speaks next in the open flock.** For an unaddressed message, ask Jev to pick which crow (or `none`) should respond, with criteria written from each persona's backstory and strengths. This prevents the three-identical-answers problem from the Lab test and keeps the room lively without spam. Trust still gates who may speak unprompted. +10. **Banter moments chosen by Jev instead of scripted triggers.** Score "how worth reacting to is this moment" (1 to 5) and pick which persona has the most to say about it. Generate banter only above a threshold, still within the 2-per-list cap. +11. **Read the owner's intent, not their mood.** A `choice` on each owner message: wants a result, wants to chat, wants to think out loud, wants a decision. Crows adapt length and banter to match. (Avoid scoring emotions; stick to what the owner seems to want from the room.) +12. **Debate quality, only in opt-in debates.** Score each turn: new point vs. restatement, concedes vs. holds, evidence offered. Restaters get a nudge; the judge crow gets the scores. +13. **Celebrate real wins.** A `noul` on deliveries: "closes something the owner asked for." That drives the Duolingo-style moment (sparkles, level-up) only when it's earned. + +## Guardrails for using Jev +- Every Jev call goes through `Flock.classify`, so budgets, the crow cap, timeouts and the flight log apply (existing rule). +- Log question, answer, probabilities and the action taken. Jev can't explain, so the log is the explanation. +- Batch questions per message to keep calls and latency down. Measure real latency from Utah. +- Jev never blocks work: if it errors or is unsure, crows carry on (existing rule). + +## Suggested first experiment (RITE) +Add two questions to every message in one room: **message kind** and **who should respond** (crows present plus `none`). Don't act on them yet. Log the answers next to what actually happened for a few sessions, then decide on bands. + +## Sources +- [Jev and System One models: typed decisions (Noze)](https://www.noze.it/en/insights/jev-system-one-typed-decisions/) +- [TypeSafe integration (LangChain docs)](https://docs.langchain.com/oss/javascript/integrations/providers/typesafe) +- [Jev, a System One model: classifier, not LLM (innFactory)](https://innfactory.ai/en/blog/jev-system-one-model-classifier-not-llm/) +- Crows source: `jev.go`