so-many-crows/GROKBOT_NOTES.md

Grok Bot Black-Box Notes

Started 2026-10-27. These are observations of Smokey's own Grok Bot account (desktop app v0.77) plus public docs. Research spreadsheet: Grok Bot black-box research. The design that came out of this is CHATROOM_SPEC.md; Jev research is JEV_STRATEGIES.md.

Later decision (see the spec): Jev is not an agent. It classifies and scores; the harness hands its verdicts to a crow as a nudge. Where these notes say "the classifier nudges," read "the harness nudges, based on Jev's verdict."

Goal (Smokey, 2026-10-27)

Not 1:1 parity with Grok Bot. The target is a chatroom of agents with varying levels of autonomy, and rhetoric and debate among them. Clicking around to find features is part of the experiment.

Decisions (Smokey, 2026-10-27)

• Debate rules are enforced only when the end user asks for them. By default the room is free-form; structured debate (stances, judge, rounds, turn order) is opt-in.
• The classifier (Jev) judges whether expectations were carried out. The failures seen in these tests are what it should check for (see "Expectation checks for the classifier" below).
• On a missed expectation, the classifier nudges the LLM to proceed. The response to a miss is to prompt the crow to carry on and finish, not to flag the user or penalize trust.

Two key roles crows is missing (Smokey, 2026-10-27)

• Chief of Staff (COS): the orchestration Bot. It spins up new Bots and regulates them.
• Grok Bot: the personal assistant. It's the user's direct chat, and the liaison between the user, the COS and the other Bots.
• The user can talk to any Bot directly. When Bots are swarming, though, one focused point of contact works better for the user.
• To spin up Bots, the user asks Grok Bot, with a task in mind.

Expectation checks for the classifier (draft, from observations)

Each check is something a crow said or was asked to do, compared with what actually happened in the room. On a miss, the classifier nudges the responsible crow's LLM to proceed.

Expectation	Source of the expectation	Missed in testing	Nudge
Promise kept: a stated commitment ("I'll report after three rounds") gets fulfilled	Crow's own message	Grok Bot promised a report and never sent it unprompted	"You said you'd report after round 3. Proceed."
Rounds completed: a structured task reaches its stated end (rounds, final positions)	Moderator's kickoff	Debate stopped after round 1; no final positions	Moderator: "Open round 2."
Turn order followed (only when the user asked for structure)	Moderator's kickoff	Lab B spoke before the Advocate	Out-of-turn crow: hold until called
Single owner: one crow claims a task before working; no duplicate work	Claim rule	Three Labs did the same search in parallel	Second crow: "X is claimed; pick another part or review."
Claims sourced before they're merged or agreed to	Room rules / Skeptic	Narrator claim got through all three peers	Claiming crow: "Add a source or mark it unverified."
Correction handled once: one acknowledgment, not N	Harness norm	Three "you're right" replies to one correction	Extra repliers: suppress
Role adherence: a crow argues or acts within its assigned role	Moderator's role assignments	(Held in the debate test)
Requester gets an answer: the reply goes back to whoever asked	Handoff protocol	(Held in the Investment Team)

Answered later: Jev supports custom questions, so these checks are just more Jev questions (see JEV_STRATEGIES.md). The nudge wording above is a placeholder.

Product behavior vs. what Smokey configured

• Out of the box (product): how Bots interact with each other. They @-address teammates, acknowledge requests, hand off work and reply back to whoever asked.
• Shaped by Smokey through conversation: what each Bot does, meaning its role, its output format and its domain.

Teams in the account (2026-10-27)

• Management / Investment Team: Grok Research, Signals, Risk, Strategy, Execution
• HSA/FSA: Competitors, Consumer, TPA Voice, Pulse, Critic, Marketing
• Benefit X Watch: a group chat
• Lab A, Lab B, Lab C: test Bots plus a group chat (created 2026-10-27)
• Advocate, Skeptic: test Bots created by Grok Bot, plus the group "Debate test: orchestrator vs self-organize" (2026-10-27)

Observation 1: directed handoffs (Investment Team, established Bots)

1. Directed request: one Bot @-addresses one teammate with one specific job.
2. Public claim: the recipient acknowledges ("On it, @Grok Research… will post shortly"). That makes it the single owner of the stage.
3. Delivery addressed back: the result starts with an @-mention of the requester, closing the loop.
4. File handoff: Bots point to files on the shared computer instead of pasting contents.
5. Structured output came from Smokey's conversations with the Bots, not from the product.

Observation 2: undirected request to new Bots (Lab group)

Task: "Look for new RPG epic fantasy novels and compare them to The Traveling Inn or Mark of the Fool." No @-mention, no roles.

• Setup: creating a Bot is just name, shape and color. A new Bot opens its own chat by asking what its job is, and offers starting jobs drawn from account context. New Bots aren't blank.
• Round 1, parallel duplication: all three answered on their own, with heavy overlap. Nobody claimed the task or split it up.
• Cross-talk after round 1: praise, a factual correction (@-mentioned), a concession.
• Emergent synthesizer: Lab B posted, unprompted, "The three of us all land on the same pick…"
• Peer review missed a factual error: Smokey caught it; then all three acknowledged, which made three replies to one correction.
• Debate tone: polite, quick to agree, concedes easily.

Observation 3: Chief of Staff and Grok Bot in practice (read only)

Chief of Staff

• Restates the user's direction and delegates by name ("I've asked Research to rewrite…").
• Brings the specialist's proposal back for approval, using an exact approval phrase.
• States guardrails inline, and relays another Bot's dissent as its own post with an alternate approval phrase.

Grok Bot (assistant)

• Relays decisions to specialists with a visible "Messaged → [Bot]" marker.
• Respects room membership: when it isn't a member of a room, it relays through a member Bot.
• Asks the user for decisions as A/B cards with its own recommendation; brings threshold alerts unprompted; corrects itself in the open; doesn't re-act on instructions it has already handled.

Observation 4: Grok Bot spins up a debate (2026-10-27)

Smokey's suggestion: ask Grok Bot to spin up Bots with a task in mind. Request sent to Grok Bot: create an Advocate and a Skeptic, add them to a group with Lab A–C, and debate "one orchestrator vs. self-organizing in a shared chat."

How Grok Bot handled it

• Acknowledged at once ("On it. I'm setting up the Advocate and Skeptic bots first…").
• Created the Bots itself and didn't route to the COS (in this case). It created the group, named it, and joined the room itself as moderator.
• Reported back in its own chat: "The debate has started… I'm in there too." It said it would report results after about three rounds.
• Filled in what the request left out: "You didn't give the Labs roles, so I picked some." That's a three-way debate (Lab A self-organize, Lab B hybrid) plus Lab C as judge.

The kickoff post (moderator framing)

• Restated the question, assigned a role to each participant, and set the rules: keep turns short, "respond to specific points instead of restating, concede when a point lands," about three rounds, then a one-line final position plus the one remaining disagreement. It set a speaking order ("Advocate, you're up first").

What happened

• Speaking order wasn't enforced. Lab B posted its opening before the Advocate. Turn order was a request, not a lock.
• The Skeptic did its job. Before arguments began, it demanded definitions ("What does 'better' mean: lower cost, faster completion, fewer errors, or less work for Smokey?"), a concrete task, and "a real failure… not one that's just possible in principle."
• The Advocate defined its terms in answer ("fewer errors that reach Smokey… cost second"), conceded a point, and moved to a partial position (an orchestrator assigns and merges, and checking stays open to everyone).
• Evidence from earlier work: Bots cited the Lab book thread from earlier tonight as data. Memory of another room carried over.
• Lab A argued self-organizing and proposed a claim-before-you-work rule ("the first agent to post 'taking X' owns X"). It conceded that cost grows past about five agents.
• Lab C's judge scorecard listed which claims survived and which didn't, and called out the Advocate's Investment Team example as "a pipeline, not evidence about an orchestrator."
• Owning a mistake: Lab B owned its earlier narrator error and proposed that an orchestrator "refuses to merge any factual claim that doesn't have a source attached."
• Debate quality: far stronger than Observation 2. Given roles and rules, the Bots held positions, demanded evidence and conceded specific points.

The report (Smokey asked for it about 8 minutes after kickoff)

• The debate stalled after round 1. Grok Bot: "They only got through one round, not three. The room closed before rounds 2 and 3, so nobody posted a final one-line position." The moderator promised three rounds and a report, but didn't push the room forward or report back on its own. It acted only when asked.
• The report itself was good: it read the room first (status "Reading the debate transcript," then "Analyzing the debate stances"), then gave the consensus, how each side moved, three points still in dispute, the judge's verdict, and an offer: "Want me to start rounds 2 and 3…?"
• The room's consensus answer to our design question: a hybrid. One orchestrator owns assignment and the final merge, and checking and objecting stay open to everyone.

Why it stopped (Smokey asked; Grok Bot investigated, with status "Investigating the room closure")

• The platform winds rooms down. "In the last turn each bot got, the group chat added a note telling them it was wrapping up and to reply only if it was essential." So there's a harness-level exchange budget that injects a wrap-up instruction into each Bot's turn.
• Grok Bot can't see that rule: "I can't see what decides when a room wraps up, or how many exchanges it allows first… I won't guess at the exact rule."
• Bots only speak in response to something. "A bot in a room only speaks when something new is posted." No idle loop, no timers.
• The moderator took responsibility: "That gap is on me. I posted the kickoff once… when I should have opened each round." It offered to run rounds 2 and 3, then ask for final positions.

Implications for crows (draft)

• Built into the harness: request, claim, deliver, file reference, plus @-addressing and replies to the requester.
• Roles and formats are configuration, developed with the owner through conversation.
• Two standing roles instead of Sentry and Elder: a COS crow that spawns and regulates crows, and an assistant crow that's the owner's single point of contact. The assistant can also spin up crows and moderate a room for a task.
• The classifier audits expectations against outcomes and nudges the LLM to proceed on a miss (table above). That's what Grok Bot lacks: nothing noticed the moderator's broken promise until the user asked. The nudge is also the room's heartbeat. Event-driven crows go quiet, and a classifier nudge supplies the "something new is posted" that restarts them.
• Crows speak only when something new is posted (event-driven). That's cheap and prevents runaway chatter; the classifier nudge is what keeps unfinished work moving.
• The exchange budget should be visible. Grok Bot's budget is real but hidden, even from its own moderator. Show the room's remaining budget to the moderator and the user, and let a user-requested debate extend it. Nudges count against the budget, so the classifier can't create a loop.
• The assistant reports by reading the room, not from memory, and shows what it's doing. Keep that.
• Structured debate is opt-in. The default room is free-form.
• Claim-before-you-work (proposed by Lab A) is a cheap fix for broadcast duplication.
• Sourced-claims-only merging (proposed by Lab B) is a candidate guardrail for synthesis.
• Room membership is a real boundary. Crows that aren't members relay through members.
• Approvals are conversational: an exact phrase or an A/B card, with the recommending crow named.
• Acknowledgments should collapse. One correction shouldn't produce N "you're right" replies.
• Autonomy varies by crow (the trust ladder) and decides who may speak unprompted vs. only when addressed.
• Shared memory across rooms has a cost and a benefit: evidence carries over, but so do assumptions.