AI Workspace Operations Copilot

The problem

A workspace copilot gets two very different kinds of request. “What does the expense policy say about client meals?” is a retrieval question — the answer is in the documents. “Is room three free this afternoon, and if so book it for Sarah from two to three?” is an action question — the answer is in a database, and answering it truthfully means actually checking, then committing a reservation.

A single large prompt with both tool sets attached handles neither well: it picks the wrong tool under ambiguity, it loses the thread across turns, and it answers confidently when the retrieved context does not support an answer at all. A user cannot tell a grounded answer from a fluent one, so every invented answer spends trust that the correct ones earned.

So the copilot splits the work into a small graph: an orchestrator that interprets each request and routes it to specialized sub-agents — a Knowledge Base Agent for document Q&A (RAG) and a Booking Agent for meeting-room reservations — then composes a single coherent answer. The graph loops between the orchestrator and the sub-agents until it decides to return to the user, so multi-step requests that touch both the knowledge base and the booking system are handled in one invocation.

Architecture

One shared state machine: the orchestrator routes to its sub-agents, the graph loops back until it returns to the user or the hop counter runs out.

The orchestrator and the sub-agents share a single LangGraph state machine. The orchestrator runs on GPT-OSS-120B via OpenRouter, emitting structured tool calls through Instructor that name the sub-agent(s) to invoke — the free-form argument it sends each agent is not part of the contract. Each sub-agent runs on Groq (Qwen3-32B) at temperature 0 and owns its tools: the KB Agent runs Pinecone vector search, the Booking Agent runs Supabase CRUD (fetch, insert, update). Control returns to the orchestrator after each step, which either dispatches the next step or finalizes a summary. A hop counter in the shared state bounds the loop so runs always terminate.

Query flow. A natural-language request hits POST /query (synchronous) or POST /query-agent (streaming). The streaming endpoint emits Server-Sent Events as the graph executes — agent calls, knowledge base agent, booking agent, then final response — so the client shows live per-agent progress instead of a spinner.

Ingestion flow. A PDF is uploaded to POST /ingestion with a target Pinecone namespace. It is split into chunks (1000 characters, 200-character overlap), embedded in batches with OpenRouter’s llama-nemotron-embed-vl truncated to 1024 dimensions (Matryoshka), and upserted under the namespace. Three embedding backends — OpenRouter, Google Gemini, and local Ollama — are wired up in embedding_config.py and can be swapped in; all three output 1024-dim vectors so they share one vector space with the index.

Decisions

An orchestrator that routes, not a chain that hopes. A linear retrieve-then-reason-then-answer chain was wrong for roughly a third of queries, because a request that needs a database action does not want a vector search in front of it. Making routing an explicit decision — the orchestrator sees the question and emits named tool calls — let each sub-agent keep a small, legible prompt, and made the routing decision the thing the evaluation harness could pin down.

Typed tool calls, validated before they run. Letting the model emit free-form JSON for tools trades one failure mode for another. Instructor and Pydantic make the tool contract a schema: arguments that do not validate are corrected or rejected rather than executed. The same schema is what the harness grades against, so the boundary between the model and the system is typed instead of a hope.

The graph owns the loop. Anything that must be remembered across a cycle lives in the shared state, not in a node — the hop counter, the agent outputs, the transcript. Early on, a retry counter tracked on the wrong object reset every time the graph re-entered the node, turning a rejected step into an infinite corridor; moving it into the typed state made the loop terminate. Per- node retry policies and a hard hop counter mean failures are bounded and explainable rather than unbounded and silent.

Honesty is graded, not suggested. Telling a model to say “I don’t know” when it’s unsure is a suggestion it follows when it feels like it. The interesting decisions — when to return to the user, whether an empty agent response means retry or honest failure — are exactly the ones the evaluation harness pins down, in the same node the production orchestrator runs. The system is honest because the harness measures the cases where it wouldn’t be.

Evaluation

Routing is the part of this system most likely to regress silently: a prompt tweak that improves one kind of request can quietly break another, and nothing crashes when it does. The evals/ directory exists to catch that.

What is measured. orchestrator_dataset.json holds 60 hand-written scenarios that each pin the exact graph state the orchestrator node sees and check the single decision it makes from that state. Sub-agents are never invoked — the engine calls the production orchestrator node directly with no sub-agent, database, or vector-store calls in the loop. One LLM call per scenario, fast and deterministic. The scenarios cover five categories:

initial routing (20) — straight after START, does it pick the right sub-agent?
after a booking response (10) — booking output is present: return to the user, or keep going?
after a knowledge-base response (10) — same question for KB output
empty agent response (10) — an agent was called and returned nothing: retry, or fail honestly?
irrelevant (10) — greetings and out-of-scope asks no sub-agent should handle

How scenarios are graded. expected.decisions is a list of acceptable decisions, not a single golden answer — routing is a judgement call, and several scenarios have more than one defensible next step. A scenario passes if the produced decision matches any entry; agent names are compared as a set, so ordering doesn’t matter. When the orchestrator swallows an exception and returns no tool calls at all, that scores as a failure rather than being hidden.

Failure modes deliberately covered. Beyond happy-path routing, the dataset targets the ways an agent is dishonest rather than broken: relaying a booking conflict instead of claiming success, saying “not found” instead of inventing a policy answer, not claiming a booking or cancellation succeeded when no confirmation came back, and stopping once the retry budget is spent rather than looping.

The harness is exposed as /eval/* endpoints (GET /eval/dataset, one POST /eval/{category} per category), and a static browser client drives them — pick a category, preview the scenarios, hit play, read the pass/fail table.

What broke

The retry path became an infinite corridor. A rejected step could be sent back to the orchestrator, which — seeing the same state and the same tools — made the same call, producing the same rejected result. The retry counter was tracked in the wrong place and reset every time the graph re-entered the node. Moving it into the shared typed state made the loop terminate. The general lesson stuck: in a graph, anything that must be remembered across a cycle belongs to the state, not to the node.

Streaming and tool-calling disagreed about when a turn was over. The streaming endpoint closed when the final text token arrived, which for a turn ending in a tool call was before the work had actually happened — users saw a truncated answer and assumed a crash. The fix was to make the graph, not the model output, own the lifecycle of a turn: the SSE stream emits events per agent step and closes when the graph finishes, so live progress is real progress.

Where it stands