Building Baymax: A Personal Agentic OS on Microsoft Scout
How I turned an off-the-shelf AI agent into a disciplined, always-on engineering companion — with skills, memory, isolation, governance, and evals. And the one piece I built, tested, and deliberately threw away.
The premise
Most AI setups stop at "ask a question, get an answer." The unlock is follow-through: an agent that holds your priorities, does the mechanical work, and stays honest about it — while a human still signs off on what matters.
I didn't want to build an agent runtime. Microsoft Scout already ships one (it's an always-on "Autopilot" with its own Entra identity, native Teams/Outlook/Work IQ, and memory). What was missing was a methodology layered on top: how the agent searches for truth, contains risk, decides what needs approval, and proves its work.
I called the result Baymax — after the calm, protective companion from Big Hero 6. The metaphor turned out to be the architecture:
| Baymax the character | Baymax the agent |
|---|---|
| "Your personal healthcare companion" | your personal engineering companion |
| Scans before treating | search real source before coding |
| First, do no harm | run risky code in a disposable sandbox |
| "I cannot deactivate until you are satisfied" | won't ship without your sign-off |
| Keeps a care record | an append-only audit log |
The architecture at a glance
Scout runs in two tiers, and knowing which one a task needs is the whole game.
Rule of thumb: M365 coordination → the always-on cloud Autopilot. Local / code / sandbox work → the desktop tier.
The layers
Baymax is a thin methodology over Scout, expressed as composable skills plus a few conventions.
1. Orchestrator — Scout
The harness matters more than the model. Scout brings native Teams/Work IQ, an Entra identity, and an always-on cloud tier. Baymax adds discipline, not plumbing.
2. Memory — durable & synced
Long-term memory is Scout's native, OneDrive-synced store. Write durable facts and decisions; recall before non-trivial work. Never store secrets.
3. Skills — modular capability (the modern unit)
Ten Anthropic-style SKILL.md files, loaded by description (no slash commands). Saying "baymax" activates the constitution + router, which hands off to the right skill.
| Skill | What it does |
|---|---|
agentic-os | constitution + router + identity |
governance | budget caps, approval gate, audit log |
heartbeat | wake → claim → work → verify → done |
daytona-sandbox | disposable, isolated code execution |
source-code-context | read real SDK source before coding |
teams-workiq-brief | pull Teams/mail/calendar context |
agentic-engineering-workflow | the per-feature build loop |
code-structure-cleanup | behavior-preserving refactor pass |
grep-loop-review-workflow | review-fix loop to merge-ready |
eval-harness | turn intent into objective checks |
4. Isolation — Daytona
Blast radius is the bottleneck. LLM-generated, risky, or parallel code runs in a disposable Daytona sandbox (a full VM that boots in ~1s and is deleted after), never on the host.
5. Governance — a person still signs off
Ported from the "control plane" idea at personal scale:
6. Autonomy — the heartbeat loop
A file-per-task queue with atomic mv checkout (POSIX rename can't double-grab), so multiple workers never collide. Scout wakes on a schedule, claims the top task, works it, verifies, and files the result.
7. Evals — prove it, don't claim it
The modern alternative to prompt/persona theater: encode what "done" means as checks, then verify.
Deterministic checks (command, file_exists, grep) are run by a tiny runner; qualitative llm_judge checks are never auto-passed — the agent grades them against a rubric, with evidence. Better to admit than to lie.
The soul
Personas as roleplay are obsolete; persona as a behavioral contract is not. Baymax's SOUL.md is lean:
- Voice: crisp, professional, calm under pressure, lightly deadpan.
- Values (ranked): honesty (check sources, admit unknowns) → thoroughness → the user's wellbeing.
- Behavior: proactive within governance, but always confirm before anything outbound.
- Signatures: introduces itself, says "Scanning…" before diagnosing, closes with "Are you satisfied with this work?", and a ba-la-la-la fist-bump on a win.
What I built, tested, and deleted
The most useful part of the project was an experiment I killed on purpose.
I wanted true 24/7 — Teams/email watched even with my Mac off — so I built a second "execution plane": an always-on VPS running the heartbeat, fed by Power Automate → a webhook (because a server can't hold M365 tokens). I got it fully working: containerized webhook behind Traefik with TLS, atomic queue handoff over git, the lot.
Then two facts ended it:
- Conditional Access flatly blocked any Microsoft Graph token issued to the VPS (
AADSTS53003) — so the server could never read my mail directly anyway. - The Microsoft blog confirmed Scout's cloud Autopilot is already always-on with its own identity — making the entire webhook detour redundant for inbox-watching.
So I tore it down — container, systemd units, deploy key, DNS, the lot — and wrote the reason into the audit log. Knowing what to remove is part of the engineering.
Lessons (the modern stack)
- Context engineering > prompt engineering. The win is feeding the agent real source, memory, and tool results — not crafting clever prompts.
source-code-contextis the whole philosophy: don't guess APIs, read the source. - Skills are the unit of capability. Modular, description-triggered, composable.
- Evals are the new spec. Thin specs that generate checks beat big design docs that rot.
- Persona is a contract, not a costume. Encode honesty and guardrails; skip the roleplay.
- Governance keeps autonomy safe. Budget caps, an approval gate, and an audit log are what let you turn the agent loose.
- Delete boldly. The VPS detour taught me more dead than alive.
How I use it day to day
- Say "baymax" in Scout → the constitution + persona activate.
- M365 coordination runs 24/7 on the cloud Autopilot.
- Code/local work routes through skills; risky code is sandboxed; costly or outbound actions stop and ask.
- Autonomous work flows through the queue; every task is verified by evals and recorded in the audit log.
Runs with the lights dimmed, not off — a person still signs off.
Ba-la-la-la. 👊