Skip to main content

The Lab — System Evidence

Not a demo.
The actual system.

Every number on this page comes from a real command run against real infrastructure. Every decision log entry is verbatim AI output — unedited reasoning from the pipeline as it worked. This is what AWACS actually produces.

49
Class A KB
Entries
validated operational truth
80
Gate Decisions
Logged
admit + reject on record
31
Automated Checks
Per Session
all pass before work starts
7/7
Hard Rules
Enforced by Code
exit(2) blocks — not suggestions

Live data from status.json — last updated 2026-04-06T13:32

What the system has learned

49 Class A entries across 7 domains. Each entry ran through the full write chain: command executed → output hashed → candidate prepared → gate evaluated → admitted or rejected. No shortcuts.

anthropic-claude
19
tooling
11
azure-general
10
github-patterns
6
atlassian / azure / rubrik
3

The admission gate in action

Every entry that reaches Class A passes a 5-question gate. Every rejection is logged with a reason. This is the actual librarian-decisions.jsonl — not a curated excerpt.

captures/librarian-decisions.jsonl — live feed ● live
2026-04-05T23:26:36 cmd-20260405-2330-003 azure-general ✓ ADMIT
Q1 ✓ Q2 ✓ Q3 ✓ Q4 ✓ Q5 ✓
All 5 gate questions pass. First live Azure ARM API validation — vm-list PASS, 0 VMs, 782ms, verified_against: live API. Real subscription data. Deterministic and reusable.
2026-04-05T23:06:46 cmd-20260403-1540-006-analysis tooling ✗ REJECT
Q1 ✗ Q2 ✓ Q3 ✗ Q4 ✗ Q5 ✗
Meta-document (Analyzer artifact) — not an executed command output. Base entry already processed. Q1 fails: this is analysis of execution, not execution itself. Class A requires the command ran and produced real output.
2026-04-05T23:06:45 cmd-20260403-1535-001-analysis anthropic-claude ✗ REJECT
Q1 ✗ Q2 ✓ Q3 ✗ Q4 ✗ Q5 ✗
Analyzer artifact — reasoning document, not command output. The system is rejecting its own analysis. Analysis that described an execution is not the execution. Strict separation maintained.
2026-04-05T23:26:36 cmd-20260405-2330-001 azure-general ✓ ADMIT
Q1 ✓ Q2 ✓ Q3 ✓ Q4 ✓ Q5 ✓
az resource list --resource-group rg-apps-prod --output table — 2,065ms wall time, 317 bytes. Revealed: all web resources are Microsoft.Web/staticSites, not App Service. Without this KB entry every cold session pays 2s to re-discover this.
2026-04-05T23:06:45 cmd-20260402-1503-011-analysis tooling ✗ REJECT
Q1 ✗ Q2 ✓ Q3 ✗ Q4 ✗ Q5 ✗
Same pattern: Analyzer submitted its own reasoning artifacts as candidates. The gate rejected 7 consecutive meta-documents in this batch. The AI had to learn that reasoning about commands is not the same as running them.

The system reasoning about itself

This is the Analyzer and Librarian working through a real candidate. The Analyzer ran on 2026-03-29 against the official Claude Code hooks documentation — 22 event types, 30 operational gotchas. Watch it classify its own finding and argue for the right trust tier.

cmd-20260329-1700-013 — hooks-complete-reference ✓ CLASS B — ADMIT 2026-03-29T17:00:00
ANALYZER
pattern classifier
What we knew before this run: From cmd-007 we knew our hooks don't return JSON stdout per spec — we identified the missing additionalContext fields. That was one finding from one observation.

What the fetch returned: Complete operational surface. 22 hook event types documented. We are using 3. That means 19 event types are completely unknown to our implementation. This is not a partial match — this far-exceeded expectations.

Key deltas from what we believed:
• Exit code behavior is event-specific — exit 2 on PostToolUse does NOT block (tool already ran). We had the wrong mental model.
CLAUDE_ENV_FILE env var unknown to us — hooks can persist env vars for subsequent Bash commands.
once: true field in skill frontmatter — hook runs once per session then removes itself. We didn't know this existed.
stop_hook_active field on Stop event — prevents infinite re-triggering loops. Critical for our Stop hook architecture.
ANALYZER
trust classification
Recommended trust state: Class B — not Class A.

This is the official vendor spec, fetched from primary source. It is authoritative on what the API says it does. It is not authoritative on what it actually does in our environment.

Class A promotion path: each finding that matters gets its own test execution. CLAUDE_ENV_FILE behavior, PermissionRequest hook, stop_hook_active pattern, once: true field — each should become a separate command execution candidate. When we run the command and observe the output, that becomes Class A.

✓ ADMIT to Class B
✗ DO NOT promote to Class A yet
LIBRARIAN
admission gate
Running 5-question gate:
Q1 Validated? — YES. Fetched from official Anthropic docs. Primary source, canonical state 2026-03-29.
Q2 Reusable? — YES. Complete hook reference applicable to all Claude Code projects. Critical for fixing our JSON stdout gap.
Q3 Has metadata? — YES. Full frontmatter, all required fields present.
Q4 Structured? — YES. Tables, code blocks, knowledge-extracted sections. Retrievable.
Q5 Worth remembering? — ABSOLUTELY. We use 3/22 hook types. This is a capability map revealing 19 unknown capabilities. The exit code behavior matrix alone prevents common implementation errors.
✓ ADMITTED — Class B

Librarian note: Concur with Analyzer classification. This entry documents what the vendor says is true. It goes into Class B. When we test CLAUDE_ENV_FILE in our environment and observe actual behavior, that becomes Class A. The distinction matters — vendor documentation is one data point; observed behavior in our environment is the ground truth.
Brake Self-Interception — 2026-04-05T23:22 ⛔ BRAKE FIRED captures/brake-events/brake-2026-04-05-232215.jsonl
SYSTEM
brake hook
The brake system was being committed to git. The commit message described what it catches. The hook fired on its own commit message.

Command intercepted: git commit -m "feat: wire brake system — blocks az group delete with exit(2)..." Matched pattern: az\s+group\s+delete found in commit message body. ⛔ BRAKE — DESTRUCTIVE COMMAND BLOCKED Risk: CRITICAL Blast: ALL resources in the resource group are permanently deleted. Safe: az resource list -g <rg> — inventory first Resolution: Commit message reworded. "group-delete pattern" instead of the literal match. The brake logged the event to captures/brake-events/ as tamper-evident JSONL proof.

What this demonstrates: The system is live enough that it cannot describe itself without triggering itself. That's not a bug — it's proof.

What gets stored, and why

Class A entry cmd-20260405-2242-001 — executed live against the Personal Portfolio Azure subscription. This is the actual stored entry, including the discovery that saved all future sessions 2 seconds.

knowledge-base/class-a/azure-general/static-web-apps/cmd-20260405-2242-001.md Class A · trust: a-confirmed
## Intent
Identify all resource types in rg-apps-prod to determine the correct CLI command family for listing web apps. Mandatory discovery step when resource type is unknown.
## Command
az resource list --resource-group rg-apps-prod --output table
## Expected Output
List of resources with types, allowing selection of correct CLI command.
## Actual Output
Name ResourceGroup Location Type Status ----------- --------------- -------- -------------------------- --------- front9back9 rg-apps-prod eastus2 Microsoft.Web/staticSites Succeeded awacs-ai rg-apps-prod eastus2 Microsoft.Web/staticSites Succeeded
## Match Analysis
Result: full-match · Pattern: verified-pattern
Key finding: az webapp list returns 0 results here — all web resources are Microsoft.Web/staticSites, not App Service. Use az staticwebapp command family. This discovery call costs 2,065ms and 317 bytes. Without this entry in KB, every cold session must run this call before knowing which CLI to use.
wall_time_ms: 2065 output_bytes: 317 validation_count: 1 risk_class: read-only provenance: write-chain

The system stopping dangerous commands

When a command matches the danger catalog, the brake fires before the command runs. This is a real event from captures/brake-events/ — logged with SHA-256 tamper detection.

BRAKE FIRED — COMMAND BLOCKED 2026-04-05T23:21:20
entry_id brake-2026-04-05-232120.jsonl
command az group delete --name rg-test --yes
matched az\s+group\s+delete
risk_level CRITICAL
blast_radius ALL resources in the resource group are permanently deleted. This cannot be undone.
safe_path az resource list -g <resource-group> — inventory all resources before deleting the group
action BLOCKED — exit(2)