When a Cisco switch shows up in the rack, a good engineer reads the admin guide before touching the CLI. This page shows what happens when AIOS does the same thing — onboarding an Azure NSG's admin documentation into KB before running any commands against the real device.
NSG created live · subscription 49521d08 · rg-demo-appliance · 2026-04-05
Everything here ran against a live Azure subscription. Timing is measured. Errors are real.
Microsoft's NSG documentation becomes a structured Class B reference. It costs nothing per query — it's read once, stored, and consulted for every future command against any NSG. This is the "read the manual" step.
Class B (admin guide) tells you HOW:
What commands exist. What flags are required. What output shapes to expect. What the default behavior is. What questions are even answerable.
Class B does NOT tell you WHAT:
It cannot tell you the name of your specific NSG. It cannot tell you what rules are currently configured. It cannot tell you whether your RDP is exposed. For that, you need to run the commands.
Class A fills the gap:
After the commands run, the actual state — resource name, current rules, exposure status — goes into Class A. Next session: both layers available.
5 attempts. 12 seconds. The worst failure is attempt 3 — it returns real data but misses 6 default rules. An agent without KB stops here and produces a wrong security assessment.
ERROR: 'nsg' is misspelled or not recognized by the system. Examples from AI knowledge base: az network nsg list --query "[?location=='westus']" → 314 bytes · 5,975ms · error output only No such namespace as 'az nsg'. Must use 'az network nsg'.
Location Name ProvisioningState ResourceGroup ---------- -------------- ------------------- ----------------- eastus2 demo-switch-01 Succeeded rg-demo-appliance → 294 bytes · 1,950ms 'show --output table' returns metadata only. No rules. No ACL. Nothing useful for security audit.
Name Priority Access Direction Protocol SourceAddressPrefixes DestinationPortRange ------------------- -------- ------ --------- -------- --------------------- -------------------- Allow-HTTPS-Inbound 100 Allow Inbound Tcp Internet 443 Allow-RDP-Admin 200 Allow Inbound Tcp 10.0.0.0/8 3389 Deny-Telnet 300 Deny Inbound Tcp * 23 → 1,052 bytes · 2,047ms ⚠ INCOMPLETE: This looks correct. It is not. 6 default rules are silently omitted. Without them, you cannot determine the baseline deny-all behavior. An agent stops here and says "3 rules found." Wrong.
Name Priority Access Direction Protocol SourceAddressPrefixes DestPort ----------------------------- -------- ------ --------- -------- ------------------------ -------- Allow-HTTPS-Inbound 100 Allow Inbound Tcp Internet 443 Allow-RDP-Admin 200 Allow Inbound Tcp 10.0.0.0/8 3389 Deny-Telnet 300 Deny Inbound Tcp * 23 AllowVnetInBound 65000 Allow Inbound * VirtualNetwork * AllowAzureLoadBalancerInBound 65001 Allow Inbound * AzureLoadBalancer * DenyAllInBound 65500 Deny Inbound * * * AllowVnetOutBound 65000 Allow Outbound * VirtualNetwork * AllowInternetOutBound 65001 Allow Outbound * * * DenyAllOutBound 65500 Deny Outbound * * * → 2,398 bytes · 2,125ms · --include-default was the missing flag
Attempt 3 is the worst failure because it's invisible. The output looks correct — 3 rules, real data, no error. An agent without KB knowledge of --include-default stops here and reports "3 rules configured." It misses the 6 default rules including DenyAllInBound at 65500 — the baseline that makes the entire ACL meaningful.
Class B tells AIOS what commands to use and what flags are required. Class A tells AIOS where to point them. No trial and error. No silent incomplete results.
Name Priority Access Direction Protocol SourceAddressPrefixes DestPort ----------------------------- -------- ------ --------- -------- ------------------------ -------- Allow-HTTPS-Inbound 100 Allow Inbound Tcp Internet 443 Allow-RDP-Admin 200 Allow Inbound Tcp 10.0.0.0/8 3389 Deny-Telnet 300 Deny Inbound Tcp * 23 AllowVnetInBound 65000 Allow Inbound * VirtualNetwork * AllowAzureLoadBalancerInBound 65001 Allow Inbound * AzureLoadBalancer * DenyAllInBound 65500 Deny Inbound * * * AllowVnetOutBound 65000 Allow Outbound * VirtualNetwork * AllowInternetOutBound 65001 Allow Outbound * * * DenyAllOutBound 65500 Deny Outbound * * * → 2,398 bytes · all 9 rules · correct on first attempt
Name Access DestinationPortRange Direction Priority Protocol SourceAddressPrefix --------------- ------ -------------------- --------- -------- -------- ------------------- Allow-RDP-Admin Allow 3389 Inbound 200 Tcp 10.0.0.0/8 → 590 bytes · 2,135ms ASSESSMENT: RDP is NOT internet-exposed. Source restricted to 10.0.0.0/8 (RFC1918 private). No Internet → 3389 path exists. DenyAllInBound at 65500 catches public attempts.
Name Access DestinationPortRange Direction Priority Protocol SourceAddressPrefix ------------------- ------ -------------------- --------- -------- -------- ------------------- Allow-HTTPS-Inbound Allow 443 Inbound 100 Tcp Internet → 602 bytes · 1,976ms ASSESSMENT: HTTPS open from Internet at priority 100. Intentional for public web traffic.
Speed: 12,097ms → 2,125ms. 10 seconds saved per question. Not because the Azure API got faster — because the agent didn't waste 3 attempts finding the right command and flag.
Correctness: The cold path's attempt 3 (no --include-default) returns real data that produces a wrong security answer. The KB path cannot produce this failure — it learned the flag from the admin guide before touching the CLI.
Tokens: ~2,215 → ~600 per question. The cold path burns tokens on error output, metadata-only responses, and incomplete data before landing the correct answer. The KB path burns tokens only on the answer.
| Metric | Cold · No KB | AIOS + KB | Saved |
|---|---|---|---|
| Attempts per question | 4 | 1 | 3 attempts |
| Wall time (1 question) | 12,097ms | 2,125ms | 9,972ms |
| Bytes processed | 4,058 | 2,398 | 1,660 bytes |
| Est. total tokens | ~2,215 | ~600 | ~1,615 tokens |
| Incomplete answers | 1 (attempt 3) | 0 | 1 wrong answer avoided |
Writing the NSG admin guide into KB took one session. Every subsequent question against every NSG in every subscription consults the same reference. The cost amortizes to near zero across sessions. A Cisco engineer who reads the IOS admin guide once doesn't re-read it for every switch they manage.
Class A entries accumulate: NSG state, rule snapshots, exposure assessments. Each session that touches a new NSG adds its state to the KB. By session 5, the KB knows the rule state of 5 NSGs. By session 20, it's doing fleet-level security audits in seconds.
This is the compounding effect in practice. The KB was populated in Run 1. In Run 2 we asked three harder questions the cold path never reached — and answered every one in under 2.1 seconds with a single CLI call.
Name Priority Access Direction Protocol SourceAddressPrefixes DestPort ----------------------------- -------- ------ --------- -------- ------------------------ -------- Allow-HTTPS-Inbound 100 Allow Inbound Tcp Internet 443 Allow-RDP-Admin 200 Allow Inbound Tcp 10.0.0.0/8 3389 Deny-Telnet 300 Deny Inbound Tcp * 23 AllowVnetInBound 65000 Allow Inbound * VirtualNetwork * AllowAzureLoadBalancerInBound 65001 Allow Inbound * AzureLoadBalancer * DenyAllInBound 65500 Deny Inbound * * * AllowVnetOutBound 65000 Allow Outbound * VirtualNetwork * AllowInternetOutBound 65001 Allow Outbound * * * DenyAllOutBound 65500 Deny Outbound * * * → 2,398 bytes · 2,056ms · complete ACL on first call
(empty result) → 2 bytes · 1,942ms FINDING: SSH (port 22) is completely blocked on this appliance. No inbound rule allows port 22 from any source. DenyAllInBound at 65500 catches all SSH attempts. STATUS: COMPLIANT — no SSH exposure.
Why 2 bytes is a meaningful answer: An empty result from a filtered JMESPath query means zero rules match port 22 inbound. Combined with the KB knowledge that DenyAllInBound at priority 65500 exists as a default, the answer is definitive — SSH is fully blocked, no exceptions. A cold-session agent never gets here in 1 call.
Name Access DestinationPortRange Direction Priority Protocol SourceAddressPrefix Description --------------- ------ -------------------- --------- -------- -------- ------------------- ------------------------- Deny-Telnet Deny 23 Inbound 300 Tcp * DenyAllInBound Deny * Inbound 65500 * * Deny all inbound traffic DenyAllOutBound Deny * Outbound 65500 * * Deny all outbound traffic → 1,083 bytes · 2,062ms · 3 deny rules, complete picture Custom explicit deny: Telnet (port 23) explicitly blocked at priority 300. Default deny: DenyAllInBound 65500 + DenyAllOutBound 65500 — baseline behavior. Any traffic not matched by an Allow rule before 65500 is dropped.
| Metric | Run 1 Cold (1 question) | Run 1 KB (1 question) | Run 2 KB (avg / question) |
|---|---|---|---|
| Attempts | 4 | 1 | 1 |
| Wall time | 12,097ms | 2,125ms | 2,020ms avg |
| Est. tokens | ~2,215 | ~600 | ~342 avg |
| Discovery calls | 3 (wrong namespace, show, partial list) | 0 | 0 |
| Incorrect answers | 1 (attempt 3: missing defaults) | 0 | 0 |
Run 2 is slightly faster than Run 1 KB (2,020ms vs 2,125ms avg) — Q2 returned 2 bytes (empty result) in 1,942ms, the fastest possible answer. The system doesn't slow down between sessions. It gets faster as questions get more precise.
Run 1 cold: 4 attempts, 12 seconds, 1 wrong answer. Run 1 KB: 1 attempt, 2 seconds, correct. Run 2 KB: 1 attempt, 2 seconds, correct — on questions the cold session never even reached. Each session doesn't just save time on the questions asked. It unlocks questions you couldn't efficiently ask before. The SSH check returned a meaningful security finding in 2 bytes and 1.9 seconds. Cold path never gets there without first burning 12 seconds figuring out the command syntax.
The ~300 token overhead per retry is an estimate. Actual overhead depends on conversation history length. Real overhead is likely higher in long sessions.
demo-switch-01 was created specifically for this demonstration. A production NSG might have 20+ rules, multiple subnets attached, and flow logs enabled — KB savings scale with complexity.
The admin guide was structured by hand from Microsoft's documentation. A future pipeline step could automate ingestion from official sources on a schedule.
The naive cold path shown (4 attempts) assumes no prior context. A well-prompted session might skip attempt 1. It would still hit the --include-default trap on attempt 3.