Sizing & failure modes

How much hardware a collector needs, how to scale, and — most importantly — what happens to traffic when something fails. Plan the failure behavior before you roll out; it's a policy decision, not a default to accept blindly.

Where you do this: Mostly a reference / decision-making page — no step-by-step commands. Concrete actions when they come up:

• Offline alerting — in the dashboard at Settings → Notifications (see Monitoring & health).
• Fail-open PAC edit — hand-edit the PAC file on the collector host; append ; DIRECT to the PROXY <collector> line.
• Explicit-proxy fail-open / closed — set never_direct (closed) or its opposite on your existing forward proxy.

Sizing a collector

Size by request rate and body size, not seat count. Pick a tier, then sanity-check it against the agent-traffic note below.

Tier	Chat seats	Coding-agent seats	Peak concurrent	Required req/s	vCPU	RAM (gateway lane)	RAM (MITM lane)	Disk
Pilot	≤200	≤25	≤60	~4	4	8 GB	8 GB	40 GB
Department	≤1,000	≤120	~300	~19	8	8 GB	16 GB	100 GB
Enterprise site	≤3,000	≤330	~900	~52	16	16 GB	32 GB	200 GB
Beyond	>3,000/site	>330/site	—	—	add collectors — one per site/subnet

The enterprise tier is capped by measured throughput, not by RAM: ~40 req/s on an 8 vCPU box with 4 workers, extrapolating to roughly 100–150 req/s on 16 vCPU. 3,000 chat seats (or 330 agents) is where the required rate meets that ceiling with headroom intact. Larger sites add collectors — that is the supported scaling path, and it is how the product is designed to grow.

How the seat numbers are derived

peak concurrent sessions = seats × 0.30          (peak-hour active ratio)
required req/s           = concurrent × R × 1.5  (1.5 = headroom + N+1)

R = requests per second per active session:

Population	R	Why
Human chat (browser, IDE assistant)	0.02–0.05	a turn every 20–50 s, small bodies
Coding / headless agents (Claude Code, SDK apps)	0.20–0.50	bursty tool loops, large contexts

The two seat columns are the same capacity expressed for each population — roughly 8 chat seats ≈ 1 coding-agent seat. A tier's real limit is its req/s column; the seat counts are a planning convenience.

Minimum viable deployment: one 4 vCPU / 8 GB collector serves a pilot of ~25 coding agents or ~200 chat users. Below that, size is set by the floor (Docker + the image + audit headroom), not by load.

These figures assume multi-worker inspection — the default since 2026-08

Throughput was measured at ~40 req/s with 4 proxy workers on an 8 vCPU box, versus ~9–20 req/s single-worker — the proxy is one asyncio event loop and cannot use more than ~1 core on its own. New installs auto-size MITM_WORKERS to min(nproc-1, 4).

Collectors installed before this change still run single-worker and support roughly a quarter of the seats above. Check with grep MITM_WORKERS /etc/kilasec/collector.env — if the line is absent, retrofit it:

curl -fsSL https://kilasec.com/retrofit-perf | sudo sh

That also enables bounded decision concurrency (AGENTFW_PDP_MAX_CONCURRENCY), which changes overload from a fail-closed outage into fast shedding — see failure modes below. It is reversible with --revert.

Sanity-check against your own traffic before committing: the R values are planning defaults, and agent populations vary far more than chat ones.

Why two RAM columns. The MITM lane holds per-connection state (~95 KB measured), buffers full request/response bodies, and caches a forged certificate context per destination — the last of which is what drives comparable products to 32–48 GB. The gateway lane pools upstream connections and mints no per-connection certs, so a fleet that is predominantly gateway-lane sizes a tier lower on memory. Browsers can't use the gateway; size for whichever lane dominates, and use the MITM column for mixed fleets.

Agent traffic is the case that breaks a single seats number. 200 coding-agent hosts at R = 0.3 is ~27 req/s of steady load — comparable to ~1,700 chat seats, and it arrives in bursts with far larger bodies. That is why the table carries two seat columns instead of one. If your population is agents, size on measured req/s and body size, and treat the seat counts as a starting point only.

Network for every tier: outbound 443 to kilasec.com; LAN reach on :8080/:9443.

Actual consumption (not a provisioning target): ~573 MB idle with 4 proxy workers, ~95 KB per held connection, ~50 MB/day of audit log. The tiers above are sized for headroom, burst, and body buffering — provision the tier, not the idle figure.

A small VM, a NUC, or a Raspberry Pi 4/5 (arm64) handles a typical office.

The bottleneck under load is decision throughput — the CPU cost of scanning request bodies — not connections, file descriptors, or TLS handshakes. Measured: at 400 concurrent connections a collector still served only ~20 req/s while latency climbed to 9.3 s. Everything queues behind the processing rate. Size CPU to your request rate and body sizes, not your connection count or rule count.

Body size dominates. The decision path alone, measured in isolation:

Body	Decision latency	Serialized rate
1 KB	0.2 ms	~3,400 req/s
10 KB	1.7 ms	~585 req/s
100 KB	37 ms	~108 req/s
1 MB	186 ms	~11 req/s

This is why agent traffic is the sizing case, not seat count: AI agents send large contexts. A few hundred coding-agent hosts can outweigh thousands of chat users.

Lane benchmarks

Devices running Kilasec Connect reach the collector by one of two lanes. Both traverse the same collector and the same policy engine — the gateway hairpins through the proxy — so the choice changes carrying cost, never whether traffic is inspected.

Measured client-side against a LAN collector (POST /v1/messages, real provider round trips, p50, lanes interleaved, a fresh connection per request):

Body	Gateway	Proxy (MITM)	Direct (no inspection)
1 KB	136 ms (1.03×)	268 ms (2.04×)	131 ms
10 KB	168 ms (1.22×)	294 ms (2.13×)	138 ms
100 KB	442 ms (2.47×)	595 ms (3.33×)	179 ms

The gateway lane runs at roughly direct speed on small requests — +4 ms at 1 KB — while the MITM lane costs ~2× direct. The gap is ~130–155 ms per request at every body size.

Why: the MITM lane pays a full connection setup per connection — CONNECT to the local proxy, the device-TLS hop, then the collector forging a leaf certificate and completing a second TLS handshake. The gateway lane is plain HTTP on loopback (the bytes never touch a network interface) and reuses one pooled upstream tunnel.

Measure with a fresh connection per request. Terminal and headless agents are short-lived processes: each claude / curl / python invocation opens a new connection and pays that setup every time. A benchmark that reuses one pooled HTTP client amortizes the setup across many requests and makes the two lanes look equivalent — that is a measurement artifact, not a result. bench_gateway.py defaults to fresh connections for this reason; --reuse-conn models a long-lived server client instead.

What this does not show:

• The gateway does not raise a collector's req/s ceiling. Both lanes hairpin through the same collector and call the PDP once per request, so per-collector throughput capacity is unchanged. The gateway wins on per-request latency and on host memory, not on decisions per second. Size collectors on req/s and body size regardless of lane.
• Under high concurrency with pooled clients the two lanes converge on median throughput; the MITM lane also showed occasional severe tail stalls (p99 38–120 s) in that configuration, which is worth root-causing but is not the basis for the recommendation.

The gateway's other advantages are operational: no CA in any trust store, cert pinning can never break, and real JSON deny bodies.

Re-run these with dev/loadtest/bench_gateway.py.

Scaling: out, not up

Add more collectors rather than one large one:

• One per site / subnet is the common pattern — it keeps interception local and avoids routing AI traffic across your WAN just to inspect it.
• Each collector enrolls independently and appears separately on the Collectors page.
• Policy and scope are managed centrally in the cloud and pulled by every collector, so more collectors doesn't mean more policy work.

There's no shared-state requirement between collectors — they're independent enforcement points reporting to one control plane.

Failure modes — decide these deliberately

If the cloud (kilasec.com) is unreachable

The collector keeps enforcing policy locally — it already has the current rules on disk. Decisions are buffered to disk and flush to the cloud when connectivity returns. You lose live dashboard visibility during the outage, not enforcement. Policy/scope changes you make in the dashboard simply apply once the collector can pull them again.

If the collector itself is down

This is the important one, and the answer depends on your routing:

• PAC-based routing: the PAC Kilasec generates returns PROXY <collector> (no DIRECT fallback) for in-scope AI hosts — so it is fail-closed by default: if the collector is down, in-scope AI calls fail rather than slipping out uninspected. If you'd rather fail-open (AI keeps working, uninspected, during an outage), hand-edit the PAC to append ; DIRECT.
• Explicit-proxy chaining: behavior is whatever your upstream proxy does when the collector peer is unreachable — configure never_direct (fail-closed) or allow direct (fail-open) on the proxy side.

Choose based on your risk posture: fail-closed (the default) favors control — nothing reaches a provider uninspected; fail-open favors availability — a collector outage never breaks users' AI tools. Either way, pair it with offline alerting so you know immediately when a collector drops.

If the collector is overloaded

Distinct from being down, and the more likely one. Past its sustainable request rate a collector's decision queue grows; without a bound, queued requests eventually exceed the 75-second decision timeout, and because a non-answering decision service is treated as fail-closed, every timed-out request becomes a block. Load turns into an outage.

New installs set AGENTFW_PDP_MAX_CONCURRENCY=-1 (auto-sizes to cores−1), which bounds how many decisions run at once and sheds with an immediate 503 when saturated for longer than AGENTFW_PDP_QUEUE_TIMEOUT_MS (2 s). Requests over AGENTFW_PDP_LARGE_BODY_KB (256 KB) use a separate smaller lane, so a burst of multi-MB scans can't starve fast small decisions.

Shed requests still fail closed — that posture is unchanged — but they fail in two seconds instead of seventy-five, and the collector stays responsive for everything else instead of collapsing. Shed counters appear in /stats (load_shed) and the bound in /healthz.

Set it to 0 to disable shedding and restore unbounded queueing.

Restart / recovery

The collector runs under systemd with Restart=always; a crashed process comes back on its own. It buffers to disk and drains cleanly on shutdown, so a restart doesn't lose decisions.

Backups

Collector state is disposable — it re-enrolls and re-pulls policy. The thing worth backing up is the cloud database (your tenants, policy, audit history). If you self-host the cloud, ensure a nightly DB snapshot is in place; the Kilasec-hosted cloud handles this for you.

Monitoring is part of the plan

A collector you can't see is a collector you can't trust. Before rollout, make sure offline alerting is set up so a dead collector pages you rather than silently failing open. See Monitoring & health.