Joshua Bellew's Blog

Build a Basic AI Agent in TypeScript, Part 4: Production Readiness and Operational Guardrails

Mon, 02 Feb 2026 00:00:00 GMT

Overview

Part 1 covered the loop, Part 2 covered tool calling, and Part 3 covered multi-step skill orchestration. This final part is about operating an agent in production, not just getting good outputs in a demo. The goal is simple, define guardrails before the agent can touch real systems, real data, and real users.

TLDR: Production agent work is operational design. Define what the agent can execute, require approval for high-impact actions, log decision paths, evaluate route quality over time, and make ownership explicit before launch.

Full Series

Part 1, The Core Loop
Part 2, Tool Calling and LLM Intent Routing
Part 3, Multi-Step Skill Orchestration
Part 4, Production Readiness and Operational Guardrails (you are here)

Note: This is not an exhaustive guide to distributed systems. It is a practical write-up based on personal tinkering with multi-step orchestration agents.

I have been exploring agents and LLMs since ChatGPT first launched, and I have spent a lot of time building small agents and multi-step orchestrations. There are excellent models available now, but moving an agent from a simple loop to something production-ready is a big jump. That jump is less about model quality, and more about applying the same software engineering basics we already rely on in distributed systems.

Sections

Why Production Readiness Is a Different Problem
Workflow Orchestration vs Single-Step Skill Execution
Safety Boundaries and Action Classification
Security, Access Control, and PII Handling
Human Approval Gates for High-Impact Actions
Idempotency and Duplicate Action Prevention
Observability Primitives You Need on Day One
Evaluation Loops: Replay, Route Accuracy, and Drift
Reliability Policies: Retries, Timeouts, and Rate Limits
Operational Readiness: Runbooks, Alerts, and Rollback
Ownership Model and Cross-Functional Rollout
Production Readiness Checklist
Final Takeaway
Official References

Why Production Readiness Is a Different Problem

Prototype quality and production quality optimize for different outcomes.

Prototypes optimize for speed and learning.
Production systems optimize for safe, repeatable outcomes.
In production, failure handling is part of the feature.

Example: a demo may show one successful tool call. Production requires retries, timeout limits, approvals, and rollback when that same call fails or duplicates.

A working demo is useful evidence, but it is not an operational design.

Workflow Orchestration vs Single-Step Skill Execution

Single-step skill execution handles one bounded task, for example, fetch one record and return it.

Workflow orchestration coordinates multiple dependent steps across systems. Each step can fail, retry, or require escalation.

The distinction matters because risk compounds across steps.

Example:

Step 1 reads account status.
Step 2 writes a billing adjustment.
Step 3 sends an external email confirmation.

Each step is understandable alone. The full chain can create user impact, billing impact, and support impact.

Safety Boundaries and Action Classification

Before launch, classify actions by impact.

Read-only actions, low risk.
Internal writes, medium risk.
External writes, billing, identity, or irreversible actions, high risk.

This classification should drive policy, not model preference. The model can suggest an action. Policy decides whether execution is allowed.

Security, Access Control, and PII Handling

Security boundaries should be explicit before launch.

Apply least privilege to tool access.
Scope tool permissions by role, tenant, and environment.
Enforce authorization checks in the runtime, not only in prompts.
Deny by default, then allowlist required capabilities.

PII handling matters just as much as execution control.

Redact or tokenize sensitive fields before logs are written.
Keep replay datasets sanitized and access-controlled.
Set retention rules for transcripts and execution traces.
Avoid copying raw PII into prompts unless required for task completion.

Example: if a support transcript includes full card details and you replay it in a lower environment, you create a second security problem while debugging the first one.

Observability and replay need controls. Logging everything without controls creates new risk.

Human Approval Gates for High-Impact Actions

Human approval is a control for high-consequence operations.

Good approval gates include:

Clear summary of intended action.
Input values and affected targets.
Reason the model selected this path.
Explicit approve or reject outcome with audit trace.

Example: "Refund $2,400 to account 9182" should never execute from model confidence alone. It should require explicit approval with a recorded decision.

Idempotency and Duplicate Action Prevention

Agents run in distributed systems where retries or duplicate execution can and will happen.

Idempotency keys turn repeated requests into one logical operation.

Example: if a network timeout triggers a retry after a charge request, idempotency prevents a second charge.

Observability Primitives You Need on Day One

At minimum, each execution should emit:

request id and correlation id,
selected skill or handler,
tool calls made,
latency,
token cost,
final status.

These fields answer three operational questions quickly: what happened, where it failed, and who owns the fix.

Evaluation Loops: Replay, Route Accuracy, and Drift

Agent quality can degrade quietly if behavior is not evaluated over time.

A lightweight evaluation loop can start with:

transcript replay tests,
expected route assertions,
periodic route accuracy tracking,
review of false-positive and false-negative routing outcomes.

Example: if route accuracy drops after a prompt update, the issue might be policy wording, not model capability.

Evaluation is for model quality, policy quality, and workflow design quality.

Reliability Policies: Retries, Timeouts, and Rate Limits

Reliability policies should be explicit and per step.

Retry only transient failures.
Apply timeout budgets per operation.
Enforce rate limits at skill and system boundaries.

Example: without a timeout budget, one slow dependency can block worker capacity and cascade latency across unrelated requests.

These controls are baseline protection against runaway failure modes.

Operational Readiness: Runbooks, Alerts, and Rollback

Production readiness requires operational expertise.

Define alert thresholds for failure rate and latency.
Maintain runbooks with triage and escalation paths.
Predefine rollback strategy for bad prompts, bad routes, or bad policy changes.

Example: if a routing change sends 30 percent of requests to the wrong skill, rollback should be one documented action, not an incident-time debate.

Ownership Model and Cross-Functional Rollout

Agent systems span product, engineering, operations, security, and support.

Set clear ownership boundaries:

Who owns routing logic.
Who owns policy changes.
Who approves high-risk skill updates.
Who handles incidents.

Clear ownership shortens incident response and reduces policy drift.

Production Readiness Checklist

Workflow scope and action risk classes are documented.
Human approval exists for high-impact actions.
Tool access is scoped by role and least-privilege policy.
Idempotency strategy is implemented for side-effecting operations.
Observability fields are emitted for every execution.
Logging and replay datasets are sanitized for PII.
Replay-based evaluation runs on real transcripts.
Retry, timeout, and rate-limit policy is defined per step.
Alerting, runbooks, and rollback paths are tested.
Ownership and escalation paths are clear.

Final Takeaway

Operating agents in production follows the same principles as any distributed system: observability, guardrails, reliability policy, rollback planning, and clear ownership.

Official References

Book Review: Latency by Pekka Enberg

Tue, 25 Nov 2025 00:00:00 GMT

Overview

Latency: Reduce delay in software systems by Pekka Enberg is a comprehensive guide that bridges latency optimization with distributed systems design. The book demonstrates how core latency concepts are tightly interwoven with what many consider best practice distributed systems techniques, revealing connections that aren't immediately obvious.

What Clicked

The book starts slow (no pun intended) but gradually builds momentum, introducing latency techniques that overlap and closely intertwine with fundamental distributed systems concepts. What surprised me most was discovering how many "obvious" best practices are actually deeply connected to latency optimization principles.

The progression from high-level system design to CPU-level mechanics strikes the perfect balance, deep enough to be valuable without getting lost in either abstraction or minutiae.

Practical Value

Standout Sections

The caching chapter provided exceptional clarity with easy-to-grasp explanations. While the middle sections initially felt disjointed due to what at times felt like a lesson in distributed systems instead of latency, persevering paid off as everything connected together when I realized the author was making it extremely obvious how they are tightly coupled and overlap with latency techniques.

The book covers impressive ground:

System colocation through CPU-level caching
CPU cache coordination and multi-cache synchronization
Locking mechanisms and concurrency patterns
Latency hiding techniques

Each topic provides enough depth to understand the concepts and research further if needed.

What Makes This Book Different

Most technical books teach the "what" and "how" but skip the crucial "when". This book excels at explaining when to apply specific techniques, making it immediately practical rather than just theoretical.

This focus on context and trade-offs transforms it from a reference manual into a decision-making guide that engineers at any level would benefit from, either from an education point of view or ability to apply at their place of work.

A Note on Rust

The code examples use Rust, which makes perfect sense for demonstrating locks, thread spawning, and low-level concurrency patterns. Coming from a TypeScript background with limited Rust experience (just the online Rust book and Rustlings), the examples required concentration to fully grasp.

For readers unfamiliar with Rust, I would expect this to be a lot harder to grasp if you want to follow along with the code examples. That said, Rust is the right choice here, although Go would be a great language for demonstrating these same concepts and perhaps a lot easier to read for those coming from more dynamic languages.

Final Thoughts

This is a book I'll read again and regularly return to for refreshing specific topics. It's valuable both as a comprehensive learning resource and a reference guide for specific latency optimization techniques.

Rating: 5/5

Would recommend for: Engineers working on distributed systems, performance-critical applications, or anyone wanting to deeply understand the relationship between latency and system design. Some programming experience required; Rust familiarity helpful but not essential.

Why I Write About Systems That Break

Tue, 30 Sep 2025 23:00:00 GMT

Why this blog exists

Production systems fail in ways that documentation never covers. Over 6+ years of building and operating distributed systems, the most valuable lessons I have learned came from real outages, real architectural trade-offs, and real debugging sessions. This blog is where I write those lessons down.

I cover three areas:

Distributed systems and reliability. Deep dives into outages and RCAs from companies like AWS and Cloudflare, breaking down what went wrong, why recovery was hard, and what we can take from it.
AI agent architectures. Building AI agents from scratch in TypeScript, covering tool routing, multi-step orchestration, and production readiness.
Backend engineering at scale. The decisions and patterns that matter when your system handles millions of users, processes payments globally, or needs to stay up at 3am.

Most engineering blogs stay surface-level. I aim for the opposite. Every post goes deep enough that you walk away understanding not just what happened, but why.

Blogs I read and recommend

These writers set the standard for the kind of technical depth I am aiming for:

Build a Basic AI Agent in TypeScript, Part 3: Multi-Step Skill Orchestration

Fri, 30 Jan 2026 00:00:00 GMT

Overview

Part 1 established the core loop, and Part 2 added model-driven tool calling. Part 3 focuses on the next mental model shift: from one tool at a time to coordinated skills with clear execution boundaries. The goal is not to build a full production platform. It is to understand the core concepts you need before scaling complexity.

TLDR: Tools execute one action. Skills coordinate related actions. A useful agent needs simple routing plus lightweight patterns for safe, observable, and controllable execution.

Full Series

Part 1, The Core Loop
Part 2, Tool Calling and LLM Intent Routing
Part 3, Multi-Step Skill Orchestration (you are here)
Part 4, Production Readiness and Operational Guardrails

Sections

What Part 3 Adds Beyond Part 2
Core Concept 1, Skills vs Tools
Claude Skills Example, How Skills Plug Into the Loop
Core Concept 2, Simple Skill Routing
Core Concept 3, Basic Multi-Step Orchestration
Core Concept 4, Safe Observable Controllable Execution
TypeScript Pseudocode, Multi-Step Skills in the Agent Loop
Common Pitfalls
Lessons Learned
Series Wrap-Up
Official References

What Part 3 Adds Beyond Part 2

Part 2 solved one key problem: how to let the model request a tool and return results in a loop.

Part 3 solves a different problem: how to organize many tool-capable behaviors without turning runtime logic into spaghetti.

In practical terms:

Part 2 answer: "Can the model call a tool?"
Part 3 answer: "Can the runtime choose and coordinate the right skill safely?"

Core Concept 1, Skills vs Tools

A tool is a single deterministic capability, for example get_weather(location).

A skill is a higher-level unit of behavior that can:

decide which tool(s) to use,
control sequence across steps,
normalize outputs for the agent response.

Think of it this way:

Tool  = one action adapter
Skill = a small workflow that may use one or more tools

This distinction matters because users ask for outcomes, not function calls.

Claude Skills Example, How Skills Plug Into the Loop

If you want a concrete implementation model, Claude Skills are a useful reference because the pattern is explicit and easy to reason about.

High-level shape:

Skills live on disk as SKILL.md files.
The SDK discovers them from configured locations.
The model invokes them when descriptions match user intent.
Your runtime still controls tool access and loop boundaries.

What a skill contains:

my-skill/
├── SKILL.md        (required)
├── scripts/        (optional, executable helpers)
├── references/     (optional, docs loaded as needed)
└── assets/         (optional, templates, fonts, icons)

Two ideas matter most for agent loops: multi-step execution and discovery.

Progressive Discovery

Skills are designed so the model can discover the right behavior without loading everything every turn:

Frontmatter in SKILL.md supports discovery and routing.
The SKILL.md body is loaded when the skill is relevant.
Linked files are opened only when needed.

This keeps context focused while still supporting deeper workflows.

Why This Helps Multi-Step Flows

For multi-step requests, skills encode repeatable workflow guidance, not only single tool calls.

Connectors and tools define what the runtime can do.
Skills define how the runtime should do it across steps.
Multiple skills can compose in the same session when tasks overlap.

This is why skills improve consistency. The model is not re-inventing the process from scratch every prompt.

Minimal SKILL.md example:

---
name: order-support
description: Handle order lookup, shipping-change eligibility checks, and guided next steps.
---

# Order Support Skill

## When to Use
- User asks about order status
- User asks to change shipping after checkout

## Instructions
1. Extract order id from user text.
2. Check order status.
3. Check change-window policy.
4. Ask for confirmation before submitting change requests.
5. Return a concise final response.

Then wire it into a custom agent loop in TypeScript:

// Startup phase, discover skills once and keep a lightweight index.
const skillIndex = loadSkillsFromFilesystem({
  cwd: "/path/to/project",
  settingSources: ["project", "user"]
});

async function runAgentLoop(): Promise<void> {
  while (true) {
    const userText = await readUserInput();
    if (!userText || userText.trim() === "") break;

    // Build context with conversation history, tool definitions, and skill metadata.
    const plan = await llmPlan({
      messages: conversation,
      tools: toolDefinitions,
      skillMetadata: skillIndex.metadata
    });

    // Progressive discovery, load the full skill only when selected.
    const activeSkill =
      plan.type === "use_skill"
        ? skillIndex.loadFullSkill(plan.skillName)
        : null;

    // Execute tool calls with runtime controls.
    const result = await executePlan({
      plan,
      activeSkill,
      toolRegistry,
      maxSteps: 6,
      requireConfirmationFor: ["submit_change_request"]
    });

    conversation.push({ role: "assistant", content: result.responseText });
    render(result.responseText);
  }
}

This fits the same architecture we built in this series. The loop remains simple. Skills help the model discover and compose multi-step behavior, while your runtime still controls actual tool execution and safety policy.

Note: In Claude SDK usage, include settingSources and allow the Skill tool so skill discovery and invocation can happen from filesystem locations.

Core Concept 2, Simple Skill Routing

Routing is the decision of which skill handles a request.

In a minimal architecture, routing can stay very simple:

The model classifies intent.
The runtime maps intent to one skill.
The skill executes and returns a structured result.

When two skills overlap, use explicit precedence rules. For example, "account safety" can outrank "general account info". This keeps behavior predictable even when user prompts are ambiguous.

Core Concept 3, Basic Multi-Step Orchestration

Orchestration means coordinating multiple internal steps to complete one user request.

Example flow:

User asks: "Can you check order 4821 and tell me if I can still change shipping?"
    -> Route to OrderSupportSkill
    -> Step 1: fetch order status
    -> Step 2: evaluate policy rules
    -> Step 3: return user-facing answer with next action

This is the conceptual jump from Part 2. The runtime is no longer just "tool call in, tool result out". It now executes a small workflow with clear boundaries.

Core Concept 4, Safe Observable Controllable Execution

This is the most important concept in this part.

Safe

Safe execution means the agent does not perform unintended or duplicated actions.

Lightweight patterns:

validate required inputs before side-effecting actions,
use idempotency keys for action-like operations,
require explicit confirmation for high-impact actions.

Observable

Observable execution means operators can answer, "What happened, where, and why?"

Minimal per-request fields:

request id,
selected skill,
tool calls made,
latency,
final status.

Controllable

Controllable execution means runtime policy stays in code, not hidden in model output.

Simple controls:

max skill steps per request,
allowlist of executable tools per skill,
clear fallback path when routing is uncertain.

You can implement these controls without building a large framework. Even minimal guardrails dramatically improve reliability and debuggability.

TypeScript Pseudocode, Multi-Step Skills in the Agent Loop

Note: This is simplified pseudocode to illustrate the concept, not actual implementation.

type Route = "weather" | "order_support" | "fallback";

const toolRegistry = {
  lookup_order: async (args: { orderId: string }) => {
    // Calls your real TypeScript tool function.
    return { ok: true, data: { id: args.orderId, status: "PROCESSING", placedAt: "2026-01-21T10:11:12Z" } };
  },
  check_change_window: async (args: { orderStatus: string; placedAt: string }) => {
    // Encapsulates policy lookup logic.
    return { ok: args.orderStatus === "PROCESSING", data: { isEligible: true } };
  },
  submit_change_request: async (args: {
    orderId: string;
    newAddress: string;
    idempotencyKey: string;
  }) => {
    // Side-effecting operation guarded by idempotency key.
    return { ok: true, data: { ticketId: "chg_12345" } };
  }
};

async function runOrderSupportSkill(input: {
  requestId: string;
  userText: string;
  userConfirmed: boolean;
  requestedAddress: string;
}): Promise<{ status: "success" | "failed"; responseText: string }> {
  const state: Record<string, unknown> = {
    userText: input.userText,
    userConfirmed: input.userConfirmed,
    requestedAddress: input.requestedAddress
  };

  // Step 1: Extract order id (could come from model extraction + validation).
  state.orderId = "4821";

  // Step 2: Call lookup tool.
  const order = await toolRegistry.lookup_order({
    orderId: String(state.orderId)
  });
  if (!order.ok) {
    return { status: "failed", responseText: "Order lookup failed." };
  }
  state.order = order.data;

  // Step 3: Call policy tool.
  const policy = await toolRegistry.check_change_window({
    orderStatus: String((state.order as { status: string }).status),
    placedAt: String((state.order as { placedAt: string }).placedAt)
  });
  if (!policy.ok) {
    return { status: "failed", responseText: "Policy check failed." };
  }
  state.isEligible = policy.data?.isEligible === true;

  // Step 4: Optionally call side-effecting tool.
  if (state.isEligible === true && state.userConfirmed === true) {
    const submit = await toolRegistry.submit_change_request({
      orderId: String(state.orderId),
      newAddress: String(state.requestedAddress),
      idempotencyKey: `${input.requestId}:shipping-change`
    });
    if (!submit.ok) {
      return { status: "failed", responseText: "Change request failed." };
    }
    state.changeTicket = submit.data?.ticketId;
  }

  // Step 5: Compose final response.
  const responseText =
    state.isEligible === true
      ? state.userConfirmed === true
        ? `Shipping change submitted. Ticket: ${state.changeTicket}`
        : "Your order is eligible for shipping changes. Confirm to continue."
      : "This order can no longer be changed under current policy.";

  return { status: "success", responseText };
}

class AgentRuntime {
  async handleRequest(input: {
    requestId: string;
    userText: string;
    userConfirmed: boolean;
    requestedAddress: string;
  }): Promise<string> {
    const route = await this.routeIntent(input.userText);
    const startedAt = Date.now();

    const result =
      route === "order_support"
        ? await runOrderSupportSkill(input)
        : { status: "success", responseText: "Fallback response." as const };

    this.logExecution({
      requestId: input.requestId,
      route,
      latencyMs: Date.now() - startedAt,
      status: result.status
    });

    return result.responseText;
  }

  private async routeIntent(_userText: string): Promise<Route> {
    // Model-assisted route selection.
    return "order_support";
  }

  private logExecution(event: {
    requestId: string;
    route: Route;
    latencyMs: number;
    status: "success" | "failed";
  }): void {
    // Structured logs for observability.
  }
}

The key learning is that one skill can run an ordered sequence of steps, and each step can decide whether to call zero, one, or many tools. The agent loop still stays simple: route request, run skill, log outcome, and return response.

The important part is not the syntax. The important part is separation of concerns:

route selection,
multi-step skill execution,
structured execution logging.

Common Pitfalls

Treating every behavior as a tool

This creates large, hard-to-manage tool surfaces. Group related behavior into skills instead.

Hidden routing rules

If routing logic is not explicit, behavior drifts and becomes hard to debug. Keep conflict handling and fallback rules visible.

No execution record

Without basic logs, misroutes and failures are guesswork. Always emit minimal structured execution events.

Lessons Learned

Part 1 gave the loop, Part 2 gave tool calling, Part 3 gives runtime structure.
Skills are the right abstraction for multi-step outcomes.
A small set of guardrails can make agent behavior much easier to trust and operate.
Safe, observable, controllable execution is a design choice, not an optional add-on.

Series Wrap-Up

You now have a complete progression:

Core loop and state management.
Tool calling and model-driven routing.
Skill orchestration and execution guardrails.

This is a practical baseline for building agent systems that stay understandable as they grow.

If you want to continue the series with production operations depth, Part 4 can focus on deployment guardrails, human approval design, incident readiness, and long-term quality evaluation.

Official References

Cloudflare December 5th 2025 Outage

Fri, 05 Dec 2025 00:00:00 GMT

Overview

On December 5th, 2025, Cloudflare experienced a 25-minute outage affecting approximately 28% of HTTP traffic.

TLDR: A combination of a configuation change, plus a killswitch (feature flag), to disable certain code paths from being executed, caused code to try and access object properties that were nil or null values.

This follows a recent previous outage on 18th Nov 2025 that was also a code exception that was caused by assumptions about the existence of the underlying data / memory being accessed -> https://blog.cloudflare.com/18-november-2025-outage/

Official post-mortem

How Requests Flow Through Cloudflare's WAF

When a request hits Cloudflare, the WAF evaluates it against a set of rules to decide whether to block, allow, or modify it.

   User's Browser
        │
        │  GET https://example.com/api/users
        ▼
   ┌─────────────────────────────────────────┐
   │         Cloudflare Edge Server          │
   │                                         │
   │  "Check if this request is malicious    │
   │   before forwarding to origin"          │
   └─────────────────────────────────────────┘
        │
        ▼
   ┌─────────────────────────────────────────┐
   │             WAF Ruleset                 │
   │                                         │
   │  Rule 1: If SQL injection → BLOCK       │
   │  Rule 2: If XSS attack   → BLOCK        │
   │  Rule 3: EXECUTE test-ruleset  ◄────────┼── Special rule
   │  Rule 4: If bot traffic  → CHALLENGE    │
   │                                         │
   └─────────────────────────────────────────┘

Most rules have simple actions like block or log. The execute action is special. It triggers evaluation of another ruleset. Cloudflare uses this internally to test new rules before releasing them publicly.

The Two-Pass Architecture

Based on the extremely brief code snippet in the blog post, coupled with the explanation, I'm going to assume cloudflare's code processes rules in two separate passes:

class WAFProcessor {
	async processRequest(request: Request) {
		const ruleset = this.loadRuleset();
		const ruleResults: RuleResult[] = [];
		const subRulesetResults: SubRulesetResult[] =
			[];

		// PASS 1: Evaluate each rule
		for (const rule of ruleset.rules) {
			const result = await this.evaluateRule(
				rule,
				request,
				subRulesetResults
			);
			ruleResults.push(result);
		}

		// PASS 2: Stitch sub-ruleset results back onto rule results
		this.attachResults(
			ruleResults,
			subRulesetResults
		);

		return ruleResults;
	}
}

Why two passes? Sub-rulesets might run in parallel, or results might be stored separately for memory efficiency. Either way, the results get attached in a second loop:

Pass 1: Evaluate rules, store results separately
─────────────────────────────────────────────────

   ruleResults[]              subRulesetResults[]
   ┌────────────────────┐     ┌─────────────────┐
   │ Rule 1 (execute)   │────▶│ Sub-results 0   │
   │ results_index: 0   │     ├─────────────────┤
   ├────────────────────┤     │ Sub-results 1   │
   │ Rule 2 (execute)   │────▶│                 │
   │ results_index: 1   │     └─────────────────┘
   ├────────────────────┤
   │ Rule 3 (block)     │
   │ (no sub-results)   │
   └────────────────────┘


Pass 2: Attach results back to rule objects
─────────────────────────────────────────────

   for each ruleResult:
       if action == "execute":
           ruleResult.execute.results = subRulesetResults[index]

The Bug

The bug lived in how these two passes communicated. Let's look at the full code:

interface RuleResult {
	ruleId: string;
	action: "block" | "log" | "execute" | "skip";
	execute?: {
		results_index: number;
		results?: SubRulesetResult;
	};
}

class WAFProcessor {
	async evaluateRule(
		rule: Rule,
		request: Request,
		subRulesetResults: SubRulesetResult[]
	): Promise<RuleResult> {
		// Check if killswitch is active for this rule
		if (this.isKillswitched(rule.id)) {
			// Skip the action, but still return result metadata
			return {
				ruleId: rule.id,
				action: rule.action, // ← Still says "execute"!
				// execute: ???      // ← Never created!
			};
		}

		// Normal execution for "execute" action
		if (rule.action === "execute") {
			const subResults = await this.runSubRuleset(
				rule.targetRuleset,
				request
			);
			const index =
				subRulesetResults.push(subResults) - 1;

			return {
				ruleId: rule.id,
				action: "execute",
				execute: {
					// ← This object gets created
					results_index: index,
				},
			};
		}

		// Handle other actions...
	}

	attachResults(
		ruleResults: RuleResult[],
		subRulesetResults: SubRulesetResult[]
	) {
		for (const result of ruleResults) {
			// THE BUG: Assumes execute object exists if action is "execute"
			if (result.action === "execute") {
				result.execute.results =
					subRulesetResults[
						result.execute.results_index
					];
				//     ▲
				//     └── 💥 result.execute is undefined when killswitched!
			}
		}
	}
}

The two functions had incompatible assumptions:

┌─────────────────────────────────────────────────────────────┐
│                                                             │
│  evaluateRule thinks:                                       │
│    "I'll skip the action but keep the action type           │
│     so logs show what kind of rule it was"                  │
│                                                             │
│  attachResults thinks:                                      │
│    "If action is 'execute', someone definitely ran it       │
│     and created the execute object"                         │
│                                                             │
│  These assumptions worked fine for years...                 │
│  until someone killswitched an "execute" rule.              │
│                                                             │
└─────────────────────────────────────────────────────────────┘

How the Code Could Be Structured

Option 1: Change the action type when skipped

The simplest fix. If you skip a rule, don't claim it's still an "execute" action:

async evaluateRule(rule: Rule, request: Request): Promise<RuleResult> {
    if (this.isKillswitched(rule.id)) {
        return {
            ruleId: rule.id,
            action: "skipped",        // ← Changed from rule.action
            originalAction: rule.action,  // ← Keep for logging
        };
    }
    // ...
}

Option 2: Check for the object before accessing it

Defensive programming in the second pass:

attachResults(ruleResults: RuleResult[], subRulesetResults: SubRulesetResult[]) {
    for (const result of ruleResults) {
        if (result.action === "execute" && result.execute) {
            //                             ▲
            //                             └── Guard clause
            result.execute.results = subRulesetResults[result.execute.results_index];
        }
    }
}

Option 3: Eliminate the second pass entirely

Why store an index and look it up later? Just attach the results immediately:

async evaluateRule(rule: Rule, request: Request): Promise<RuleResult> {
    if (this.isKillswitched(rule.id)) {
        return { ruleId: rule.id, action: "skipped" };
    }

    if (rule.action === "execute") {
        const subResults = await this.runSubRuleset(rule.targetRuleset, request);

        return {
            ruleId: rule.id,
            action: "execute",
            execute: {
                results: subResults,  // ← Attach immediately, no second pass
            },
        };
    }
    // ...
}

This eliminates the bug entirely because there's no second pass with mismatched assumptions.

Option 4: Use types to make invalid states unrepresentable

The real fix. Design your types so the bug can't exist:

// Instead of one type with optional fields...
type RuleResult =
	| { action: "block"; ruleId: string }
	| { action: "log"; ruleId: string }
	| {
			action: "skipped";
			ruleId: string;
			originalAction: string;
	  }
	| {
			action: "execute";
			ruleId: string;
			execute: { results: SubRulesetResult };
	  };
//    ▲
//    └── If action is "execute", execute MUST exist.
//        TypeScript enforces this at compile time.

function attachResults(result: RuleResult) {
	if (result.action === "execute") {
		// TypeScript KNOWS result.execute exists here
		// No runtime check needed, no bug possible
		console.log(result.execute.results);
	}
}

Cloudflare noted that this bug didn't occur in their Rust-based FL2 proxy. Rust's type system would force you to handle the "skipped" case explicitly. You can't just leave the execute field undefined and hope for the best.

Why Increase the Request Buffer Size?

What is a Request Buffer?

When you submit a form or upload data to a website, that data travels in the HTTP request body. Before Cloudflare forwards the request to the origin server, the WAF needs to scan it for malicious content like SQL injection or XSS payloads.

But scanning happens in memory. Cloudflare can't just stream the body through, it needs to hold it somewhere to analyze it. That "somewhere" is the request buffer.

┌──────────────────────────────────────────────────────────────────┐
│                        HTTP Request                              │
├──────────────────────────────────────────────────────────────────┤
│  Headers: POST /api/submit                                       │
│           Content-Type: application/json                         │
│           Content-Length: 500000                                 │
├──────────────────────────────────────────────────────────────────┤
│  Body: { "data": "...500KB of content..." }                      │
│                                                                  │
│        ┌─────────────────────────────────────────────────────┐   │
│        │                  Request Buffer                     │   │
│        │                                                     │   │
│        │  WAF loads body here to scan for:                   │   │
│        │  • SQL injection patterns                           │   │
│        │  • XSS payloads                                     │   │
│        │  • Known exploit signatures                         │   │
│        │  • Malicious file uploads                           │   │
│        │                                                     │   │
│        │  Buffer size: 128KB (old) → 1MB (new)               │   │
│        └─────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────┘

Why 1MB?

Cloudflare was responding to CVE-2025-55182, a critical vulnerability in React Server Components. From their post:

"We started rolling out an increase to our buffer size to 1MB, the default limit allowed by Next.js applications. We wanted to make sure as many customers as possible were protected."

The exact behavior for content beyond the 128KB buffer limit isn't documented. It could be truncated, streamed without scanning, or handled some other way. What we do know is that increasing the buffer to match Next.js's 1MB default ensures the WAF can analyze the full request body that applications will accept.

The Tradeoff

Larger buffers mean more memory per request. At Cloudflare's scale (millions of requests per second), increasing from 128KB to 1MB is significant.

Lessons Learned

Two-pass architectures need clear contracts. When different parts of your code make assumptions about data shape, those assumptions need to be explicit and enforced, preferably by types.

The "off" path is rarely tested. The killswitch had been used many times, but never on an "execute" rule. The bug hid for years in an untested code path.

Simpler code is safer code. The second pass existed for some reason (parallelism? memory?), but it introduced a coupling between two functions that could fall out of sync. Sometimes the simpler design is worth the tradeoff.

AWS DynamoDB US-East-1 Outage Explained

Mon, 27 Oct 2025 00:00:00 GMT

Overview

On October 19th / 20th, 2025, AWS's Northern Virginia (US-East-1) region experienced a cascading outage affecting numerous services, starting with DynamoDB and spreading to EC2, Lambda, NLB, and others. This post breaks down the technical causes and explores the underlying concepts.

Official summary

Timeline of Events

Oct 19, 11:48 PM PDT - DynamoDB DNS race condition triggers failure
    ↓
Oct 20, 2:25 AM PDT - DynamoDB DNS state manually restored
    ↓
Oct 20, 2:25-2:40 AM PDT - Clients recover as DNS caches expire
    ↓
Oct 20, 2:25-5:28 AM PDT - DWFM congestive collapse
                           (Selective restarts at 4:14 AM)
                           (Leases re-established by 5:28 AM)
    ↓
Oct 20, 6:21-10:36 AM PDT - Network Manager propagation delays
                            (New instances launch but lack connectivity)
    ↓
Oct 20, 5:30 AM-2:09 PM PDT - NLB health check failures
                              (Automatic AZ failover disabled at 9:36 AM)
    ↓
Oct 20, 1:50 PM PDT - Full EC2 recovery

Root Cause: DNS Race Condition in DynamoDB

The Problem

A race condition in DynamoDB's DNS management system resulted in an incorrect empty DNS record for the regional endpoint (dynamodb.us-east-1.amazonaws.com), making DynamoDB completely unreachable. The automated recovery system failed to repair this invalid state, requiring manual intervention.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    DynamoDB DNS System                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────┐                ┌──────────────┐           │
│  │ DNS Planner  │───────────────▶│ DNS Enactor  │           │
│  │              │  Creates Plans │ (Instance 1) │           │
│  │ Monitors     │                └──────────────┘           │
│  │ Load Bal     │                          │                │
│  │ Health &     │                          │                │
│  │ Capacity     │                ┌──────────────┐           │
│  └──────────────┘                │ DNS Enactor  │           │
│           │                      │ (Instance 2) │           │
│           │                      └──────────────┘           │
│           │                                │                │
│           ▼                                ▼                │
│  ┌──────────────────────────────────────────────────┐       │
│  │   Route 53 (Updates DNS Records for Endpoints)   │       │
│  │         dynamodb.us-east-1.amazonaws.com         │       │
│  └──────────────────────────────────────────────────┘       │
│                                                             │
│  Note: Single regional DNS plan                             │
│        DNS Enactors run redundantly across multiple AZs     │
└─────────────────────────────────────────────────────────────┘

How the Race Condition Occurred

Note: This is a conceptual illustration of the race condition based on the AWS post-mortem. The exact implementation details are AWS-internal.

Time    DNS Enactor 1 (Delayed)         DNS Enactor 2 (Fast Path)
─────────────────────────────────────────────────────────────────────
T0      Working on plan #10             Waiting
        Gets delayed on one endpoint...

T1      Still delayed...                Picks up newer plan #15

T2      Still delayed...                Rapidly processes all endpoints
                                        ✓ Endpoint A
                                        ✓ Endpoint B
                                        ✓ Endpoint C
                                        ✓ ... all complete
                                        Invokes cleanup process
                                        (deletes plans < #12)

T3      Finally completes its delayed   Cleanup: Deletes plan #10
        work from plan #10 based on     (appears to be old/unused)
        stale view of state

T4      - Interleaving causes Route 53 to end up with
           incorrect empty DNS record
        - Automation fails to self-repair
        - DynamoDB unreachable, requires manual intervention

Code Example: Conceptual Implementation

Note: This is simplified pseudocode to illustrate the concept, not actual AWS implementation.

// Conceptual DNS Enactor Logic
class DNSEnactor {
	async applyPlan(
		planId: number,
		endpoint: string
	) {
		// Check if this plan is newer than the current one
		const currentPlan = await this.getCurrentPlan(
			endpoint
		);
		if (planId <= currentPlan.version) {
			console.warn("Plan is older, skipping");
			return;
		}

		// Apply the plan to Route 53
		// This can take time and might be delayed
		// Problem: No verification that the plan is still valid when we commit
		await this.updateRoute53(endpoint, planId);
	}

	async cleanupOldPlans(latestPlanId: number) {
		// Find and delete plans that are significantly older
		const plans = await this.getAllPlans();
		for (const plan of plans) {
			if (plan.version < latestPlanId - 3) {
				await this.deletePlan(plan.version);
				// Problem: This can delete a plan that's being applied by a delayed enactor
			}
		}
	}
}

// The Bug: Race Between Apply and Cleanup
// Enactor 1: Reads plan #10, starts applying (gets delayed)
// Enactor 2: Reads plan #15, applies quickly, runs cleanup
// Enactor 2: Deletes plan #10 (seems too old)
// Enactor 1: Finally commits plan #10 based on stale view
// Result: Route 53 ends up with inconsistent/empty record

// The Fix: Compare-and-Set (CAS) semantics
class FixedDNSEnactor {
	async applyPlan(
		planId: number,
		endpoint: string
	) {
		const currentPlan = await this.getCurrentPlan(
			endpoint
		);
		if (planId <= currentPlan.version) {
			return;
		}

		// Use atomic compare-and-set when committing to Route 53
		const success = await this.updateRoute53(
			endpoint,
			planId,
			{
				// Only succeed if version unchanged
				ifCurrentVersionIs: currentPlan.version,
			}
		);

		if (!success) {
			console.warn(
				"Plan version changed during apply, aborting"
			);
		}
	}

	async cleanupOldPlans(latestPlanId: number) {
		const plans = await this.getAllPlans();
		for (const plan of plans) {
			if (plan.version < latestPlanId - 3) {
				// Only delete if not actively being applied
				const isActive =
					await this.checkIfBeingApplied(
						plan.version
					);
				if (!isActive) {
					await this.deletePlan(plan.version);
				}
			}
		}
	}
}

Concept Deep Dive: Leases

Technical context

To understand what happened, we need to share some information about a few subsystems that are used for the management of EC2 instance launches, as well as for configuring network connectivity for newly launched EC2 instances.

The first subsystem is DropletWorkflow Manager (DWFM), which is responsible for the management of all the underlying physical servers that are used by EC2 for the hosting of EC2 instances – we call these servers “droplets”.

The second subsystem is Network Manager, which is responsible for the management and propagation of network state to all EC2 instances and network appliances. Each DWFM manages a set of droplets within each Availability Zone and maintains a lease for each droplet currently under management. This lease allows DWFM to track the droplet state, ensuring that all actions from the EC2 API or within the EC2 instance itself, such as shutdown or reboot operations originating from the EC2 instance operating system, result in the correct state changes within the broader EC2 systems. As part of maintaining this lease, each DWFM host has to check in and complete a state check with each droplet that it manages every few minutes.

What is a Lease?

A lease is a temporary ownership claim or lock on a resource. In AWS EC2's case, it's the agreement between DropletWorkflow Manager (DWFM) and each physical server ("droplet") that tracks server state and availability.

Visual Representation

┌──────────────────────────────────────────────────────────┐
│         DropletWorkflow Manager (DWFM) Fleet             │
│      [Multiple DWFM hosts per AZ, each managing          │
│       a set of droplets within that AZ]                  │
└─────────────────┬────────────────────────────────────────┘
                  │ Lease Check (every few minutes)
                  │ "Are you still alive? Ready for work?"
                  │ State checks depend on DynamoDB
                  │
                  ▼
┌─────────────────────────────────────────────────────────┐
│              Physical Servers (Droplets)                │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐ │
│  │ Droplet 1│  │ Droplet 2│  │ Droplet 3│  │ Droplet N│ │
│  │  Lease:  │  │  Lease:  │  │  Lease:  │  │  Lease:  │ │
│  │  Active  │  │  Active  │  │ Expired  │  │  Active  │ │
│  │          │  │          │  │          │  │          │ │
│  │ Status:  │  │ Status:  │  │ Unusable │  │ Status:  │ │
│  │ Ready    │  │ In-Use   │  │ for new  │  │ Ready    │ │
│  │          │  │ (2 VMs)  │  │ launches │  │          │ │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘ │
└─────────────────────────────────────────────────────────┘

How Leases Work

// Conceptual Lease Mechanism
class DropletWorkflowManager {
	private leases = new Map<string, Lease>(); // droplet_id → lease

	// Periodic heartbeat to renew leases
	async renewLease(
		dropletId: string
	): Promise<void> {
		try {
			// Check if droplet is healthy
			const health =
				await this.checkDropletHealth(dropletId);

			// DWFM's state checks depend on DynamoDB
			// During the outage, this call failed due to DNS issues
			await this.completeStateCheckWithDynamoDB(
				dropletId,
				{
					status: health ? "active" : "degraded",
					lastCheckIn: Date.now(),
					expiryTime: Date.now() + LEASE_TIMEOUT, // e.g., 5 minutes
				}
			);

			this.leases.set(dropletId, {
				status: "active",
				expiryTime: Date.now() + LEASE_TIMEOUT,
			});
		} catch (error) {
			console.error(
				`Failed to renew lease for ${dropletId}`
			);
			// Lease will expire if not renewed in time
		}
	}

	// Check if droplet is available for new EC2 launches
	isAvailableForLaunch(
		dropletId: string
	): boolean {
		const lease = this.leases.get(dropletId);
		if (!lease) return false;

		// Lease must be active and not expired
		return (
			lease.status === "active" &&
			lease.expiryTime > Date.now()
		);
	}

	// Handle expired leases
	onLeaseExpired(dropletId: string): void {
		console.warn(
			`Lease expired for ${dropletId}`
		);
		// Mark droplet as unavailable for new instance launches
		this.markUnavailable(dropletId);
	}
}

Why Leases Matter

State Coordination: Ensures DWFM knows which servers are available
Failure Detection: Expired leases indicate unhealthy or unreachable servers
Resource Management: Only droplets with active leases can host new EC2 instances

The Outage Scenario

Normal Operation:
┌──────────────────────────────────────────────────────┐
│ DWFM → Renew Lease → DynamoDB → Success              │
│ (every few minutes)                                  │
│ Droplets available for launches                      │
└──────────────────────────────────────────────────────┘

During Outage:
┌──────────────────────────────────────────────────────┐
│ DWFM → Renew Lease → DynamoDB → DNS Failure          │
│ Cannot update lease state                            │
│ Lease expires → Droplets marked unavailable          │
│ "Insufficient capacity" errors for new launches      │
└──────────────────────────────────────────────────────┘

After DynamoDB Recovery (Thundering Herd):
┌──────────────────────────────────────────────────────┐
│ DWFM attempts to renew 100,000+ leases simultaneously│
│ Queue overload → Processing delays                   │
│ Renewal attempts timeout before completing           │
│ Leases re-expire → More work queued                  │
│ Retries pile up faster than completions              │
│ → Congestive collapse                                │
│ Solution: Selective DWFM restarts + throttling       │
│           Cleared queues and restored progress       │
└──────────────────────────────────────────────────────┘

Cascading Failures

Service Dependency Chain

DynamoDB Root Cause and Chain of Failures
┌────────────────────────────────────────────────────────────┐
│                   Layer 0: Root Cause                      │
│  ┌──────────────────────────────────────────────────┐      │
│  │         DynamoDB DNS Race Condition              │      │
│  │    Incorrect empty DNS record published          │      │
│  │    dynamodb.us-east-1.amazonaws.com unreachable  │      │
│  └──────────────┬───────────────────────────────────┘      │
└─────────────────┼──────────────────────────────────────────┘
                  │
                  ▼
┌────────────────────────────────────────────────────────────┐
│               Layer 1: Direct Dependencies                 │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │    EC2      │  │   Lambda    │  │   Redshift  │         │
│  │ DWFM leases │  │ Function ops│  │  Query ops  │         │
│  │   expire    │  │   blocked   │  │   blocked   │         │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘         │
└─────────┼────────────────┼────────────────┼────────────────┘
          │                │                │
          ▼                ▼                ▼
┌────────────────────────────────────────────────────────────┐
│         Layer 2: Secondary Effects (Thundering Herd)       │
│  ┌──────────────────────────────────────────────────┐      │
│  │ EC2 DWFM: Massive backlog of lease renewals      │      │
│  │ → Congestive collapse                            │      │
│  │ → Blocks new instance launches                   │      │
│  │ → Network Manager backlog (6:21-10:36 AM)        │      │
│  │    New instances launch but lack connectivity    │      │
│  └──────────────┬───────────────────────────────────┘      │
└─────────────────┼──────────────────────────────────────────┘
                  │
                  ▼
┌────────────────────────────────────────────────────────────┐
│               Layer 3: Dependent Services                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │     NLB     │  │    ECS      │  │   Lambda    │         │
│  │ Health check│  │ Container   │  │  Execution  │         │
│  │   failures  │  │   launches  │  │  throttled  │         │
│  │  (network   │  │   blocked   │  │  (priority  │         │
│  │   delays)   │  │             │  │   to sync)  │         │
│  │ Mitigated:  │  │             │  │             │         │
│  │  Disabled   │  │             │  │             │         │
│  │  auto AZ    │  │             │  │             │         │
│  │  failover   │  │             │  │             │         │
│  │  @ 9:36 AM  │  │             │  │             │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└────────────────────────────────────────────────────────────┘

Key Concepts Explained

1. Thundering Herd Problem

When a resource becomes available after being unavailable, all waiting processes try to access it simultaneously, overwhelming the system.

// Example: What happened in DWFM after DynamoDB recovered
class ThunderingHerdExample {
	async demonstrate() {
		// Simulate expired leases during DynamoDB outage
		const expiredLeases = []; // 100,000+ droplets
		for (let i = 0; i < 100000; i++) {
			expiredLeases.push({
				id: `droplet-${i}`,
				leaseExpired: true,
			});
		}

		// DynamoDB comes back online
		const dynamoDBRecovered = true;

		if (dynamoDBRecovered) {
			// BAD: All requests at once
			await Promise.all(
				expiredLeases.map((lease) =>
					this.renewLease(lease.id)
				)
			);
			// Result: System overload, timeouts, more queued work

			// GOOD: Rate-limited approach with jitter and backoff
			const BATCH_SIZE = 100;
			const MAX_CONCURRENT = 1000; // Global concurrency limit
			let retryDelay = 100; // Start with 100ms

			for (
				let i = 0;
				i < expiredLeases.length;
				i += BATCH_SIZE
			) {
				const batch = expiredLeases.slice(
					i,
					i + BATCH_SIZE
				);

				// Add jitter to prevent synchronized retries
				const jitter = Math.random() * 50; // 0-50ms random delay

				await Promise.all(
					batch.map((lease) =>
						this.renewLease(lease.id)
					)
				);

				// Queue-size-based rate limiting (AWS's planned fix)
				const queueSize =
					await this.getQueueSize();
				if (queueSize > 10000) {
					// Exponential backoff when queue is large
					retryDelay = Math.min(
						retryDelay * 1.5,
						5000
					); // Cap at 5s
				} else {
					retryDelay = 100; // Reset when queue drains
				}

				await this.sleep(retryDelay + jitter);
			}
		}
	}
}

2. Congestive Collapse

When a system tries to process more work than it can handle, processing delays increase, leading to more work being queued, which further exacerbates the problem.

What happened with DWFM: Retries piled up because processing took so long that in-flight leases re-expired before completion, creating more work than could be drained.

Normal: [Request] → [Process] → [Done] (5ms)
        [Request] → [Process] → [Done] (5ms)

Congested:
[Request spiral] → [Queue growing 1000s] → [Processing at max capacity]
    ↑                                          ↓
    └────── [More requests queued ────────┘ (faster than processing)
             (Leases re-expire during       (Creating even more work)
              processing delays)

Result: Queue grows infinitely, response times explode
Solution: Restart DWFM hosts to clear queue + throttle incoming work

3. DNS Management Systems

Large distributed systems use DNS for service discovery and load balancing. Managing thousands of load balancer IPs requires sophisticated automation.

Traditional Service:
┌─────────┐
│ Client  │──────▶ [dynamodb.us-east-1.amazonaws.com]
└─────────┘       ┌──────────────────────────────────┐
                  │ Route 53 DNS Resolver            │
                  │ Returns single IP: 52.94.1.1     │
                  └──────────────────────────────────┘

AWS's Approach (Hundreds of IPs):
┌─────────┐
│ Client  │──────▶ [dynamodb.us-east-1.amazonaws.com]
└─────────┘       ┌──────────────────────────────────┐
                  │ Route 53 DNS Resolver            │
                  │ Returns 50+ IPs for load sharing │
                  │ 52.94.1.1, 52.94.1.2, 52.94.1.3  │
                  │ 52.94.1.4, 52.94.1.5, ...        │
                  │    ... 52.94.1.50                │
                  └──────────────────────────────────┘
                         ↓
                  ┌──────────────────────────────────┐
                  │ Load Balancer Pool               │
                  │ (Hundreds of NLBs across AZs)    │
                  └──────────────────────────────────┘

Note: In practice, Route 53 answers and LB pools are more complex
      (weighted, health-checked, versioned). Numbers simplified for clarity.

AWS's Planned Fixes

1. DynamoDB DNS System

// Before (Race condition bug)
class DNSEnactor {
	async cleanupOldPlans(latestPlanId: number) {
		// Deletes any plan significantly older
		// Doesn't check if plan is currently being applied
		const plans = await this.getAllPlans();
		for (const plan of plans) {
			if (plan.version < latestPlanId - 3) {
				await this.deletePlan(plan.version);
			}
		}
	}
}

// After (Partial Fix - still has a subtle race condition!)
class FixedDNSEnactor {
	async cleanupOldPlans(latestPlanId: number) {
		const plans = await this.getAllPlans();
		for (const plan of plans) {
			if (plan.version < latestPlanId - 3) {
				// Check if not actively being applied
				const isActive =
					await this.checkIfBeingApplied(
						plan.version
					);
				// RACE CONDITION: Between this check and the delete below,
				// another enactor could start applying this plan!
				// This is a TOCTOU (Time-of-Check to Time-of-Use) bug
				if (!isActive) {
					await this.deletePlan(plan.version);
				}
			}
		}
	}
}

// Better Fix: Atomic check-and-delete operation
class BetterDNSEnactor {
	async cleanupOldPlans(latestPlanId: number) {
		const plans = await this.getAllPlans();
		for (const plan of plans) {
			if (plan.version < latestPlanId - 3) {
				// Atomically delete only if not in use
				// This combines the check and delete into a single atomic operation
				const deleted =
					await this.deleteIfNotInUse(
						plan.version
					);
				if (deleted) {
					console.log(
						`Cleaned up plan ${plan.version}`
					);
				}
			}
		}
	}

	// Atomic operation: check and delete in one transaction
	async deleteIfNotInUse(
		planVersion: number
	): Promise<boolean> {
		// This would be implemented as a single database transaction or
		// using conditional delete with DynamoDB's ConditionExpression
		// Example: DELETE WHERE version = X AND inUse = false
		return await this.db.conditionalDelete({
			version: planVersion,
			condition: { inUse: false },
		});
	}
}

Important: Even the "partial fix" above has a subtle TOCTOU (Time-of-Check to Time-of-Use) race condition between checking if a plan is active and deleting it. In a multi-threaded environment with multiple DNS Enactors, another enactor could start applying a plan in the tiny window between the check and the delete.

The better approach is to use atomic operations that combine the check and delete into a single transaction. In AWS's case, this might use DynamoDB's conditional writes, distributed locks (like with Redis or ZooKeeper), or database transactions that guarantee the check and delete happen atomically.

This is a great example of how fixing one race condition can introduce another if you're not careful. Distributed systems are hard!

2. NLB Velocity Controls

Limit how quickly health check failures can remove capacity to prevent overreaction to temporary issues.

3. EC2 Improvements

Enhanced testing for DWFM recovery workflows
Better throttling mechanisms for data propagation queues
Queue-size based rate limiting

Lessons Learned

Note: All examples are using Typescript / Node.js for learning purposes only. Do not use this code in production without understanding atomic operations.

Most of the above race conditions, thundering herd and other cascading issues mentioned are very much only ever visible in high scale systems, and even more so in AWS's case.

I've only ever once seen the thundering herd problem in production when I was on-call at DAZN. Maybe thats a blog post for another day!

What have we learned?

Race conditions. You can build the most sophisticated system in the world, but if two pieces of code can step on each other's toes at the wrong time, it can cause unexpected and unknown issues. The DynamoDB DNS issue was a subtle timing bug that only appeared under specific conditions.

The thundering herd problem. When DynamoDB came back online, DWFM tried to renew 100,000+ leases all at once. The system collapsed under its own recovery attempts. Jitter, backoff, queue-size based throttling, are common techniques to counter this.

Not everything can be automated (can it?). The automation failed to fix the DNS issue. Engineers had to step in, disable systems, restart services, and manually restore things.

Cascading failures. DynamoDB broke, which broke EC2 leases, which triggered a thundering herd, which overloaded Network Manager, which broke NLB health checks. Each failure created the conditions for the next one.

Complexity. AWS have a large scale and complex DNS management system with planners and enactors running across multiple availability zones for resilience. The race condition created an invalid state that the system couldn't recover from.

Build a Basic AI Agent in TypeScript, Part 2: Tool Calling and LLM Intent Routing

Mon, 29 Dec 2025 00:00:00 GMT

Overview

Part 1 introduced the core agent loop. Part 2 adds tool calling. A weather tool is registered with the model, then the model decides whether to call that tool based on user intent. This is a common production pattern in major agent stacks: tool definitions are passed in context, and the model routes per turn.

TLDR: A tool performs deterministic external actions, while the LLM decides when to invoke the tool. The runtime executes tool calls, returns results, and continues the loop.

Full Series

Part 1, The Core Loop
Part 2, Tool Calling and LLM Intent Routing (you are here)
Part 3, Multi-Step Skill Orchestration
Part 4, Production Readiness and Operational Guardrails

What Changes from Part 1

Add a weather tool implementation backed by Open-Meteo.
Pass tool definitions to the model on each inference call.
Let the model route by choosing between natural-language answer and tool calls.
Add a tool execution loop that sends tool results back to the model.

Core Concepts and Keywords

Tool

A tool is a deterministic capability adapter, for example, get_weather(location). It executes external work and returns structured data.

Tool Calling

Tool calling is the request and response handshake where the model emits tool calls, the runtime executes them, then returns tool results back to the model.

Intent Routing

Intent routing is the decision of which path to use for a request. In this post, routing is model-driven. The model decides if the weather tool should be called.

Tool Call Loop

A tool call loop repeats until there are no more tool calls in the model response:

Model response includes tool calls.
Runtime executes tool calls.
Runtime appends tool results to conversation.
Model is called again with updated conversation.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                Agent Loop with LLM Tool Routing              │
├──────────────────────────────────────────────────────────────┤
│ User message                                                 │
│   │                                                          │
│   ▼                                                          │
│ Append to conversation[]                                     │
│   │                                                          │
│   ▼                                                          │
│ Call model with messages + tools[]                           │
│   │                                                          │
│   ├── no tool_calls ──▶ return assistant text                │
│   │                                                          │
│   └── tool_calls ──▶ execute tool(s)                         │
│                        │                                     │
│                        ▼                                     │
│                  append tool results                         │
│                        │                                     │
│                        ▼                                     │
│            call model again with updated history             │
└──────────────────────────────────────────────────────────────┘

Tool Definition

The model needs a clear tool schema and description:

const tools: ToolDefinition[] = [
  {
    type: "function",
    function: {
      name: "get_weather",
      description:
        "Get current weather for a location. Use this when the user asks about current weather, temperature, forecast, rain, wind, or humidity.",
      parameters: {
        type: "object",
        properties: {
          location: {
            type: "string",
            description: "City or place name, for example Tokyo or London, UK"
          }
        },
        required: ["location"],
        additionalProperties: false
      }
    }
  }
];

Why LLM Intent Routing

This implementation intentionally uses model-driven routing, not code-based intent matching.

Better handling for varied natural language.
Better compatibility with mainstream tool-calling patterns.
Cleaner separation: model chooses tools, runtime executes tools.

The runtime still controls execution limits, error handling, and unknown tool behavior.

Context Budget and Tool Surface Area

Tool names, descriptions, and schemas are usually included in prompt context, so tool count and tool verbosity affect token usage.

Practical implications:

Keep tool descriptions precise and high-signal.
Scope tools to agent domain.
Avoid passing unrelated tools.

A weather-focused agent should not include unrelated operational tools like API key rotation tools. That increases cost and misrouting risk.

Runnable Example

Working file:

src/basic-weather-tool-routing-openrouter.ts

Run it:

bun src/basic-weather-tool-routing-openrouter.ts

Create .env first. Bun loads this automatically at runtime, see Bun environment variables.

OPENROUTER_API_KEY=your_api_key_here

Full Runnable File

import { createInterface } from "node:readline/promises";
import { stdin as input, stdout as output } from "node:process";

type Role = "system" | "user" | "assistant" | "tool";

interface ToolCall {
  id: string;
  name: string;
  argumentsJson: string;
}

interface Message {
  role: Role;
  content: string;
  toolCallId?: string;
  toolCalls?: Array<{
    id: string;
    type: "function";
    function: {
      name: string;
      arguments: string;
    };
  }>;
}

interface ToolDefinition {
  type: "function";
  function: {
    name: string;
    description: string;
    parameters: {
      type: "object";
      properties: Record<string, unknown>;
      required?: string[];
      additionalProperties?: boolean;
    };
  };
}

interface ModelClient {
  createMessage(input: {
    model: string;
    messages: Message[];
    tools: ToolDefinition[];
    maxTokens: number;
  }): Promise<{ text: string; toolCalls: ToolCall[] }>;
}

interface AgentConfig {
  model: string;
  maxTokens: number;
  maxTurns: number;
  maxToolRoundsPerTurn: number;
}

interface WeatherSnapshot {
  location: string;
  tempC: string;
  description: string;
  humidity: string;
  windKmph: string;
}

interface WeatherTool {
  getCurrentWeather(location: string): Promise<WeatherSnapshot | null>;
}

class OpenRouterModelClient implements ModelClient {
  constructor(private readonly apiKey: string) {}

  async createMessage(input: {
    model: string;
    messages: Message[];
    tools: ToolDefinition[];
    maxTokens: number;
  }): Promise<{ text: string; toolCalls: ToolCall[] }> {
    const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${this.apiKey}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        model: input.model,
        messages: input.messages.map((m) => {
          if (m.role === "assistant" && m.toolCalls) {
            return {
              role: "assistant",
              content: m.content,
              tool_calls: m.toolCalls
            };
          }

          if (m.role === "tool") {
            return {
              role: "tool",
              tool_call_id: m.toolCallId,
              content: m.content
            };
          }

          return {
            role: m.role,
            content: m.content
          };
        }),
        tools: input.tools,
        tool_choice: "auto",
        max_tokens: input.maxTokens
      })
    });

    if (!response.ok) {
      const errorBody = await response.text();
      throw new Error(`OpenRouter error ${response.status}: ${errorBody}`);
    }

    const data = (await response.json()) as {
      choices?: Array<{
        message?: {
          content?: string | null;
          tool_calls?: Array<{
            id: string;
            type: "function";
            function: {
              name: string;
              arguments: string;
            };
          }>;
        };
      }>;
    };

    const message = data.choices?.[0]?.message;
    const text = (message?.content || "").trim();
    const toolCalls =
      message?.tool_calls?.map((call) => ({
        id: call.id,
        name: call.function.name,
        argumentsJson: call.function.arguments
      })) || [];

    return { text, toolCalls };
  }
}

class OpenMeteoWeatherTool implements WeatherTool {
  async getCurrentWeather(location: string): Promise<WeatherSnapshot | null> {
    const geoUrl = `https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(location)}&count=1`;
    const geoResponse = await fetch(geoUrl, { signal: AbortSignal.timeout(8000) });
    if (!geoResponse.ok) return null;

    const geo = (await geoResponse.json()) as {
      results?: Array<{
        name?: string;
        country?: string;
        latitude?: number;
        longitude?: number;
      }>;
    };

    const place = geo.results?.[0];
    if (!place?.latitude || !place?.longitude) return null;

    const weatherUrl =
      `https://api.open-meteo.com/v1/forecast?latitude=${place.latitude}` +
      `&longitude=${place.longitude}` +
      "&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m";

    const weatherResponse = await fetch(weatherUrl, { signal: AbortSignal.timeout(8000) });
    if (!weatherResponse.ok) return null;

    const weatherData = (await weatherResponse.json()) as {
      current?: {
        temperature_2m?: number;
        relative_humidity_2m?: number;
        weather_code?: number;
        wind_speed_10m?: number;
      };
    };

    const current = weatherData.current;
    if (typeof current?.temperature_2m !== "number") return null;

    return {
      location: `${place.name || location}${place.country ? `, ${place.country}` : ""}`,
      tempC: String(current.temperature_2m),
      description: this.describeWeatherCode(current.weather_code),
      humidity:
        typeof current.relative_humidity_2m === "number"
          ? String(current.relative_humidity_2m)
          : "unknown",
      windKmph:
        typeof current.wind_speed_10m === "number"
          ? String(current.wind_speed_10m)
          : "unknown"
    };
  }

  private describeWeatherCode(code?: number): string {
    const labels: Record<number, string> = {
      0: "Clear sky",
      1: "Mainly clear",
      2: "Partly cloudy",
      3: "Overcast",
      45: "Fog",
      48: "Depositing rime fog",
      51: "Light drizzle",
      53: "Moderate drizzle",
      55: "Dense drizzle",
      61: "Slight rain",
      63: "Moderate rain",
      65: "Heavy rain",
      71: "Slight snow",
      73: "Moderate snow",
      75: "Heavy snow",
      80: "Rain showers",
      95: "Thunderstorm"
    };

    if (typeof code !== "number") return "Unknown conditions";
    return labels[code] || `Weather code ${code}`;
  }
}

class Agent {
  private conversation: Message[] = [];
  private readonly tools: ToolDefinition[];

  constructor(
    private readonly modelClient: ModelClient,
    private readonly config: AgentConfig,
    private readonly weatherTool: WeatherTool,
    private readonly getUserInput: () => Promise<string | null>,
    private readonly render: (text: string) => void
  ) {
    this.tools = [
      {
        type: "function",
        function: {
          name: "get_weather",
          description:
            "Get current weather for a location. Use this when the user asks about current weather, temperature, forecast, rain, wind, or humidity.",
          parameters: {
            type: "object",
            properties: {
              location: {
                type: "string",
                description: "City or place name, for example Tokyo or London, UK"
              }
            },
            required: ["location"],
            additionalProperties: false
          }
        }
      }
    ];
  }

  async run(): Promise<void> {
    let turnCount = 0;
    this.render("Chat started. Submit empty input to exit.");

    while (turnCount < this.config.maxTurns) {
      const userInput = await this.getUserInput();
      if (!userInput || userInput.trim() === "") {
        this.render("Session ended by input.");
        break;
      }

      this.conversation.push({ role: "user", content: userInput });

      try {
        const reply = await this.runTurnWithTools();
        this.conversation.push({ role: "assistant", content: reply });
        this.render(reply);
      } catch (error) {
        const message = error instanceof Error ? error.message : "Unknown runtime error";
        this.render(`Request failed: ${message}`);
      }

      turnCount += 1;
    }

    if (turnCount >= this.config.maxTurns) {
      this.render("Session ended at max turn limit.");
    }
  }

  private async runTurnWithTools(): Promise<string> {
    for (let round = 0; round < this.config.maxToolRoundsPerTurn; round += 1) {
      const response = await this.modelClient.createMessage({
        model: this.config.model,
        maxTokens: this.config.maxTokens,
        messages: this.conversation,
        tools: this.tools
      });

      if (response.toolCalls.length === 0) {
        return response.text || "[No text response]";
      }

      this.conversation.push({
        role: "assistant",
        content: response.text || "",
        toolCalls: response.toolCalls.map((call) => ({
          id: call.id,
          type: "function",
          function: {
            name: call.name,
            arguments: call.argumentsJson
          }
        }))
      });

      for (const toolCall of response.toolCalls) {
        const result = await this.executeTool(toolCall);
        this.conversation.push({
          role: "tool",
          toolCallId: toolCall.id,
          content: result
        });
      }
    }

    return "The agent reached max tool rounds for this turn.";
  }

  private async executeTool(toolCall: ToolCall): Promise<string> {
    if (toolCall.name !== "get_weather") {
      return JSON.stringify({ error: `Unknown tool: ${toolCall.name}` });
    }

    let parsedArgs: { location?: string } = {};
    try {
      parsedArgs = JSON.parse(toolCall.argumentsJson) as { location?: string };
    } catch {
      return JSON.stringify({ error: "Invalid JSON arguments for get_weather" });
    }

    const location = parsedArgs.location?.trim();
    if (!location) {
      return JSON.stringify({ error: "Missing location argument for get_weather" });
    }

    const weather = await this.weatherTool.getCurrentWeather(location);
    if (!weather) {
      return JSON.stringify({ error: `Weather lookup failed for ${location}` });
    }

    return JSON.stringify(weather);
  }
}

async function main(): Promise<void> {
  const apiKey = process.env.OPENROUTER_API_KEY;
  if (!apiKey) {
    throw new Error("Missing OPENROUTER_API_KEY in environment");
  }

  const modelClient = new OpenRouterModelClient(apiKey);
  const weatherTool: WeatherTool = new OpenMeteoWeatherTool();

  const readline = createInterface({ input, output });
  const getUserInput = async (): Promise<string | null> => {
    try {
      return await readline.question("You: ");
    } catch {
      return null;
    }
  };

  const render = (text: string): void => {
    console.log(`Assistant: ${text}`);
  };

  const agent = new Agent(
    modelClient,
    {
      model: "openai/gpt-4o-mini",
      maxTokens: 1024,
      maxTurns: 20,
      maxToolRoundsPerTurn: 4
    },
    weatherTool,
    getUserInput,
    render
  );

  try {
    await agent.run();
  } finally {
    readline.close();
  }
}

main().catch((error) => {
  const message = error instanceof Error ? error.message : "Unknown startup error";
  console.error(`Fatal error: ${message}`);
  process.exit(1);
});

Real Conversation Example

Assistant: Chat started. Submit empty input to exit.
You: what is the weather in tokyo
Assistant: The current weather in Tokyo is as follows:
- Temperature: 4.9°C
- Description: Overcast
- Humidity: 81%
- Wind Speed: 9.8 km/h
You: what is the current stock price of nvidia
Assistant: I currently don't have access to real-time stock prices. You can check the latest stock price of NVIDIA on financial news websites, stock market apps, or your brokerage platform.

The weather request triggers get_weather. The stock request does not match available tools, so the model answers directly.

Why This Matters in Production

1. Natural-Language Flexibility

Model-driven routing handles varied user phrasing better than brittle keyword matching.

2. Standard Tool-Calling Pattern

This approach matches modern agent SDK workflows: pass tools to the model, run the tool loop, and return tool results.

3. Better Extensibility

Adding new capabilities becomes a tool-definition problem instead of accumulating route-condition code.

4. Clear Runtime Control Points

Even with model-driven routing, runtime controls remain explicit: max tool rounds, tool execution validation, and unknown-tool handling.

Common Pitfalls

Overloading Tool Surface

Too many unrelated tools in a single agent increases token cost and tool-selection ambiguity.

Weak Tool Descriptions

Sparse descriptions reduce routing quality. Tool descriptions should include when to use and when not to use.

Missing Tool Loop Limits

Without max tool rounds per turn, badly formed loops can become expensive and unstable.

Unvalidated Tool Arguments

Tool arguments can be malformed. Always parse and validate input before external calls.

Lessons Learned

Tool calling is the bridge from conversational agents to real-world actions.
LLM intent routing is practical when paired with strict runtime controls.
Tool quality depends heavily on schema and description quality.
Scoped tool sets improve both cost and reliability.

What Comes Next in Part 3

Part 3 adds multi-tool and multi-skill orchestration, then introduces stronger controls for observability, idempotency, human approval gates, and production runbook readiness.

Official References

Build a Basic AI Agent in TypeScript, Part 1: The Core Loop

Fri, 14 Nov 2025 00:00:00 GMT

Overview

At its core, an agent is a control loop wrapped around model calls and state management. This post introduces that loop first, then shows a runnable TypeScript implementation that matches the same flow. The aim is to keep the mental model simple while still giving readers something practical to execute.

TLDR: A basic agent is mostly a deterministic loop that sends conversation history to a model, appends responses, and repeats until an exit condition is met.

Full Series

Part 1, The Core Loop (you are here)
Part 2, Tool Calling and LLM Intent Routing
Part 3, Multi-Step Skill Orchestration
Part 4, Production Readiness and Operational Guardrails

Sections

What We Are Building in Part 1
Core Concepts and Keywords
Architecture at a Glance
Turn Lifecycle
Clarifying Two Terms
TypeScript Implementation
Why This Design Matters
Common Pitfalls in Basic Agent Loops
Lessons Learned
What Comes Next in Part 2
Official References

What We Are Building in Part 1

In this part, we are intentionally building the smallest useful baseline:

A terminal-style chat loop.
Persistent conversation memory across turns.
A model call wrapper.
Basic error handling and exit conditions.

We are not adding tools or skills yet. That is intentional. If the loop is weak, everything built on top will be fragile.

Core Concepts and Keywords

Before code, here are the key terms used throughout the series.

Agent

An agent is a program that repeatedly observes input, decides what to do next, and produces output. In practical LLM apps, the "decide" step often means making a model call and interpreting the response.

Turn

A turn is one cycle of interaction:

User message enters the system.
Model generates a response.
Response is shown and stored.

Conversation State

Conversation state is the full message history sent back to the model on each turn. Most model APIs are stateless by request, so memory lives in the application, not on the model server.

Inference

Inference is one model generation call, for example, "Given these messages, produce the next assistant message."

Deterministic Control Loop

A deterministic control loop means runtime flow is explicit and predictable. The model generates text, but the program decides:

when to call the model,
what context to send,
when to stop,
how to handle failures.

Architecture at a Glance

┌────────────────────────────────────────────────────────┐
│                 Basic Agent Runtime                    │
├────────────────────────────────────────────────────────┤
│ while (turnCount < maxTurns)                           │
│   │                                                    │
│   ▼                                                    │
│ Read user input                                        │
│   │                                                    │
│   ├── empty input ──▶ Exit                             │
│   │                                                    │
│   ▼                                                    │
│ Append user message to conversation[]                  │
│   │                                                    │
│   ▼                                                    │
│ Model call (inference) with full conversation[]        │
│   │                                                    │
│   ▼                                                    │
│ Append assistant message and display response          │
│   │                                                    │
│   ▼                                                    │
│ Next loop iteration                                    │
└────────────────────────────────────────────────────────┘

This diagram shows system structure and control flow. It answers "what components exist and how data moves through the while loop".

Turn Lifecycle

Single iteration (Turn N) inside the while loop
    ↓
Read user input
    ↓
Append user message to conversation state
    ↓
Call model with full conversation
    ↓
Append assistant response to conversation state
    ↓
Render response
    ↓
Loop condition checked again
    ↓
Turn N+1 begins if condition still holds

This diagram is intentionally narrower. It zooms into one turn only and ignores broader component boundaries.

Clarifying Two Terms

Model Call (Inference)

"Model call" and "inference" refer to the same step. The runtime sends current conversation history to the model API and receives the assistant's next message.

Append + Display

Append the assistant response to conversation[] so memory is preserved.
Display the response in the terminal, for example with console.log.

TypeScript Implementation

This section uses a hybrid approach:

Start with a short conceptual loop.
Follow with a runnable implementation.
Call out runtime details that are useful, but secondary to the core concept.

Conceptual Loop (Read First)

This snippet is intentionally small and focused on loop semantics.

Note: This is simplified pseudocode to illustrate the concept, not actual implementation.

class Agent {
	private conversation: Message[] = [];

	async run(): Promise<void> {
		let turnCount = 0;

		while (turnCount < this.config.maxTurns) {
			const input = await this.getUserInput();
			if (!input || input.trim() === "") break;

			this.conversation.push({
				role: "user",
				content: input,
				timestamp: Date.now(),
			});

			const assistantText =
				await this.runInference(
					this.conversation,
				);

			this.conversation.push({
				role: "assistant",
				content: assistantText,
				timestamp: Date.now(),
			});

			this.render(assistantText);
			turnCount += 1;
		}
	}
}

Runnable Example (Copy and Run)

Local Setup

The following commands create a minimal runnable TypeScript project.

mkdir basic-agent-loop
cd basic-agent-loop
bun init -y

Create .env and add the API key. Bun automatically loads .env files at runtime, see Bun environment variables.

OPENROUTER_API_KEY=your_api_key_here

`src/main.ts`

This is a minimal working loop that can be copied and run.

import { createInterface } from "node:readline/promises";
import {
	stdin as input,
	stdout as output,
} from "node:process";

type Role = "user" | "assistant";

interface Message {
	role: Role;
	content: string;
	timestamp: number;
}

interface ModelClient {
	createMessage(input: {
		model: string;
		messages: Array<{
			role: Role;
			content: string;
		}>;
		maxTokens: number;
	}): Promise<{ text: string }>;
}

interface AgentConfig {
	model: string;
	maxTokens: number;
	maxTurns: number;
}

class OpenRouterModelClient
	implements ModelClient
{
	constructor(
		private readonly apiKey: string,
	) {}

	async createMessage(input: {
		model: string;
		messages: Array<{
			role: Role;
			content: string;
		}>;
		maxTokens: number;
		}): Promise<{ text: string }> {
		const response = await fetch(
			"https://openrouter.ai/api/v1/chat/completions",
			{
				method: "POST",
				headers: {
					Authorization: `Bearer ${this.apiKey}`,
					"Content-Type": "application/json",
				},
				body: JSON.stringify({
					model: input.model,
					messages: input.messages,
					max_tokens: input.maxTokens,
				}),
			},
		);

		if (!response.ok) {
			const errorBody = await response.text();
			throw new Error(
				`OpenRouter error ${response.status}: ${errorBody}`,
			);
		}

		const data = (await response.json()) as {
			choices?: Array<{
				message?: {
					content?:
						| string
						| Array<{
								type?: string;
								text?: string;
							}>;
				};
			}>;
		};

		const content = data.choices?.[0]?.message?.content;
		const text =
			typeof content === "string"
				? content.trim()
				: Array.isArray(content)
					? content
							.map((item) =>
								item.type === "text"
									? item.text || ""
									: "",
							)
							.join("\n")
							.trim()
					: "";

		return { text: text || "[No text response]" };
	}
}

class Agent {
	private conversation: Message[] = [];

	constructor(
		private readonly modelClient: ModelClient,
		private readonly config: AgentConfig,
		private readonly getUserInput: () => Promise<
			string | null
		>,
		private readonly render: (
			text: string,
		) => void,
	) {}

	async run(): Promise<void> {
		let turnCount = 0;

		this.render(
			"Chat started. Submit empty input to exit.",
		);

		while (turnCount < this.config.maxTurns) {
			const userInput = await this.getUserInput();

			if (!userInput || userInput.trim() === "") {
				this.render("Session ended by input.");
				break;
			}

			this.appendUserMessage(userInput);

			try {
				const assistantText =
					await this.runInference();
				this.appendAssistantMessage(
					assistantText,
				);
				this.render(assistantText);
			} catch (error) {
				const message =
					error instanceof Error
						? error.message
						: "Unknown inference error";
				this.render(
					`Inference failed: ${message}`,
				);
			}

			turnCount += 1;
		}

		if (turnCount >= this.config.maxTurns) {
			this.render(
				"Session ended at max turn limit.",
			);
		}
	}

	private appendUserMessage(
		content: string,
	): void {
		this.conversation.push({
			role: "user",
			content,
			timestamp: Date.now(),
		});
	}

	private appendAssistantMessage(
		content: string,
	): void {
		this.conversation.push({
			role: "assistant",
			content,
			timestamp: Date.now(),
		});
	}

	private async runInference(): Promise<string> {
		const response =
			await this.modelClient.createMessage({
				model: this.config.model,
				maxTokens: this.config.maxTokens,
				messages: this.conversation.map(
					(message) => ({
						role: message.role,
						content: message.content,
					}),
				),
			});

		return response.text;
	}
}

async function main(): Promise<void> {
	const apiKey = process.env.OPENROUTER_API_KEY;
	if (!apiKey) {
		throw new Error(
			"Missing OPENROUTER_API_KEY in environment",
		);
	}

	const modelClient = new OpenRouterModelClient(
		apiKey,
	);

	const readline = createInterface({
		input,
		output,
	});
	const getUserInput = async (): Promise<
		string | null
	> => {
		try {
			return await readline.question("You: ");
		} catch {
			return null;
		}
	};

	const render = (text: string): void => {
		console.log(`Assistant: ${text}`);
	};

	const agent = new Agent(
		modelClient,
		{
			model: "openai/gpt-4o-mini",
			maxTokens: 1024,
			maxTurns: 20,
		},
		getUserInput,
		render,
	);

	try {
		await agent.run();
	} finally {
		readline.close();
	}
}

main().catch((error) => {
	const message =
		error instanceof Error
			? error.message
			: "Unknown startup error";
	console.error(`Fatal error: ${message}`);
	process.exit(1);
});

Run it:

bun src/main.ts

Example conversation from a real run:

Assistant: Chat started. Submit empty input to exit.
You: hello world
Assistant: Hello! How can I assist you today?
You: what is the weather in london
Assistant: I don't have real-time data access to provide current weather information. However, you can easily check the weather in London by using a weather website, app, or a voice-activated assistant. If you're looking for typical weather conditions in London or historical data, I can help with that!

This result is expected in Part 1. The loop is working, but there are no tools yet, so the model cannot fetch live weather. Part 2 adds a custom tool so the agent can call external systems for real-time data.

What to Ignore for Now

To stay focused on the core idea, treat these as implementation details in Part 1:

API client wiring (fetch setup and headers).
Environment loading from .env (handled by Bun runtime).
Terminal wiring (readline).

The only required mental model in this part is still the same:

Read input.
Append to conversation state.
Call model with full state.
Append assistant response.
Display response.
Repeat until exit condition.

Why This Design Matters

1. Testability

Injected input, output, and model dependencies make Agent.run() easy to test with fakes, which means regressions in loop behavior are caught before deployment. In practice, this lowers the risk of breaking basic interaction flow when adding tools, retries, or routing logic later. Mock Service Worker is a practical choice for mocking LLM API responses.

2. Replaceable Model Providers

Provider portability protects the architecture from API churn, pricing changes, and model quality shifts. With ModelClient as an interface, the core loop remains stable while provider-specific code stays isolated in one adapter. In production systems, this isolation reduces migration cost and makes multi-provider fallback strategies realistic. It also makes it easier to switch providers or models based on use case.

3. Explicit Failure Handling

Explicit per-turn error handling improves runtime resilience. A single failed model call can be surfaced to the user without killing the process, which preserves session continuity and helps operators debug failure patterns. This becomes especially important when rate limits, transient network issues, and provider timeouts appear under load.

4. Operational Guardrails

A max-turn cap and clear exit paths are basic but important guardrails. They prevent unbounded loops, runaway token spend, and confusing terminal behavior during incidents. Production hardening starts with these deterministic limits, then expands to rate limits, retries with backoff, and structured logging.

Common Pitfalls in Basic Agent Loops

Pitfall 1: Forgetting Full Context

If only the latest user message is sent each time, the assistant appears to forget prior turns. Beyond poor UX, this also breaks task continuity and can cause inconsistent outputs in multi-step workflows. Stateful conversation replay is what makes the system behave like a coherent session instead of unrelated single prompts.

Pitfall 2: Mixing Business Logic and I/O

When stdin, rendering, and model calls are tightly coupled, testing and debugging become painful. Tight coupling also blocks reuse, for example, reusing the same loop in CLI, API, and background worker contexts. Separating concerns early keeps future integrations low-risk.

Pitfall 3: No Exit Strategy

Without turn limits and stop conditions, loops can run indefinitely. In production this translates directly into cost leaks and hard-to-diagnose runtime behavior. Deterministic stop rules are required for operational predictability.

Pitfall 4: Silent Failure Paths

If model errors are swallowed, the interface appears unresponsive with no clear reason. Silent failures are expensive during incident response because there is no signal for users or operators. Even minimal loops should return visible errors and leave room for structured logs later.

Lessons Learned

The core loop is the foundation, behavior comes from control flow and state before tools are introduced.
Stateless model APIs require explicit conversation persistence in application code.
Clean boundaries between loop logic, model adapter, and I/O make future hardening much easier.
Production readiness starts with simple controls, explicit errors, deterministic exits, and replaceable integrations.

What Comes Next in Part 2

In Part 2, the implementation adds a custom tool and the request and response handshake that lets the model ask the runtime to perform real actions. That is the step where the agent starts affecting the outside world.

Keeping this baseline code clean makes the next layers, tools and skills, easier to reason about and maintain.

Official References

Why Elixir and the BEAM Might Be the Best Runtime for AI Agents

Sat, 28 Mar 2026 00:00:00 GMT

Overview

Most agent frameworks today are built in TypeScript or Python. They are mature, well-supported, and familiar. But there is a growing argument that the BEAM VM, the runtime behind Erlang and Elixir, is a better foundation for running agent systems at scale.

I came across Jido, an open-source Elixir agent framework that shipped version 2.0 in March 2026. It makes some strong claims about why the BEAM is the right runtime for agents. Some of those claims are worth understanding, even if you have no plans to leave TypeScript.

TLDR: Jido is an Elixir agent framework built on the BEAM VM. The BEAM gives agents multi-core concurrency, per-process fault isolation, and supervision out of the box. Jido adds a command-pattern architecture that separates agent decisions from side effects. That last idea is the most useful takeaway for TypeScript developers.

Sections

What Is Jido
Why These Patterns Matter for Agent Systems
The BEAM Runtime: Why Jido Chose Elixir
So Why Use Jido?
Official References

What Is Jido

Jido is an Elixir framework for orchestrating agents, built on tried and tested patterns and designed to take advantage of the BEAM VM. It shipped version 2.0 in March 2026. Here are the core ideas from their release post.

Agents as Data

A Jido agent is a struct with state, actions, and tools. There is no hidden mutation or implicit state management. The agent is just data.

The Command Pattern

All agent logic flows through a single function, cmd/2. You pass in an agent and an action. You get back an updated agent and a list of directives, typed data structures that describe side effects. The agent never executes side effects directly. It describes what should happen, and a separate runtime layer executes it.

This is the <a href="#" class="glossary-link" data-glossary="command-pattern">command pattern</a>. Decisions are pure. Execution is separate.

Pluggable Strategies

How an agent processes actions is controlled by a strategy module. Jido ships with Direct (sequential execution) and <a href="#" class="glossary-link" data-glossary="fsm">FSM</a> (state machines with transition guards). Jido AI adds LLM-based strategies like <a href="#" class="glossary-link" data-glossary="react">ReAct</a>, <a href="#" class="glossary-link" data-glossary="cot">Chain-of-Thought</a>, and <a href="#" class="glossary-link" data-glossary="tot">Tree-of-Thoughts</a>.

The key point is that the strategy is swappable. The agent's logic, tools, and state stay the same. Only the execution pattern changes.

Schema-Validated Actions as LLM Tools

Actions are defined once with a typed schema. They work both as executable code and as tool definitions the LLM can call. There is no separate tool definition to maintain.

Why These Patterns Matter for Agent Systems

The most useful ideas from Jido are not Elixir-specific. They are architectural patterns that can improve any agent system. Here is what they look like in TypeScript and why you should care.

Separate Decisions from Side Effects

This is the biggest takeaway. In a typical TypeScript agent, the tool call happens inside the agent loop. The decision and the execution are interleaved.

Jido's approach is different. The agent's decision function is pure. It takes state and an action, returns new state and a list of commands. A separate layer executes the commands.

Note: This is simplified pseudocode to illustrate the concept, not actual implementation.

interface Directive {
  type: string;
  payload: Record<string, unknown>;
}

interface CommandResult {
  state: AgentState;
  directives: Directive[];
}

// Pure function. No network calls, no database writes, no side effects.
function decide(state: AgentState, action: Action): CommandResult {
  const newState = { ...state, orderCount: state.orderCount + 1 };
  const directives: Directive[] = [
    { type: "send_email", payload: { to: "customer@example.com", subject: "Order confirmed" } },
    { type: "update_database", payload: { table: "orders", id: "123", status: "confirmed" } },
  ];

  return { state: newState, directives };
}

// Separate runtime executes directives after the decision is made.
async function execute(directives: Directive[]): Promise<void> {
  for (const directive of directives) {
    await handlers[directive.type](directive.payload);
  }
}

This separation buys you several things.

Testability Without Mocks

When agent decisions are pure functions, testing does not require mocking HTTP clients, databases, or LLM providers. You pass in state and an action. You assert on the returned state and directives. No setup, no teardown, no flaky network dependencies.

// Test the decision, not the execution
test("processes order and increments count", () => {
  const state: AgentState = { orderCount: 0 };
  const result = decide(state, { type: "process_order", orderId: "123" });

  expect(result.state.orderCount).toBe(1);
  expect(result.directives).toContainEqual({
    type: "send_email",
    payload: expect.objectContaining({ to: "customer@example.com" }),
  });
});

Compare this to testing an agent where the tool call triggers a real HTTP request inside the loop. You need to mock the dependency, which couples your test to the tool's implementation rather than the agent's decision.

Observability for Free

When side effects are described as data before execution, you get observability properties without extra work:

Log every decision before execution. Every directive is a data structure. Log it, inspect it, or require approval before it runs.
Replay agent decisions. Feed the same state and action into the decision function. Get the same directives back. Deterministic replay without hitting external systems.
Dry-run mode. Run the decision logic, collect the directives, skip execution. Useful for testing new strategies against production-like input.
Audit trails. Every directive is a record of what the agent intended to do. Combined with execution results, you get a complete decision history.

Interleaved (typical TypeScript agent)
┌─────────────────────────────────────────────────┐
│ Agent Loop                                      │
│   Think ──▶ Call Tool ──▶ Get Result ──▶ Think  │
│              ▲                                  │
│              Side effect happens here.          │
│              Decision and execution are coupled.│
└─────────────────────────────────────────────────┘

Separated (command pattern)
┌──────────────────────┐    ┌─────────────────────┐
│ Decision Layer       │    │ Execution Layer     │
│                      │    │                     │
│ decide(state, action)│───▶│ Execute directives  │
│   Returns:           │    │   Log each one      │
│   new state          │    │   Approve if needed │
│   directives (data)  │    │   Then execute      │
│                      │    │                     │
│ Pure. Testable.      │    │ Observable. Audited.│
└──────────────────────┘    └─────────────────────┘

This connects directly to the observability requirements from Part 4 of the agent series. The same primitives, decision logging, replay, audit trails, become easier to implement when the architecture enforces separation.

The BEAM Runtime: Why Jido Chose Elixir

The patterns above work in any language. But Jido chose Elixir for a reason. The BEAM VM, the runtime behind Erlang and Elixir, was built by Ericsson in the 1980s for telecom switching systems. It has properties that are hard to replicate in Node.js.

Where Node.js Is Strong

Node.js handles I/O-bound concurrent work well. When an agent calls an LLM API or queries a database, await yields control back to the event loop. Other agents continue running. A single Node process can manage thousands of concurrent agents all waiting on network responses.

For most agent workloads today, where the majority of time is spent waiting on LLM API responses, Node's concurrency model works fine. TypeScript also has the largest ecosystem of AI/LLM libraries, the widest provider support, and a far larger developer community than Elixir.

Where the BEAM Has a Structural Advantage

Multi-core utilization. Node.js runs JavaScript on a single core by default. Using additional cores requires worker_threads or the cluster module, which add complexity around state sharing and lifecycle management. The BEAM runs a scheduler per CPU core and distributes processes across all of them automatically. No application code changes needed.

This matters when agents do non-trivial local computation: parsing large LLM responses, running complex decision trees, serializing state. These are CPU-bound operations that block the Node event loop but get time-sliced across cores on the BEAM.

Fault isolation. In Node, all agents share a single process. An unhandled error, a memory leak in one agent's closure, or an out-of-memory condition affects every agent in that process. On the BEAM, each agent runs in its own lightweight process with its own heap and garbage collector. One agent crashing does not affect others.

Node.js
┌────────────────────────────────────────────────┐
│              Single Node Process               │
│                                                │
│  Agent A    Agent B    Agent C    Agent D      │
│  (promise)  (promise)  (promise)  (promise)    │
│                                                │
│  Shared event loop, shared memory, shared fate │
│  One OOM or unhandled error kills all agents   │
└────────────────────────────────────────────────┘

BEAM
┌────────────────────────────────────────────────┐
│              BEAM VM (all cores)               │
│                                                │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐           │
│  │Agent A  │ │Agent B  │ │Agent C  │  ...      │
│  │Own heap │ │Own heap │ │Own heap │           │
│  │Own GC   │ │Own GC   │ │Own GC   │           │
│  └─────────┘ └─────────┘ └─────────┘           │
│                                                │
│  Agent B crashes. Supervisor restarts it.      │
│  Agents A and C are unaffected.                │
└────────────────────────────────────────────────┘

Supervision trees. Processes are organized into hierarchies where parent processes monitor children and restart them on failure. In Node, you would need something like PM2 or custom retry logic. On the BEAM, this is built into the runtime.

Built-in distribution. The BEAM natively supports connecting VMs across machines. Processes on different nodes communicate directly without an external message broker. In TypeScript, scaling agents across machines typically requires a distributed data store (Redis, DynamoDB), a message queue, or a messaging protocol like gRPC.

An Important Caveat

Whether these advantages matter in practice depends on the workload. If your agents spend 95% of their time waiting on LLM API responses, the multi-core and CPU scheduling benefits are largely theoretical. Without instrumenting a real agent system and measuring where time is actually spent, the runtime comparison is an architectural argument, not an empirical one.

So Why Use Jido?

If you already know Elixir and are comfortable with its concurrency primitives, Jido looks like a well-designed framework. The architecture is clean, the separation of concerns is strong, and the BEAM gives you runtime properties that would take significant effort to replicate in Node.js. For teams already invested in the Elixir ecosystem, it is a compelling choice for building agent systems.

If you are coming from TypeScript, the calculus is different. Node.js handles I/O-bound agent workloads well, the ecosystem of AI libraries is far larger, and the developer community around TypeScript agents is more active. Switching runtimes is a big commitment, and for most agent systems today, the bottleneck is not the runtime. It is the LLM API latency, the quality of your prompts, and the reliability of your orchestration logic.

The reasons to seriously consider Jido and the BEAM come down to scale and resilience. If you are building a system with hundreds of concurrent agents, need hard fault isolation between them, want automatic multi-core utilization without infrastructure complexity, or require agents that stay up for months without restarts, those are problems the BEAM was designed to solve.

For everyone else, the architectural ideas are the real value. The command pattern, pluggable strategies, and schema-validated actions are patterns you can adopt in TypeScript today to make your agents more testable, observable, and maintainable.

<div id="glossary-data" style="display:none;"> <div data-glossary-id="command-pattern"> <strong>Command Pattern</strong> <p>The command pattern is a design pattern where an operation is represented as an object (or data structure) rather than being executed immediately. Instead of calling a function that sends an email, you create a "send email" command object and hand it to something else to execute. This decouples the decision ("what should happen") from the execution ("make it happen"). It makes it easy to log, queue, undo, replay, or test operations because they are just data until someone runs them.</p> <p><a href="https://en.wikipedia.org/wiki/Command_pattern" target="_blank" rel="noopener">Read more: Command pattern, Wikipedia →</a></p> </div> <div data-glossary-id="fsm"> <strong>Finite State Machine (FSM)</strong> <p>A finite state machine is a model where a system can only be in one state at a time, and transitions between states are triggered by specific events. Think of a traffic light: it is either red, yellow, or green, and it moves between those states in a defined order. In agent systems, FSMs are useful for modelling workflows where an agent should only do certain things depending on what stage it is in. For example, a support agent might move through states like "gathering info", "looking up order", and "resolving issue", with rules about which transitions are allowed.</p> <p><a href="https://en.wikipedia.org/wiki/Finite-state_machine" target="_blank" rel="noopener">Read more: Finite-state machine, Wikipedia →</a></p> </div> <div data-glossary-id="react"> <strong>ReAct (Reasoning + Acting)</strong> <p>ReAct is a prompting strategy where the LLM alternates between reasoning about the current situation and taking an action (like calling a tool). The loop looks like: think about what to do, do it, observe the result, think again. Most tool-calling agent loops today follow this pattern, even if they do not call it ReAct explicitly.</p> <p><a href="https://arxiv.org/abs/2210.03629" target="_blank" rel="noopener">Read more: ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022) →</a></p> </div> <div data-glossary-id="cot"> <strong>Chain-of-Thought (CoT)</strong> <p>Chain-of-Thought prompting asks the LLM to show its working. Instead of jumping straight to an answer, the model generates intermediate reasoning steps. For example, instead of answering "what is 23 minus 7 plus 12" with "28", it would say "23 minus 7 is 16, then 16 plus 12 is 28". This improves accuracy on tasks that require multi-step reasoning like maths, logic, and planning.</p> <p><a href="https://arxiv.org/abs/2201.11903" target="_blank" rel="noopener">Read more: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022) →</a></p> </div> <div data-glossary-id="tot"> <strong>Tree-of-Thoughts (ToT)</strong> <p>Tree-of-Thoughts extends Chain-of-Thought by exploring multiple reasoning paths in parallel rather than following a single chain. The model generates several possible next steps, evaluates which ones are most promising, and can backtrack if a path leads nowhere. Think of it as the difference between walking down one corridor and mapping out a maze. More expensive (more LLM calls) but significantly better on tasks that require planning or search.</p> <p><a href="https://arxiv.org/abs/2305.10601" target="_blank" rel="noopener">Read more: Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Yao et al., 2023) →</a></p> </div> </div>

document.querySelectorAll(".glossary-link").forEach((link) => { link.addEventListener("click", (e) => { e.preventDefault(); const key = link.getAttribute("data-glossary"); const source = document.querySelector([data-glossary-id="${key}"]); if (!source) return;

  // If clicking the same term, toggle it off
  if (activeCard && activeCard.getAttribute("data-active-glossary") === key) {
    activeCard.style.maxHeight = activeCard.scrollHeight + "px";
    requestAnimationFrame(() => {
      activeCard.style.maxHeight = "0";
      activeCard.style.opacity = "0";
    });
    setTimeout(() => { activeCard.remove(); activeCard = null; }, 200);
    return;
  }

  // Remove existing card
  if (activeCard) {
    activeCard.remove();
    activeCard = null;
  }

  // Find the closest block-level parent (p, li, etc.)
  const block = link.closest("p") || link.closest("li") || link.parentElement;

  // Create the card
  const card = document.createElement("div");
  card.setAttribute("data-active-glossary", key);
  card.innerHTML = source.innerHTML;
  card.style.cssText = `
    margin: 0.75rem 0 1rem 0;
    padding: 1rem 1.25rem;
    border-left: 3px solid #3b82f6;
    border-radius: 0.375rem;
    background: #f0f4ff;
    font-size: 0.925rem;
    line-height: 1.6;
    overflow: hidden;
    max-height: 0;
    opacity: 0;
    transition: max-height 0.25s ease, opacity 0.2s ease;
  `;

  // Dark mode
  if (document.documentElement.classList.contains("dark")) {
    card.style.background = "#1e293b";
    card.style.borderLeftColor = "#60a5fa";
  }

  // Style inner elements
  card.querySelectorAll("p").forEach((p) => { p.style.margin = "0.4rem 0"; });
  card.querySelector("strong").style.display = "block";
  card.querySelector("strong").style.marginBottom = "0.25rem";

  block.after(card);
  activeCard = card;

  // Animate in
  requestAnimationFrame(() => {
    card.style.maxHeight = card.scrollHeight + "px";
    card.style.opacity = "1";
  });
});

}); }); </script>

Official References

Railway CDN Caching Incident: When Opt-In Becomes Opt-Everyone-In

Sun, 29 Mar 2026 23:00:00 GMT

Overview

On March 30th, 2026, Railway experienced an incident where a CDN configuration update accidentally enabled caching for domains that had not enabled CDN. For 52 minutes, HTTP GET responses were incorrectly cached across ~0.05% of Railway domains with CDN disabled. Railway says cached responses may have been served to users other than the original requester, including authenticated responses without Set-Cookie.

TLDR: A configuration change to Railway's CDN provider accidentally enabled caching on domains that had not enabled CDN. Authenticated responses (without Set-Cookie headers) were cached at the edge and may have been served to other users. Cache-Control directives were respected where set, but most GET responses without explicit cache headers were cached by default.

Official incident report

Timeline of Events

Mar 30, 10:42 UTC - CDN configuration update deployed
                     Accidentally enables caching for CDN-disabled domains
    ↓
Mar 30, 10:42-11:34 UTC - 52-minute window
                           Authenticated GET responses cached and served
                           to wrong users across ~0.05% of domains
    ↓
Mar 30, 11:34 UTC - Issue identified and change reverted
                     All cached assets purged globally

How Railway's CDN Works

Railway offers CDN caching as an opt-in feature. When enabled, your application's responses are cached at edge servers around the world for faster delivery. When disabled, requests route directly to your application with no caching layer in between.

┌─────────────────────────────────────────────────────────────┐
│                    CDN Enabled (Opt-In)                     │
│                                                             │
│  User Request                                               │
│       │                                                     │
│       ▼                                                     │
│  ┌──────────┐    Cache     ┌──────────────┐                 │
│  │  CDN     │────HIT──────▶│  Serve from  │                 │
│  │  Edge    │              │  edge cache  │                 │
│  │  Server  │              └──────────────┘                 │
│  │          │                                               │
│  │          │    Cache     ┌──────────────┐                 │
│  │          │────MISS─────▶│  Forward to  │                 │
│  └──────────┘              │  origin app  │                 │
│                            └──────────────┘                 │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                   CDN Disabled (Default)                    │
│                                                             │
│  User Request                                               │
│       │                                                     │
│       ▼                                                     │
│  ┌──────────────────────────────────────────┐               │
│  │  Direct passthrough to origin app        │               │
│  │  No caching, no edge servers involved    │               │
│  └──────────────────────────────────────────┘               │
└─────────────────────────────────────────────────────────────┘

The distinction matters. There are plenty of reasons to keep CDN off:

Your application returns user-specific or authenticated data that should never be shared.
Your origin is fast enough and you have no need for edge caching.
Your responses are highly dynamic and caching would add complexity without meaningful performance gains.
You want full control over response handling at the application level.

What Went Wrong

The configuration change flipped the caching behavior for domains that should have been in passthrough mode. Here is what the CDN did during the 52-minute window:

┌─────────────────────────────────────────────────────────────────┐
│              During the Incident (10:42 - 11:34 UTC)            │
│                                                                 │
│  User A: GET /api/dashboard                                     │
│       │                                                         │
│       ▼                                                         │
│  ┌──────────┐                                                   │
│  │  CDN     │  1. Cache MISS (first request)                    │
│  │  Edge    │  2. Forwards to origin app                        │
│  │  Server  │  3. Receives response with User A's data          │
│  │          │  4. CACHES the response <-- Should not happen!    │
│  └──────────┘                                                   │
│                                                                 │
│  User B: GET /api/dashboard (same URL, seconds later)           │
│       │                                                         │
│       ▼                                                         │
│  ┌──────────┐                                                   │
│  │  CDN     │  1. Cache HIT                                     │
│  │  Edge    │  2. Returns User A's cached response              │
│  │  Server  │  3. User B sees User A's data  <-- Cross-user     │
│  │          │                                    data exposure! │
│  └──────────┘                                                   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

The above is an illustrative hypothetical. Railway did not disclose specific endpoints or data types involved, but this pattern shows how cross-user data exposure could occur.

There are two important details from Railway's report:

Set-Cookie headers were not cached. Railway says Set-Cookie response headers were not cached during the incident. This means session cookies in response headers were not replayed from cache, but response bodies (which could contain user-specific data) may have been.

Cache-Control directives were respected. If your application explicitly set Cache-Control: no-store or Cache-Control: private, the CDN honoured that. The problem was that most GET responses without explicit cache headers were cached by default.

The Real Problem: Cache-Control Defaults

This is the core of the issue. Most web frameworks do not set Cache-Control headers on API responses by default. Developers rarely think about caching headers on authenticated endpoints because they assume the response will go directly to the requesting client.

// A typical Express API endpoint
// No Cache-Control header set, because why would you?
// The CDN is supposed to be OFF.
app.get("/api/dashboard", authMiddleware, async (req, res) => {
    const user = await getUser(req.userId);
    res.json({
        name: user.name,
        email: user.email,
        billing: user.billingInfo,
    });
});

When the CDN is disabled, this is perfectly fine. The response goes straight from your app to the user. But the moment a CDN is inserted into the path, even accidentally, the lack of Cache-Control headers becomes a vulnerability.

// What Railway's CDN saw for a GET request without Cache-Control:
//
// Response headers:
//   Content-Type: application/json
//   (no Cache-Control header)
//
// CDN behavior during this incident: "No cache directive? I'll cache it."

During this incident, Railway's CDN treated GET responses without explicit cache directives as cacheable. The exact default caching behaviour varies by CDN provider and configuration, but the broader point holds: if you are not setting Cache-Control headers, you are leaving the caching decision up to whatever intermediary sits in the request path. Railway's users never opted into the CDN in the first place, so they had no reason to think about this.

How a Managed CDN Decides What to Cache

Understanding CDN caching decisions helps explain why this went wrong. Below is a simplified model of the decision points that were relevant during this incident. Note that exact behaviour varies by CDN provider and configuration.

Incoming Response from Origin
         │
         ▼
┌─────────────────────────┐
│ Is caching enabled for  │──── NO ─────▶ Passthrough (no caching)
│ this domain/route?      │
└─────────────────────────┘
         │ YES
         ▼
┌─────────────────────────┐
│ Has Cache-Control:      │
│ no-store or private?    │──── YES ────▶ Do NOT cache
└─────────────────────────┘
         │ NO
         ▼
┌─────────────────────────┐
│ Has Cache-Control:      │
│ max-age or s-maxage?    │──── YES ────▶ Cache for specified duration
└─────────────────────────┘
         │ NO
         ▼
┌─────────────────────────┐
│ Has Expires header?     │──── YES ────▶ Cache until expiry
└─────────────────────────┘
         │ NO
         ▼
┌─────────────────────────┐
│ Is it a GET request?    │──── YES ────▶ Apply default caching <-- HERE
└─────────────────────────┘
         │ NO
         ▼
     Do NOT cache

Note: Per RFC 9111 §3.5, a spec-compliant shared cache must not reuse a response to a request containing an Authorization header unless the response explicitly allows it (e.g. via public, s-maxage, or must-revalidate). However, managed CDN products do not always follow this strictly, and Railway's CDN cached authenticated responses during this incident.

That bottom path, "apply default caching", is what caught Railway's users. Their applications were not setting cache headers because they were never meant to go through a CDN. When the CDN was accidentally enabled, the responses fell through to the default caching behaviour.

What Could Have Prevented This

1. Defence in Depth: Always Set Cache-Control Headers

Even if you think your responses will never be cached, set Cache-Control headers on authenticated endpoints. This is defence in depth. You are not just relying on infrastructure configuration being correct, you are telling any intermediary in the request path how to handle your response.

// Middleware that sets Cache-Control on all authenticated routes
function noCacheMiddleware(
    req: Request,
    res: Response,
    next: NextFunction
) {
    res.set("Cache-Control", "no-store");
    next();
}

// Apply to all authenticated routes
app.use("/api", authMiddleware, noCacheMiddleware);

// Or per-route
app.get("/api/dashboard", authMiddleware, noCacheMiddleware, async (req, res) => {
    const user = await getUser(req.userId);
    res.json({
        name: user.name,
        email: user.email,
    });
});

The key directives to know:

no-store: Do not store this response in any cache, ever. This is the strongest directive and is sufficient on its own for sensitive authenticated responses.
private: Only the end user's browser may cache this. No shared caches (CDNs, proxies). Useful if you want browser caching but not CDN caching.
no-cache: You may store it, but you must revalidate with the origin before serving.

For sensitive authenticated API responses, Cache-Control: no-store is the simplest and most effective choice. If you want to allow browser caching but prevent shared/CDN caching, use Cache-Control: private, no-cache.

2. Understanding Cache Keys

A cache key is how a CDN decides whether it has already seen a request before. By default, most CDNs construct the cache key from the HTTP method and the URL, nothing else.

Default Cache Key Construction
──────────────────────────────

  Request from User A:  GET /api/dashboard
  Cache key:            GET:/api/dashboard

  Request from User B:  GET /api/dashboard
  Cache key:            GET:/api/dashboard  <-- Same key, same cached response

  The CDN does not look at:
  - Authorization header
  - Cookie header
  - Any other request header
  (unless explicitly told to via Vary or CDN-specific config)

This is exactly why Railway's incident leaked data. The CDN saw GET /api/dashboard from User A, cached the response, and then served that same cached response to User B because the cache key was identical. The CDN had no way to know these were different users, because nothing in the cache key distinguished them.

You can influence cache keys in two ways: through the Vary response header (standard HTTP), or through CDN-specific configuration (e.g. Cloudflare's Cache Key settings, CloudFront's cache policies).

3. Vary Header for Cacheable Variants

The Vary header tells caches to include additional request headers in the cache key. This is useful when you intentionally want to cache responses that differ per some request attribute (e.g. Accept-Language, Accept-Encoding).

// Example: intentionally caching public content that varies by language
app.get("/api/articles", async (req, res) => {
    res.set("Vary", "Accept-Language");
    res.set("Cache-Control", "public, max-age=3600");

    const articles = await getArticles(req.headers["accept-language"]);
    res.json(articles);
});

With Vary: Accept-Language, the cache key changes from just the URL to the URL plus the language header. Users requesting different languages get separate cache entries.

Important: Vary is not the right tool for protecting personalized authenticated responses. For those, Cache-Control: no-store is the correct approach. Do not rely on Vary: Authorization or Vary: Cookie as a security mechanism. These headers can explode cache cardinality, behave inconsistently across CDN providers, and do not prevent caching, they just partition it. If a response should never be shared between users, prevent caching entirely.

4. CDN-Level Safeguards

From a platform perspective (which is Railway's responsibility), there are architectural safeguards that can prevent this class of issue:

┌─────────────────────────────────────────────────────────────────┐
│                   CDN Configuration Safeguards                  │
│                                                                 │
│  1. Separate CDN Configs                                        │
│     ┌──────────────┐         ┌──────────────┐                   │
│     │ CDN-Enabled  │         │ CDN-Disabled │                   │
│     │ Config Pool  │         │ Config Pool  │                   │
│     │              │         │              │                   │
│     │ cache: true  │         │ cache: false │                   │
│     │ edge: active │         │ passthrough  │                   │
│     └──────────────┘         └──────────────┘                   │
│     Changes to one pool cannot affect the other                 │
│                                                                 │
│  2. Canary Rollouts                                             │
│     Deploy to 1% ──▶ Monitor ──▶ 10% ──▶ Monitor ──▶ 100%       │
│     (hours, not minutes)                                        │
│                                                                 │
│  3. Automated Cache Behaviour Tests                             │
│     Before every config change:                                 │
│     - Verify CDN-disabled domains return no cache headers       │
│     - Verify CDN-enabled domains cache correctly                │
│     - Verify Set-Cookie responses are never cached              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Railway's post-mortem mentions they have already rolled out additional tests for correct/incorrect caching behaviours, and moved to aggressive sharding of CDN rollouts over hours instead of minutes. Both of these directly address the problem. Railway also said users with affected domains would be notified by email.

Note: Railway's report does not disclose the CDN provider, exact cache key construction, TTLs, authentication mechanisms involved, or the number of incorrect responses served. The analysis sections above are general HTTP caching guidance rather than Railway-confirmed implementation details.

5. Framework-Level Defaults

If you are building a web framework or a platform, consider making safe cache defaults the norm rather than the exception:

// A framework that defaults to no-cache for authenticated responses
class SecureRouter {
    authenticatedRoute(
        path: string,
        handler: RequestHandler
    ) {
        return this.router.get(path, (req, res, next) => {
            // Always set safe defaults for authenticated routes
            if (!res.getHeader("Cache-Control")) {
                res.set("Cache-Control", "no-store, private");
            }
            handler(req, res, next);
        });
    }
}

This is the principle of secure defaults. Developers should have to opt in to caching on authenticated routes, not opt out.

Lessons Learned

CDN configuration is a trust boundary. Railway called this out explicitly in their report. Domains with CDN disabled should never have content cached. When a single configuration change can cross that boundary, you need both infrastructure-level and application-level defences.

Defaults matter more than you think. Railway's CDN cached GET responses by default when no Cache-Control header was present. Most web frameworks do not set Cache-Control on API responses by default. This gap between CDN behaviour and framework defaults is where incidents like this live.

Defence in depth is not optional. Setting Cache-Control: no-store on authenticated endpoints costs nothing and protects against an entire class of infrastructure misconfiguration. It is the WHERE 1=1 of caching, a safety net you hope you never need.

Slow rollouts save incidents. Railway has since moved to sharding CDN rollouts over hours instead of minutes. If this change had rolled out to 1% of domains first with monitoring, the 52-minute exposure window would have been much shorter.

This pattern will keep repeating. Any platform that sits behind a CDN is one misconfiguration away from the same issue. The fix is not just better infrastructure processes, it is application-level awareness that your response might be cached by something you did not put there.