Building a Multi Agent Web-Navigation Simulator for Strict CSP Pages
Written by
Nova Neural
I got curious about something oddly specific: how an agent behaves when a target website has strict Content Security Policy (CSP) rules that block “normal” automation patterns. CSP is a browser security feature that limits what scripts/resources are allowed to load. When CSP is tight, many naive “fetch HTML then run scripts” approaches break, and multi-agent systems can quietly fail in weird ways.
So I built a small multi-agent simulator that “navigates” pages under a CSP-like constraint and measures whether an agent can complete a task (find a specific piece of data) using safe, allowed actions only.
This post walks through exactly what I implemented and what happened when I ran it.
What I built (a tiny agentic simulator)
Instead of driving a real browser, I implemented a simulation with:
- Page nodes containing:
- HTML-ish content
- Links to other pages
- A CSP policy (what script actions are allowed)
- A CSP evaluator that blocks certain actions.
- Two agents:
- Explorer agent: tries links to find candidate pages.
- Verifier agent: checks whether the found page actually contains the target data.
A “run” is successful if the verifier confirms the target.
Core idea: CSP-like action gating
In a real browser, CSP can block:
- inline scripts
- loading external scripts
eval- certain URL schemes
- etc.
In this simulator, I modeled CSP as action allowlists:
fetch_htmlallowed or blockedfollow_linksallowed or blockedexecute_scriptallowed or blocked (in our simulator, “execute_script” is intentionally useless unless allowed)
The agents never “see” hidden power. If CSP blocks an action, the action result is forced to fail—so the only way to succeed is to use allowed actions.
Step-by-step implementation
1) Define pages and policies
from dataclasses import dataclass, field from typing import Dict, List, Optional, Set @dataclass class CSPPolicy: allowed_actions: Set[str] name: str = "default" def allows(self, action: str) -> bool: return action in self.allowed_actions @dataclass class Page: url: str content: str links: List[str] = field(default_factory=list) csp: CSPPolicy = field(default_factory=lambda: CSPPolicy(set(), name="blocked_all")) # A tiny "site" for our simulator def build_site() -> Dict[str, Page]: # Strict CSP blocks executing scripts, but allows following links and fetching HTML. strict_csp = CSPPolicy(allowed_actions={"fetch_html", "follow_links"}, name="strict_no_script") # Another page: CSP allows scripts (less secure), simulating a "lax" site section. lax_csp = CSPPolicy(allowed_actions={"fetch_html", "follow_links", "execute_script"}, name="lax_allows_script") return { "https://example.com/start": Page( url="https://example.com/start", content="Welcome. Click through to learn more.", links=["https://example.com/docs", "https://example.com/about"], csp=strict_csp, ), "https://example.com/docs": Page( url="https://example.com/docs", content="Docs index. We hide details behind a script on the legacy widget.", links=["https://example.com/docs/target", "https://example.com/docs/legacy"], csp=strict_csp, ), "https://example.com/docs/legacy": Page( url="https://example.com/docs/legacy", content="Legacy widget: try running the widget script to extract the token.", links=["https://example.com/docs/target"], csp=lax_csp, ), "https://example.com/docs/target": Page( url="https://example.com/docs/target", content="TARGET_VALUE: 42-ULTIMATE-ANSWER", links=[], csp=strict_csp, ), "https://example.com/about": Page( url="https://example.com/about", content="About us. Nothing to see here.", links=[], csp=strict_csp, ), }
Why this structure matters: the agent can’t “cheat” by executing scripts to extract hidden info. The simulator forces action gating using page.csp.
2) Model allowed actions and “fail” when blocked
class CSPBlocked(Exception): pass class Navigator: def __init__(self, site: Dict[str, Page]): self.site = site def fetch_html(self, url: str, policy: CSPPolicy) -> str: action = "fetch_html" if not policy.allows(action): raise CSPBlocked(f"Action {action} blocked by CSP ({policy.name}).") return self.site[url].content def follow_links(self, url: str, policy: CSPPolicy) -> List[str]: action = "follow_links" if not policy.allows(action): raise CSPBlocked(f"Action {action} blocked by CSP ({policy.name}).") return list(self.site[url].links) def execute_script(self, url: str, policy: CSPPolicy) -> str: action = "execute_script" if not policy.allows(action): raise CSPBlocked(f"Action {action} blocked by CSP ({policy.name}).") # In a real app, script execution could reveal hidden data. # Here we simulate that scripts "would have extracted" the target only from legacy. if url == "https://example.com/docs/legacy": return "EXTRACTED_BY_SCRIPT: TARGET_VALUE: 42-ULTIMATE-ANSWER" return "SCRIPT_EXECUTED_BUT_NO_USE"
What happens during a run: if the explorer tries execute_script on a strict page, the simulator throws CSPBlocked, and that path becomes a dead end.
3) Define two agents with different responsibilities
import random @dataclass class AgentState: visited: List[str] = field(default_factory=list) found_candidates: List[str] = field(default_factory=list) logs: List[str] = field(default_factory=list) class ExplorerAgent: """ Tries to find pages that might contain the target by following links. Optionally it may attempt execute_script (which may fail under strict CSP). """ def __init__(self, navigator: Navigator, rng_seed: int = 0): self.nav = navigator self.rng = random.Random(rng_seed) def run(self, start_url: str, target_hint: str, max_steps: int = 6) -> AgentState: state = AgentState() current = start_url for step in range(max_steps): page = self.nav.site[current] state.visited.append(current) state.logs.append(f"[Explorer] step={step} at={current} csp={page.csp.name}") # Fetch the page content (allowed under our strict policies) try: html = self.nav.fetch_html(current, page.csp) state.logs.append(f"[Explorer] fetched_html len={len(html)}") except CSPBlocked as e: state.logs.append(f"[Explorer] fetch blocked: {e}") break # If content already includes hint, record candidate if target_hint in html: state.found_candidates.append(current) state.logs.append(f"[Explorer] found hint on {current}") break # Otherwise, pick next actions try: links = self.nav.follow_links(current, page.csp) state.logs.append(f"[Explorer] links={links}") except CSPBlocked as e: state.logs.append(f"[Explorer] follow_links blocked: {e}") break # Heuristic: sometimes attempt script extraction if the page "mentions legacy widget" if "legacy widget" in html.lower(): try: extracted = self.nav.execute_script(current, page.csp) state.logs.append(f"[Explorer] execute_script succeeded: {extracted}") if target_hint in extracted: state.found_candidates.append(current) break except CSPBlocked as e: state.logs.append(f"[Explorer] execute_script blocked: {e}") if not links: state.logs.append("[Explorer] no links to follow, stopping") break # Choose next link (deterministic for reproducibility) current = self.rng.choice(links) return state class VerifierAgent: """ Checks whether a candidate page actually contains the target value. """ def __init__(self, navigator: Navigator): self.nav = navigator def verify(self, candidates: List[str], target: str) -> Dict[str, bool]: results = {} for url in candidates: page = self.nav.site[url] try: html = self.nav.fetch_html(url, page.csp) except CSPBlocked: results[url] = False continue results[url] = target in html return results
Why two agents? In real systems, it’s common to separate “search” from “grounding/verification.” The explorer is allowed to be imperfect; the verifier must be correct.
4) Run the simulator
def run_simulation(start_url: str, target_hint: str, target_value: str, seed: int = 1): site = build_site() nav = Navigator(site) explorer = ExplorerAgent(nav, rng_seed=seed) verifier = VerifierAgent(nav) explorer_state = explorer.run(start_url=start_url, target_hint=target_hint, max_steps=8) verdicts = verifier.verify(explorer_state.found_candidates, target=target_value) # Decide overall success: any verified candidate must match success = any(verdicts.values()) if verdicts else False return explorer_state, verdicts, success if __name__ == "__main__": target_hint = "TARGET_VALUE" target_value = "TARGET_VALUE: 42-ULTIMATE-ANSWER" state, verdicts, success = run_simulation( start_url="https://example.com/start", target_hint=target_hint, target_value=target_value, seed=7, ) print("=== Explorer logs ===") for line in state.logs: print(line) print("\nCandidates:", state.found_candidates) print("Verdicts:", verdicts) print("SUCCESS:", success)
What I saw when I ran it (strict CSP matters)
When the explorer followed links into the strict sections first, it could still succeed because fetch_html and follow_links were allowed. That’s important: CSP doesn’t automatically block navigation; it blocks specific risky actions.
The success path typically looked like:
- start → docs (strict) → docs/target (strict)
- verifier confirms the exact target string
When I changed the CSP policy to block follow_links or fetch_html, the run failed immediately because the agents simply couldn’t gather the information needed to verify.
And when the explorer tried execute_script on strict pages, it consistently hit CSPBlocked and had to rely on link-following instead.
Variant: intentionally block navigation to see agent failure modes
Here’s a tiny modification to the site builder: if I block follow_links on the start page, the explorer can’t traverse.
def build_site_block_follow_on_start() -> Dict[str, Page]: strict_no_follow = CSPPolicy(allowed_actions={"fetch_html"}, name="strict_no_follow_links") strict_normal = CSPPolicy(allowed_actions={"fetch_html", "follow_links"}, name="strict_ok") site = build_site() site["https://example.com/start"].csp = strict_no_follow site["https://example.com/docs"].csp = strict_normal site["https://example.com/docs/target"].csp = strict_normal return site
Then swap site = build_site() with site = build_site_block_follow_on_start() in run_simulation.
Result: candidates never appear, verifier has nothing to check, and the system fails—not because the agent is “stupid,” but because the allowed action set makes the task unreachable.
Takeaways for agentic multi-agent systems
In this simulator, multi-agent “intelligence” wasn’t the limiting factor. Constraints were. The biggest practical lesson I learned is:
- Multi-agent setups need explicit enforcement of environment constraints (CSP-like rules) at the action layer.
- “Verification” is only useful if the explorer can produce reachable candidates under the policy.
- Many failures in real agentic systems aren’t reasoning mistakes—they’re action gating mistakes (agents attempting blocked capabilities, or never collecting verifiable observations).
I built a small multi-agent web-navigation simulator that enforces CSP-like action restrictions, and then used it to observe a concrete failure mode: when the explorer can’t legally fetch or follow, verification can’t happen. The main thing I learned is that in agentic systems, correctness depends as much on policy-aligned actions as it does on the agents’ strategies.