AI Agents Need a Real Browser

Headless browser automation gets blocked and cannot hand a login to a human. VNC-only automation solves the login and then forces the agent to push pixels. We built the thing that holds both ends: Agent Desktop, an open-source, containerized Linux desktop where a human and an AI agent work the same browser session at the same time.

The human connects over noVNC, in the browser, over HTTPS, and logs in like a normal person. Password, 2FA code, CAPTCHA, whatever the site demands. The agent connects over CDP and MCP tools and drives that same browser: same tabs, same cookies, same logged-in state.

Human-grade login, machine-grade driving.

Why the two obvious approaches fail

The headless route has a measured cost. A 2026 measurement study of 10,000 websites (40,000 page visits across four browser configurations) found Chromium headless hit a 15% soft-block rate against 7% for the other configurations, with 82% of the blocks attributable to bot detection (arXiv:2606.14525). The signals are not subtle: navigator.webdriver is true, CDP leaves cdc_ markers, the plugins array is empty, and the User-Agent literally contains HeadlessChrome.

The second problem is worse than the blocking. A headless browser has no human in it. When a console asks for a hardware key or a six-digit code, there is nobody to type it, and the run is over.

The pixel route has the opposite problem. Anthropic's computer-use toolset gives a model screenshot, left_click and type: it looks at a picture and clicks coordinates (Anthropic docs). OpenAI's CUA does the same thing, it "processes raw pixel data" and "uses a virtual mouse and keyboard" (OpenAI). Every action costs a screenshot, a vision inference, and a guess about where the button is. Playwright's own MCP server documents the better interface: it drives pages "through structured accessibility snapshots, bypassing the need for screenshots or visually-tuned models" (playwright-mcp).

Even the leading browser-agent library concedes the login half. Browser Use's official guidance for 2FA and SSO is "use Human in the loop: your user logs in once via the live view, then the agent reuses the session" (docs). That is the hole we closed, and we closed it structurally instead of with a workflow.

Three layers, each optional

The image is composable. You can take the desktop and stop, or you can add the agent side-doors.

The 25 is not marketing. @playwright/mcp is pinned to 0.0.83 in the Dockerfile, and a tools/list call against a running server returns exactly 25 browser_* tools with nothing else in the default set. The desktop-actions server exposes screenshot, click, type_text, key, scroll, list_windows, activate_window and open_url. Both live behind one token, one proxy, one audit stream, path-routed to /mcp and /desktop.

Log in once, then the agent drives

The session is the product. Everything else is plumbing around it.

Figure 1: One desktop, two drivers. The human path holds the credentials, the agent path only ever holds tools.

Two details make this hold up. Chromium refuses --remote-debugging-port on the default profile, so the profile lives in a dedicated --user-data-dir, mounted from a volume and pinned to a stable container hostname. The result is that the logged-in state survives restarts, image upgrades and pod reschedules. And the noVNC session stays visible while the agent works, which is a control, not a demo feature.

Security is the part we refuse to simplify

A desktop holding logged-in sessions is a crown jewel. CDP has no authentication at all, because Chrome was never designed for a remote caller: anyone who can route to the port owns the browser. So the network shape is the auth.

The reference manifests publish exactly one NodePort, and it is noVNC. The security model requires NetworkPolicies to pin which namespaces may reach CDP and MCP. The MCP proxy enforces the bearer token per request, validates Origin when a browser-context client sends one (the MCP spec marks Origin validation as MUST on Streamable HTTP), rewrites Host for the upstream's DNS-rebinding protection, and writes one JSON audit line to stdout per request. In Kubernetes, kubectl logs is your audit trail.

Secrets never enter git. scripts/create-secrets.sh pipes a password from stdin into a Secret, so the value never touches a file or a manifest, and CI runs gitleaks on every push with a planted-secret test that keeps the gate honest.

The tradeoffs we accepted, stated plainly: one human, one agent, one desktop, no multi-tenant isolation. The noVNC certificate is self-signed and generated at first start, so the private key never ships in an image layer. Chromium runs with --no-sandbox because inside a container the container is the sandbox. And prompt injection is real: a hostile page can try to make the agent do something, so you keep the session visible or isolate the profile when you browse untrusted sites.

MCP is the interface, not a vendor lock

We did not build an agent. We built a place for one. The session is exposed as MCP tools over Streamable HTTP, so any MCP client can drive it. We verified against three independent clients rather than one: raw HTTP, the @modelcontextprotocol/inspector CLI, and the Python mcp SDK. That covers Claude Desktop, Cursor, Hermes and your own code without a single adapter on our side.

mcp_servers:
  desktop-browser:
    url: http://127.0.0.1:8931/mcp
    headers:
      Authorization: Bearer ***

The workload that made us build it

A client runs a support console behind SSO with a hardware token. No API, no service account, and a vendor who will not create one. A human can log into it in ninety seconds. An agent can log into it never.

So a Technical Support Engineer opens the noVNC session, logs in once, and closes the tab. The profile persists on the volume. A Hermes Agent running qwen3.8:27b then works the authenticated session through the MCP tools: it reads the ticket queue, pulls the account record, correlates the error against our own logs, and drafts replies. When something needs a human judgement call, the engineer takes over in the same browser, in the same tab, with the cursor. Nobody re-logs in, and nobody watches a screenshot stream.

That is the whole argument. The login was a human's, so no bot wall fires. The driving is a machine's, so no pixels get pushed.

Run it

Anywhere Docker runs. ghcr.io/sevenseven-tech/agent-desktop-{base,cdp,mcp} are prebuilt for every main commit (latest plus the git SHA), and building from the Dockerfiles is fully equivalent. CI runs gitleaks, builds all three layers, boots the desktop, checks that the MCP proxy refuses to start without a token, and pushes the images.

# base desktop
docker build -t agent-desktop-base base/
docker run -d -p 6080:6080 -e VNC_PASSWORD=*** rand -base64 18)" agent-desktop-base
# human drives: https://localhost:6080

# same desktop with the agent side-doors
openssl rand -base64 18 > /tmp/mcp.token
docker build -t agent-desktop-cdp --build-arg BASE_IMAGE=agent-desktop-base cdp/
docker build -t agent-desktop-mcp --build-arg BASE_IMAGE=agent-desktop-cdp mcp/
docker run -d -p 6080:6080 -p 127.0.0.1:8931:8931 \
  -e VNC_PASSWORD=*** rand -base64 18)" \
  -e MCP_TOKEN="$(cat /tmp/mcp.token)" \
  -v "$PWD/profile:/root/browser-profile" --hostname desktop \
  agent-desktop-mcp
# agent drives: http://127.0.0.1:8931/mcp

Reference Kubernetes manifests ship with it, secret-free by design: ./scripts/create-secrets.sh desktop pipes the password from stdin, and kubectl apply -f manifests/ gives you the Deployment, the PVC for the browser profile, a ClusterIP for the agent ports, and a NodePort for noVNC only. We run ours on two boxes with self-hosted Kubernetes; nothing about the design depends on that.

What it is not: a scraping tool, and not an evasion tool. If a site blocks you, the honest answer is that you should not be automating that site. Agent Desktop exists for the case where you own the account and a human is willing to log in.

The repo is MIT, public, and it needs feedback: github.com/sevenseven-tech/agent-desktop. If you have a console behind a login and nobody will give you an API, sevenseven.tech can put this on your hardware and wire your agent into it.

Contact us at support@sevenseven.tech or fill out our contact form.