quickstarted¶
Test whether an AI agent can complete your quickstart using only your docs.
No API key, no cost, nothing to clone:
[ 0s] started on seatbelt
[ 2s] read https://docs.streamlit.io/get-started/installation
[ 3s] read https://docs.streamlit.io/get-started/fundamentals/main-concepts
[ 24s] blocked from the shell: checkip.amazonaws.com
[ 24s] check exited 0
[PASS] streamlit-quickstart (replay)
classification: passed
turns: 2, duration: 24.4s
backend: seatbelt
docs pages read: 2
Two pages, because that is the route a reader takes: install on one, the first app on the next. The fourth line is the sandbox refusing Streamlit's own call home, which is what enforcement looks like from outside.
The agent gets your documentation and nothing else: no browser, no search engine, no network of its own, and a prompt that forbids leaning on what it already knows about your project. A script you wrote decides whether it got there. When it does not, you get the page it was reading.
[FAIL] fastapi-quickstart (openai:gpt-5.2-2025-12-11)
classification: docs_gap
turns: 7, duration: 65.7s
backend: docker (python:3.12-slim)
success check exit code: 1 ( pip install "fastapi[standard]")
last docs page read before failure: https://fastapi.tiangolo.com/tutorial/first-steps/
docs pages read: 1
That is a real run, and the last two lines are the product. The harness owns the only tool that can read documentation, so the pages in the report are the pages the agent really read, and the one it was on when things went wrong is a fact rather than a guess.
The obvious reading of that run is that the model ignored an instruction. It is worth being careful, because that reading is wrong, and the record is what shows it. On the same day, with the same model and the same task, eight runs split cleanly:
| Pages read | Result |
|---|---|
first-steps/ only |
0 of 3 passed |
first-steps/ and /tutorial/ |
5 of 5 passed |
first-steps/ contains no install instruction at all. The install line lives on
the tutorial index, which a human reaches by clicking through in order and an
agent arriving from a search result never sees. So the failure is not a page
that is wrong. It is a page that is not self-contained, plus a task that named
only one page and let the harness blame it.
That is what this measures, and the distinction is the whole product: a run proves a reader arrived at a broken result and names the pages they saw getting there. Whether the page buried something, omitted something, or the model skipped it is a judgement, and the report hands you the evidence rather than the verdict. It is also why a task names a route rather than a page.
Start here¶
-
Install
One dependency plus the SDK for whichever model you point at your docs, or nothing at all with
uvx. -
Your first run
Scaffold a task against real documentation and point a model at it.
-
Writing tasks
Success checks you can copy, and the rules that keep a task from measuring luck.
-
Agent mode in depth
Choosing models, bounding cost, reading the trace.
What makes a result trustworthy¶
Three decisions do most of the work, and each has a page explaining why.
A script decides, never a model. The success script runs after the agent stops, in the same workspace, and its exit code is the verdict. An agent that reports triumph over a missing file still fails. Most of these scripts are two or three lines: the file exists, the import works, the output contains the number your page promised. How scoring works and checks you can copy
The sandbox is a boundary. Documentation hosts are unreachable from the
shell, so read_docs is the only route to a page and every read is recorded.
Commands run in a container, or under a kernel sandbox on macOS.
The egress proxy and Sandboxing
One run is one sample. Repeat runs produce a pass rate, and runs that died on a rate limit are excluded from it instead of being counted as documentation failures. Pass rates
Did the change help?¶
That is the question you have after you edit the page, and it is the one a single verdict cannot answer.
Every comparison carries a two-sided Fisher exact test, and when the samples were too small for any outcome to have cleared the bar, it says that instead of reporting a result. Three attempts a side never can. Four can. quickstarted diff
Does llms.txt help?¶
Nobody knows, because presence is easy to check and effect is not. quickstarted never scores affordances. It withholds them and measures what changes. Measuring llms.txt
For agents reading this¶
Every page here is available as raw Markdown: append .md to any URL. An index
of the whole site lives at
llms.txt. Both are
generated at build time from the same navigation as the site.