Writing tasks¶

The rules that separate a task which measures your docs from one that measures luck.

Start from a scaffold¶

quickstarted init https://fastapi.tiangolo.com/tutorial/first-steps/

That writes tasks/fastapi-quickstart.yaml with the documentation path filled in, the host allowlist derived from the URL, and a schema line that gives any language-server editor completion on every field below. It validates as written, so you can edit it one field at a time and check your work as you go.

Add the other pages a reader passes through on the way. FastAPI's install instruction is not on the page above, it is on /tutorial/, and a task that omits it measures whether the agent goes looking rather than whether the tutorial works. See why a path rather than a page.

Success checks you can copy¶

The success script is the only thing standing between a report and a model's opinion of itself, which is why there is no way to skip it. It is usually two or three lines. You are asserting what your tutorial already promises, so start by asking what you would type in a terminal to check that the tutorial worked.

# The file the tutorial told the reader to create, in the tutorial's own words
success:
  script: test -f main.py

# The package installed and imports
success:
  script: .venv/bin/python -c "import streamlit"

# The command exists and runs
success:
  script: .venv/bin/mytool --version

# The run printed what the page said it would
success:
  expect_output:
    contains: "200"

# Several of the above, stopping at the first failure
success:
  expect_output:
    contains: "200"
  script: |
    set -e
    .venv/bin/python -c "import httpx"
    test -f main.py  # only because this tutorial names main.py

set -e is what makes a multi-line script stop at the first failing line. It is the only piece of shell syntax you need.

When a check is easier to express in Python than in shell, write it in Python. Nothing prefers bash:

success:
  script: |
    set -e
    .venv/bin/python - <<'PY'
    import csv
    rows = list(csv.DictReader(open("output.csv")))
    assert [r["name"] for r in rows][0] == "Alice Archer", rows
    PY

A weaker check that you trust beats a strict one you cannot debug, and you can tighten it later.

Keep a long check in a file¶

Once a check is more than a few lines, put it beside the task and point at it:

success:
  file: checks/fastapi.sh

Now shellcheck, syntax highlighting and bash -n work on it, and you can read it without counting YAML indentation. The file is read when the task loads, so it never lands in the workspace where the agent could read its own success criteria.

Develop a check without paying for a run¶

--keep-sandbox leaves the workspace in place, and quickstarted check runs your success script against it again. No model, no key, no cost, and the same backend that will judge it for real:

quickstarted run tasks/mine.yaml --agent claude --keep-sandbox
# ... sandbox kept at: /tmp/quickstarted-8ilw9l6v/workspace

quickstarted check tasks/mine.yaml --sandbox /tmp/quickstarted-8ilw9l6v/workspace

That loop takes about a second, so the check can be wrong ten times before it is right. --show prints the script the harness will actually run, helpers included.

Running the script by hand instead judges it in a different environment from the one that will judge it for real: another Python, another PATH, and no container.

Make a failing check say what it saw¶

The exit code decides the verdict. The output is the bug report, and they are separate jobs. test -f output.csv needs no output because the message is obvious from the check itself. A check that starts a server and polls it needs to say what happened, or a failure arrives with an exit code and nothing else.

The failure mode is specific: a check that reports docs_gap with exit code 1 and an empty message gives no way to tell a missing route from a server that never booted. It usually comes from set -e aborting the script at the failing command, before the lines meant to report the problem could run.

serve and wait_http exist to prevent that. They keep the last error rather than swallowing it, print the server log when they give up, and put the reason on the final line, which is the line the console summary shows. "Connection refused" and "HTTP 200 with the wrong body" are different bugs in your documentation, and a bare exit code cannot tell them apart.

For a check with several assertions, one line each is enough. qs_fail prints the reason and stops, and || keeps set -e from aborting before the message prints:

success:
  script: |
    set -e
    test -f pyproject.toml || qs_fail "no pyproject.toml, so uv never created a project"
    test -f uv.lock || qs_fail "no uv.lock, so the project was never locked"
    grep -q httpx pyproject.toml || qs_fail "httpx is not a dependency in pyproject.toml"

That run reports check failed: httpx is not a dependency in pyproject.toml instead of exit code: 1, which is the difference between a page to go and read and a page to go and guess about.

quickstarted validate warns when a check has several assertions and no way to report which one failed, so this is catchable before a run rather than after.

quickstarted run says so when a check stays quiet:

  success check exit code: 1
  note: the check printed nothing, so this failure cannot be diagnosed.
        Have it say what it saw.

Assert the data¶

Check the outcome the documentation promises. Do not check how the agent got there.

# Good: any correct route passes.
success:
  script: |
    set -e
    .venv/bin/python - <<'PY'
    import csv
    rows = list(csv.DictReader(open("output.csv")))
    assert [r["name"] for r in rows][0] == "Alice Archer", rows
    PY

# Bad: passes only if the agent used one particular function.
success:
  script: grep -q "pl.read_csv" example.py

The second version fails a reader who used scan_csv, which the documentation also recommends. You would be measuring your own expectations.

Name nothing the documentation does not name¶

This is the rule the rest of this page is really about, and it is the one that is easiest to break without noticing.

Write the goal from the page, then the check from the goal. If you find yourself writing the goal from the check, stop: you are about to invent an artefact so that the check has something to look at, and the task will then measure your invention.

Here is the gap, for six real quickstarts:

Task	The documentation says	The task demanded
fastapi	"copy that to a file `main.py`"	`app.py`
uv	`uv init` generates a `main.py` that prints a greeting	overwrite it to print `ok`
prisma	ends at `console.log`	write `out.json`
polars	ends at `print(df_csv)`	write `script.py` and `out.csv`
duckdb	`con.table("test").show()`	write `script.py`, `shop.db`, `total.txt`
httpx	a REPL session	write `fetch.py`

A reader following FastAPI's tutorial exactly produces main.py. The task told them to produce app.py. Whatever that measures, it is not the tutorial.

The forcing function was the check: it could only see the filesystem, so goals were written backwards from what it could assert. expect_output exists to remove that pressure. When a quickstart ends at a value on a terminal, assert the value on the terminal.

Do not assert incidental paths¶

A check that asserts .venv/bin/python fails an agent that created venv/ instead, did everything else correctly, and had no way to know which name you wanted. That is a working run recorded as a documentation failure.

If setup creates something the success script depends on, that is fine. The agent is told what setup already ran, so it will not rebuild it. Anything else your script depends on has to come from the documentation, or you are testing telepathy.

Let the harness verify¶

If the goal is a running server, the harness starts it, asks it a question, and stops it. Do not ask the agent to leave a process running, and never take its word for the result:

success:
  serve: .venv/bin/fastapi run app.py --host 127.0.0.1 --port $QS_PORT
  wait_http:
    path: /items/42
    json:
      item_id: 42
  script: test -f app.py

$QS_PORT is a free port picked for this task. The polling, the log capture, the last error and the kill are the harness's job, which is the point: the hand-written version of this block is twenty lines, every task that serves each had their own copy, and the copies were where the if ! idiom got dropped.

When the shape does not fit, the same helpers are available directly:

success:
  script: |
    set -e
    if [ -x .venv/bin/fastapi ]; then
      qs_serve .venv/bin/fastapi run main.py --port "$QS_PORT"
    elif .venv/bin/python -c "import uvicorn" 2>/dev/null; then
      qs_serve .venv/bin/python -m uvicorn main:app --port "$QS_PORT"
    else
      qs_fail "neither the fastapi CLI nor uvicorn is installed"
    fi
    qs_wait_http / --json message="Hello World"

That is close to the real FastAPI check, and it branches because the tutorial documents more than one way to serve. Requiring one of them would measure your expectation rather than the documentation: an agent that installed plain fastapi rather than fastapi[standard] has no uvicorn, and a working application would be recorded as a documentation gap.

Separate documentation hosts from registries¶

docs.allow hosts are readable only through read_docs. The shell cannot reach them. Put pypi.org there and pip install stops working, which shows up as a harness_error rather than a documentation problem.

docs:
  path:
    - https://docs.pola.rs/user-guide/getting-started/
  allow:
    - docs.pola.rs      # documentation
network:
  allow:
    - files.pythonhosted.org   # only if the defaults are not enough

Common registries are allowed by default. quickstarted validate warns when a task declares one as a documentation host.

When a host genuinely serves both, name it under network.allow as well. The installs then work, and the report notes that reads from that host are no longer fully attributable.

Write the goal for a stranger¶

The goal is the only instruction the agent gets. It should describe an outcome in the words a user would use, and avoid naming the API that produces it.

# Good
goal: >
  Using Polars, read people.csv, keep rows where age is over 30, sort by age
  descending, and write the result to out.csv with the same column names.

# Bad: hands over the answer
goal: >
  Call pl.read_csv, then .filter(pl.col("age") > 30), then .sort, then
  .write_csv.

Start in replay¶

Write the replay commands first and run them. If the documented commands do not pass, the task is not ready for a model, and any failure you see afterwards tells you nothing about your documentation.

quickstarted run tasks/mine.yaml --agent replay
quickstarted run tasks/mine.yaml --agent claude

Budget deliberately¶

budgets:
  max_turns: 20            # tool-use rounds
  max_seconds: 420         # wall clock for the agent phase
  max_command_seconds: 300 # one command
  max_output_chars: 20000  # per command, head and tail kept
  max_tokens: 0            # 0 means unlimited

A task that routinely exhausts its budget produces budget_exhausted, which is excluded from pass rates. That is the correct outcome, and it also means a too-small budget quietly removes the task from your results. Check the discarded counts in the summary.

Say the shared parts once¶

A suite of tasks usually wants the same setup and the same budgets, and a repo usually wants the same flags on every invocation. quickstarted.yaml at the root of your project says so once:

run:
  backend: docker
  cache_dir: .cache
tasks:
  setup:
    - python3 -m venv .venv
  budgets:
    max_seconds: 420

The more specific statement wins in both directions. A task file beats tasks:, and a flag you typed beats run:. Lists replace rather than combine, because a config setup and a task setup running one after the other would execute both in an order nobody chose.

run: accepts backend, image, cache_dir, prices, out, junit and workers. It deliberately refuses agent, model, repeat and affordances: a file that quietly changed which model served a task, or how many attempts a rate was computed over, would make two runs incomparable for a reason invisible in the command you typed.

Full field list: task schema.