Agents
Drive the verifier from a coding agent, over MCP or from the shell.
MCP server
A coding agent can drive the verifier without shelling out:
haystack verify mcp # stdio MCP server for agents
mcp exposes the verifier as tools, so an agent can run a verification and read the result without shelling out.
| Tool | What it does |
|---|---|
| verify_start | Start a run. Returns run_id immediately. |
| verify_wait | Block until the run is terminal, then return the summary and outcome. |
| verify_status | Per-test verdicts, counts, findings digest, live environment URLs. |
| verify_findings | Just the problems: failing tests, readings that moved, the judge's reasoning. |
| verify_cell | One test in full: finding, readings, request timeline, SDK logs, artifacts. |
| verify_list | Runs on disk, newest first. |
| verify_diff | Build the base vs head screenshot diff; returns per-step byte-identical verdicts. |
| verify_universe | Get or open a live universe for one test; reuses a running sandbox or forks a fresh one. |
| verify_reopen | Attach fresh compute to retained state, by fork id or run/test/side. Returns the sandbox id. |
| verify_refork | Checkpoint a reopened universe into a durable child under a stable child_key. |
| verify_cleanup | Retained-state cleanup for one universe: shared references, deletion attempts, SLO verdict. |
| verify_artifact | Fetch one artifact (png, html, webm, json) locally. |
Three rules for agents using these tools:
Do not poll. Start a run, then call verify_wait. Tests can legitimately sit in provisioning for minutes, and an agent polling verify_status in a loop will read that as stuck when it is not.
Take visual verdicts from verify_diff. It reports, per step, whether the screenshots are byte-identical. Use that to answer whether the pixels changed, and give the diff page to a human to look at.
Infrastructure failures are not findings. Anything that could not run is reported as a verifier defect, never as a finding. A broken sandbox does not show up as a bug in your code.
The loop
An agent working in the shell needs one command:
haystack verify --json # run, wait, print what broke and why
It runs the change, waits for the verdict, and prints the findings: which tests broke, the readings that moved, and the judge's reasoning. The exit code is 1 if tests failed. The agent reads the findings, fixes the code, and runs the same command again.
If the change came from an agent, pass --intent-transcript as well, so the run has evidence of what the change intended.