# Haystack - Complete Documentation > Lightning-fast QA before you ship — verify a change in matched cloud environments before it merges ## Haystack — lightning-fast QA before you ship URL: https://haystack.sh/ # Haystack — lightning-fast QA before you ship Haystack replays your customers' real journeys against every change, each on its own copy of prod, and tells you which ones broke. ## How it works 1. **Blast radius.** Haystack reads the diff and works out every case the change can affect. The cases are your customers' real journeys, learned from your production traffic, narrowed to the ones the change can reach — not scenarios somebody invented for a test. In one worked example a permissions refactor touching two files reaches 3 services and 27 code paths, producing 14 cases worth running, mapped in about five seconds. You do not write the cases. 2. **Fork your infrastructure.** Haystack snapshots your services and the data behind them, from Postgres and Redis to queues and object storage, then forks that snapshot once per case. Fourteen copies of prod, ready in 26 seconds. These are real environments, not mocks, and nothing that happens inside one reaches prod. 3. **Run every case at once.** Each environment is seeded into a different state: a workspace over its seat limit, an invite nobody accepted, a member whose role just changed. Every environment runs twice, with and without your change, so any difference is caused by the change. 4. **See what actually broke.** Failures come with the screen before and after, the network calls that changed, and the events that stopped firing. The environment stays live, so you can open it. 5. **Your agent runs the loop.** The failing case goes back to your coding agent in a form it can act on. It fixes, re-runs, and keeps going until nothing fails. ## Why it catches what tests miss The failures that reach production are usually quiet. In the worked example a frontend stopped sending a member's role and the access API treated a missing role as editor, so anyone with a pending invite could edit and delete anything in the workspace. Nothing threw, every request returned 200, and the test suite passed. ## Using it The CLI is haystack verify. See [haystack verify](/docs) for the command and its flags, [Working with runs](/docs/runs) for reading results, and [Agents](/docs/agents) for the MCP server and the agent loop. ## Getting started Haystack is early and taking on a small number of teams. Sign up for early access at https://haystack.sh/ or book a demo. --- ## Haystack for AI Agents URL: https://haystack.sh/agents # Haystack for AI Agents Haystack verifies a change by running it in matched cloud environments, each seeded into a state the change can land in, and reports which ones broke. This is the `AGENTS.md`-style entry point; it is written for you, not for a human reading the site. ## The loop If you have changed code and want to know whether it works: ```bash haystack verify --json # run, wait, print what broke and why ``` It runs the change in matched environments, waits for the verdict, and prints the findings: which tests broke, the readings that moved, and the judge's reasoning. Exit code 1 means tests failed. Read the findings, fix the code, run it again. If you wrote the change yourself, pass `--intent-transcript ` as well, so the run has evidence of what the change was meant to do. Each run builds two universes per test case, one on the base revision and one on the head revision, from the same seeded starting point. The only difference between them is your change, so any difference in behaviour is attributable to it. ## MCP server ```bash haystack verify mcp # stdio MCP server ``` Tools: `verify_start`, `verify_wait`, `verify_status`, `verify_findings`, `verify_cell`, `verify_list`, `verify_diff`, `verify_universe`, `verify_reopen`, `verify_refork`, `verify_cleanup`, `verify_artifact`. Full descriptions at [/docs/agents](/docs/agents), along with three rules worth reading before you use them: do not poll, take visual verdicts from `verify_diff`, and never report an infrastructure failure as a finding. ## Setting up a repository Follow the setup skill at [/skill.md](/skill.md) -- it has the full, correctly ordered flow: install the CLI, authenticate, then write `.haystack/system.yml` so the verifier knows how to build, boot, seed and log into the system. ## Reading the docs as markdown - Site index of every page: [/llms.txt](/llms.txt) - Full content in one fetch: [/llms-full.txt](/llms-full.txt) - Any page as markdown: append `.md` to its URL, or send `Accept: text/markdown`. - Documentation home: [/docs](/docs) ## Other CLI surfaces `@haystackeditor/cli` on npm (bin name `haystack`) also carries the PR surface for teams using it: `haystack submit` to open a PR through Haystack, `haystack triage owner/repo#123` to read its findings, and `haystack mcp` for a separate stdio server exposing that read surface (`triage_get`, `pr_get`, `pr_status`, `schema`). That surface needs the GitHub App; verification does not. --- ## Skill: Set up Haystack on a repository URL: https://haystack.sh/skill # Skill: Set up Haystack on a repository You are a coding agent. Use this skill when your user asks to add Haystack — verification of a change against copies of production — to a repository. The end state is that `haystack verify` runs the repository's system in matched cloud environments and reports what a change broke. Most of the work is step 4: writing down how the system builds, boots, seeds data and logs in. The verifier decides what to test on its own; it cannot infer how to run your system. ## Prerequisites (check before starting) - The repository has a Git remote (`git remote -v` → `owner/name`) - Node.js 18+ is available - You can run the repository's install and build locally — step 4 requires verifying commands, not guessing them ## Steps ### 1. Install the CLI ```bash npm install -g @haystackeditor/cli ``` If global installs aren't allowed in your environment, every command below also works without installing: ```bash npx -y @haystackeditor/cli@latest ``` **Always pin `@latest` with npx** — a bare `npx @haystackeditor/cli` reuses whatever version is in the npx cache, which can be months old and missing the flags documented here. If a documented flag is rejected, check `haystack --version` against `npm view @haystackeditor/cli version` first. The canonical package is `@haystackeditor/cli` (bin name: `haystack`). The unscoped `haystack` package on npm is an unrelated string-matching library — do not install it. ### 2. Authenticate ```bash haystack login ``` This starts a GitHub device flow: it prints a one-time code and the URL `https://github.com/login/device`. Relay the code to your user and wait for them to complete authorization in their browser. The command blocks until login finishes. This authenticates the CLI to Haystack, which the hosted verifier and the pull request surface both need. A control plane you run yourself takes `--token` (or `HAYSTACK_VERIFY_TOKEN`) instead. Steps 3 to 5 need no authentication at all, so if your user is not at their keyboard, do those first and come back. ### 3. Install the skills ```bash haystack skills install # portable skills in .agents/skills haystack skills install --cli claude # also install Claude Code command shims ``` This writes `/submit`, `/map-your-system` and `/map-cloud-verifier-universe` into the repository's `.agents/skills` directory, where Claude Code, Codex and Cursor all discover them. ### 4. Map the system Run the `/map-your-system` skill you just installed. It writes `.haystack/system.yml`: verified facts about how this repository's system works, so the verifier can run it. That means the package manager and install command, any build or codegen step, each service's start command and how "ready" is observable, how to seed and reset data, and any dev-only login that makes a seeded test user reachable. Two rules the skill enforces, and you should too: - **Facts only, never test selection.** Keys like `scenarios`, `tests`, `skip` or `focus` are rejected. Haystack decides what to test on its own; this file only says how to run things. What it cannot infer is how your system works, and every fact you add converts a risk it identified but could not execute into a check that actually runs. - **Only verified facts.** Run every command before you declare it. A wrong fact is worse than a missing one, and ten verified lines beat sixty guesses. No secret values — environment variable *names* only. Then check it: ```bash haystack system-map validate # --json for machine-readable output ``` ### 5. Commit the generated config ```bash git add .haystack/ .agents/ git commit -m "Add Haystack configuration" git push ``` ### 6. Run a verification From inside the repository, with a change in the working tree: ```bash haystack verify --json # run, wait, print what broke and why ``` It builds two universes per test case, one on the base revision and one on the head revision, from the same seeded starting point, so any difference in behaviour is attributable to the change. Exit code 1 means tests failed. See [/docs](/docs) for the flags and [/docs/agents](/docs/agents) for the agent loop. ## Optional: MCP server The verifier is available over MCP, so an agent can run a verification and read the result without shelling out: ```bash # Claude Code claude mcp add haystack -- haystack verify mcp ``` Or in a JSON MCP config (Cursor, Windsurf, etc.): ```json { "mcpServers": { "haystack": { "command": "haystack", "args": ["verify", "mcp"] } } } ``` The tools, and the three rules for using them, are at [/docs/agents](/docs/agents). ## Optional: the pull request surface Separately from verification, Haystack can review and triage pull requests on GitHub. That path needs the GitHub App, installed on the repository by someone with admin access: ``` https://github.com/apps/haystack-code-reviewer-pr-hook/installations/new ``` Then `haystack setup --repo owner/name --yes` (or `--json` for NDJSON agent mode) scans the repo for conventions, CI signals and review policies and writes `.haystack.json`, `.haystack/pr-rules.yml` and `review-policy.md`. Read a PR's findings with `haystack triage owner/name#`, and open a PR through Haystack with `haystack submit`. None of it is required to run `haystack verify`. ## If you cannot complete a human step One step needs a human: the device-flow login (step 2). It does not block the rest of setup — do steps 3 to 5 first, which need no authentication, then send your user this message and run the verification once they confirm: > I've mapped how this system builds, boots and seeds data into `.haystack/system.yml` and committed it. To run a verification I need one thing from you: run `haystack login` in a terminal (or give me the go-ahead to run it and relay you the one-time code for github.com/login/device). ## Troubleshooting - A documented flag is rejected (e.g. `unknown option '--repo'`) → you're running a stale cached version; re-run via `npx -y @haystackeditor/cli@latest`. - `haystack system-map validate` rejects the file → most often a test-selection key (`scenarios`, `tests`, `skip`, `focus`) or something that looks like a credential. Both are refused by design. - A run reports that a service never became ready → a readiness signal in `.haystack/system.yml` was declared but not verified. Run it yourself from a clean state, and drop anything you cannot make work rather than declaring it with a caveat. - All docs are agent-readable: append `.md` to any docs URL, or fetch [/llms-full.txt](/llms-full.txt). --- ## Haystack Pricing URL: https://haystack.sh/pricing # Haystack Pricing One price per team. No per-seat maths, and no bill that grows because you shipped more. ## Team - $1,000 / month For one product team shipping to prod. Everything included, however much you ship. | | | |---|---| | Changes verified | Unlimited | | Cases per change | As many as it takes, chosen per change | | Environments | Run and torn down for you | | Seats | Everyone on the team | | Support | Shared Slack channel with our engineers | Also included: the CLI and agent loop, a live environment kept for every failure, and runs on every pull request or on demand. ## Enterprise - custom For orgs with more than one team, their own infrastructure, or procurement. Everything in Team, plus unlimited teams, your cloud or ours, an engineer rather than a support queue, annual invoicing, SSO and audit logging, and security review and DPA. [Contact sales](https://haystack.sh/pricing) ## Common questions **Do you touch our prod systems?** No. Haystack forks a replica of your stack from a snapshot, app and database and queues together, and every run happens inside that copy. Writes, deletes and outbound calls stay in the fork. **How is the state realistic, then?** Each case gets its own fork of the same snapshot, seeded into the state that case describes. That is how a run can put a user mid-trial or a queue mid-drain without touching anything real, and why two cases never contaminate each other. --- ## Haystack Changelog URL: https://haystack.sh/changelog # Haystack Changelog Recent updates and improvements to Haystack. ## July 2026 ### Jul 5 - Built for Coding Agents - Agent setup skill at /skill.md — a coding agent can install and configure Haystack on a repo end-to-end - llms.txt and markdown for agent fetches — Claude, ChatGPT, and Perplexity agents get real content instead of an app shell - CLI 0.15.14 — `haystack init` detects your dev server and names configs after the repo - `haystack-cli` and `haystack-review` on npm now resolve to the real CLI ## June 2026 ### Jun 28 - Sharper Analysis, Bigger PRs - Blast-radius verdicts grounded in named callers and callsites - Materiality filter drops nitpick findings before they reach you - Huge PRs with generated files no longer stall analysis; 4× analysis capacity - New home page built around risk-based QA: one risky change, caught, tested, escalated ### Jun 22 - GitHub Access, Hardened - Every GitHub read and write runs on GitHub App installation tokens — OAuth fallbacks removed - Private-repo analysis results authorized by signed grants instead of live GitHub calls ## May 2026 ### May 31 - The CLI Becomes a Platform - `haystack setup --repo owner/name --yes` runs fully non-interactive; `--json` speaks NDJSON for coding agents - MCP server (`haystack mcp`) exposes triage results and PR state to any MCP client - Webhooks, headless tokens, and versioned JSON schemas for every `--json` output ### May 20 - Onboarding That Survives Real Repos - Protected default branch? Setup opens a bootstrap PR through the Git Data API — works with rulesets and CODEOWNERS - Repo scans clone and search your code directly, replacing the GitHub Search API - Extracted rules arrive in plain language with per-card toggles - Existing open PRs are backfilled for analysis when your config PR merges ### May 18 - Review Chat, Persistent and Polished - Chat turns persist and survive reconnects - Syntax-highlighted before/after diffs, themed code blocks and tables - Resizable inbox/chat split; Drafts bucket with Undraft; live review-comment Threads tab ## April 2026 ### Apr 12 - The Inbox & Feed - Haystack now opens to the Inbox — a unified home for every PR that needs your attention - The Feed surfaces one actionable PR at a time — Issues Found, Review Requested, Needs Assignment, and more - Chat with any PR directly from a feed card - Tabbed sections — Good to Merge, Snoozed, Analyzing, Recently Merged ### Apr 8 - Auto-Fixer Agent (Alpha) - Haystack can attempt to auto-fix straightforward issues in your PRs with a sandbox agent - Smart dismissals remember what you've already reviewed ## February 2026 ### Feb 12 - Cloud Sandbox and In-Review Editing - Pro users can now preview frontend and backend changes in a cloud sandbox - Edit code directly in the review and see live results ### Feb 11 - Documentation Page - Added a /docs page with step-by-step verification flow setup guide - Learn how to set up automated PR verification with /prepare-haystack and /setup-haystack ### Feb 5 - Mobile Support - Haystack now works on mobile devices with a fully responsive design - Floating chat input and swipeable Code/Browser/Terminal tabs for easy navigation - Long-press on any symbol for go-to-definition on touch devices ### Feb 1 - Slash Commands & Dashboard Improvements - New slash commands: /approve, /merge, /comment, and /request-changes to take action directly from chat - Quick approve and merge buttons added to the dashboard for faster PR workflows - Stacked PR navigator for Graphite and ghstack-style PR stacks - Collapsible dashboard sections and ability to dismiss PRs you don't need to review ## January 2026 ### Jan 31 - React File Chunking Fix - Fixed chunking of React files so components are properly grouped in PR sections ### Jan 29 - Agent-First UI and Bug Detection - Redesigned the UI to prioritize Haystack Agent — the agent is now front and center in your review workflow - Made chat a lot faster, especially suggestion pills which now load and respond near-instantly - Haystack now automatically detects possible bugs and issues in your PR so you can catch problems before they ship ### Jan 23 - Edit Code Right in Your Review - Make changes and run them in one click — no local setup required - Pro subscribers get cloud sandboxes with full frontend/backend preview ### Jan 15 - Haystack Agent - Chat with an AI that understands your entire PR context - Ask questions about the code, get explanations, and request changes ## Links - [Home](/) - Learn more about Haystack - [Pricing](/pricing) - Plans and pricing --- ## Install Haystack GitHub App URL: https://haystack.sh/github-app # Install Haystack GitHub App Get AI-powered code review on your repositories in minutes. ## Installation Steps 1. **Click Install**: Visit [github.com/apps/haystack-code-reviewer-pr-hook](https://github.com/apps/haystack-code-reviewer-pr-hook/installations/new) 2. **Select Repositories**: Choose which repos to enable Haystack on 3. **Authorize**: Grant read access to your code for analysis 4. **Start Reviewing**: Open any PR and visit haystackeditor.com/review/{owner}/{repo}/{pr} ## Permissions Required Haystack requests the following permissions: - **Read access to code**: To analyze your PR diffs - **Read access to pull requests**: To fetch PR metadata - **Read access to issues**: To understand linked issues We never write to your repository without explicit action from you. ## Pricing Haystack is priced per team. [Contact sales](/enterprise-contact) for pricing and to get your organization set up. Enterprise adds a self-hosted option, SSO, and dedicated support. [View Pricing Details](/pricing) ## Security - Your code is analyzed in real-time and not stored permanently - We use encrypted connections for all data transfer - SOC 2 compliance in progress ## Support Having trouble? Contact us at support@haystackeditor.com ## Links - [Home](/) - Learn more about Haystack - [Pricing](/pricing) - Plans and pricing - [haystack verify](/docs) - Verify a change against copies of prod, no GitHub App required --- ## Haystack Enterprise URL: https://haystack.sh/enterprise-contact # Haystack Enterprise For teams that need advanced security, compliance, and support. ## Enterprise Features - **Self-hosted deployment**: Run Haystack in your own infrastructure - **SSO/SAML**: Integrate with your identity provider - **SOC 2 compliance**: Enterprise-grade security - **Dedicated support**: Private Slack channel with our team - **Custom integrations**: Connect to your existing tools - **Volume pricing**: Discounts for large teams ## Contact Sales Fill out the form at [haystackeditor.com/enterprise-contact](https://haystack.sh/enterprise-contact) or email enterprise@haystackeditor.com. ## What to Include - Company name and size - Number of developers - Key requirements (self-hosting, SSO, compliance) - Timeline for evaluation ## Links - [Home](/) - Learn more about Haystack - [Pricing](/pricing) - Standard plans --- ## haystack verify URL: https://haystack.sh/docs # haystack verify Verify a change by running it in matched cloud environments, seeded into the states it can land in, and see which ones broke. ## How a run works A run builds two universes per test case, one on the base revision and one on the head revision, from the same seeded starting point. The only difference between them is the change under review, so any difference in behaviour is attributable to it. Results land on a page you can open, and every verdict is backed by the output the run captured. ## Starting a run ```bash haystack verify # what you are changing right now haystack verify --base main --head HEAD # a specific range of commits haystack verify posthog/posthog-js#3191 # a pull request ``` Run it from inside your repository. With no arguments it verifies what you are changing. `--base` and `--head` pin the exact commits to compare, and a pull request reference verifies that PR instead. The run starts, prints its id and results URL, and opens the results page in your browser. Pass `--no-open` to skip that. | Option | Purpose | |---|---| | --repo | Repository to analyze, when you are not running from inside it | | --base / --head | Revisions to compare for a repository run | | --intent-transcript | Coding-agent transcript, used as evidence of what the change intended | | --server | Server URL. Default `http://127.0.0.1:3000`, or `HAYSTACK_VERIFY_SERVER` | | --token | Bearer token when the server requires auth, or `HAYSTACK_VERIFY_TOKEN` | | --no-open | Do not open the results page | | --json | Wait for the verdict and print the summary and findings as JSON | ## Next - [Working with runs](/docs/runs) - [Agents](/docs/agents) --- ## Working with runs URL: https://haystack.sh/docs/runs # Working with runs Follow a run, read what it found, and get into the environments it left behind. ## Following a run ```bash haystack verify watch # block until it finishes; exit 1 if tests failed haystack verify status # one screen: verdicts and live environment URLs haystack verify list # recent runs ``` `watch` is the one to put in CI, because its exit code carries the verdict. `--interval` sets the poll period and `--timeout` gives up after a while; by default it waits indefinitely. Every command below accepts a run selector. `latest` is usually what you want, and is the default where the argument is optional. Every command takes `--json` for scripting. ## Reading the results ```bash haystack verify findings # every problem found, with the readings that moved haystack verify findings --full # include the judge's full explanation haystack verify show latest 3 # one test in full: finding, readings, timeline, log ``` A test is selected by its number, its id, or a substring of its title, so `haystack verify show latest sort` works if the title is unambiguous. ## Looking at artifacts Every test leaves artifacts: a screenshot after each step, a video of the replay, and the final DOM. You can open any of them, replay what the test did, jump into the live universe, or compare base against head. ```bash haystack verify open latest 3 --step 4 # the screenshot after replay step 4 haystack verify open latest 3 --live # the test's live environment haystack verify open latest 3 --side base # the base universe instead of head haystack verify diff latest 3 # side-by-side screenshot diff page haystack verify artifacts latest # pull screenshots, DOM and video locally ``` `open` takes `--kind screenshot|video|dom`; it shows the video if there is one, otherwise the last screenshot. `diff` builds a comparison page and opens it; `--out` controls where it is built and `--no-open` leaves it on disk. ## Getting inside a failure ```bash haystack verify shell latest 3 # terminal into that test's live universe haystack verify reopen latest 3 # fresh compute against the retained state haystack verify refork latest 3 attempt:fix-2 # checkpoint a durable child haystack verify reopen archil:…:branch:… # open that child later ``` This connects to the sandbox the test ran in, so you can reproduce the failure by hand in the exact environment that produced it. `reopen` attaches new compute to state that was retained after the original sandbox went away. `refork` quiesces that exact environment and checkpoints its current application state into a child that can be reopened or nested again by its returned fork ID. `cleanup` shows exact retained-state references, provider deletion and cleanup-SLO status. ## Where runs are stored Runs persist under `.haystack/verify/` in the repository, so `status`, `findings` and `show` work after the process that started the run has exited. `--repo-root` points at that directory explicitly if you are running from somewhere else. --- ## Agents URL: https://haystack.sh/docs/agents # Agents Drive the verifier from a coding agent, over MCP or from the shell. ## MCP server A coding agent can drive the verifier without shelling out: ```bash haystack verify mcp # stdio MCP server for agents ``` `mcp` exposes the verifier as tools, so an agent can run a verification and read the result without shelling out. | Tool | What it does | |---|---| | verify_start | Start a run. Returns `run_id` immediately. | | verify_wait | Block until the run is terminal, then return the summary and outcome. | | verify_status | Per-test verdicts, counts, findings digest, live environment URLs. | | verify_findings | Just the problems: failing tests, readings that moved, the judge's reasoning. | | verify_cell | One test in full: finding, readings, request timeline, SDK logs, artifacts. | | verify_list | Runs on disk, newest first. | | verify_diff | Build the base vs head screenshot diff; returns per-step byte-identical verdicts. | | verify_universe | Get or open a live universe for one test; reuses a running sandbox or forks a fresh one. | | verify_reopen | Attach fresh compute to retained state, by fork id or run/test/side. Returns the sandbox id. | | verify_refork | Checkpoint a reopened universe into a durable child under a stable `child_key`. | | verify_cleanup | Retained-state cleanup for one universe: shared references, deletion attempts, SLO verdict. | | verify_artifact | Fetch one artifact (png, html, webm, json) locally. | Three rules for agents using these tools: **Do not poll.** Start a run, then call `verify_wait`. Tests can legitimately sit in provisioning for minutes, and an agent polling `verify_status` in a loop will read that as stuck when it is not. **Take visual verdicts from `verify_diff`.** It reports, per step, whether the screenshots are byte-identical. Use that to answer whether the pixels changed, and give the diff page to a human to look at. **Infrastructure failures are not findings.** Anything that could not run is reported as a verifier defect, never as a finding. A broken sandbox does not show up as a bug in your code. ## The loop An agent working in the shell needs one command: ```bash haystack verify --json # run, wait, print what broke and why ``` It runs the change, waits for the verdict, and prints the findings: which tests broke, the readings that moved, and the judge's reasoning. The exit code is 1 if tests failed. The agent reads the findings, fixes the code, and runs the same command again. If the change came from an agent, pass `--intent-transcript` as well, so the run has evidence of what the change intended.