Working with runs

Follow a run, read what it found, and get into the environments it left behind.

Following a run

haystack verify watch  # block until it finishes; exit 1 if tests failed
haystack verify status  # one screen: verdicts and live environment URLs
haystack verify list  # recent runs

watch is the one to put in CI, because its exit code carries the verdict. --interval sets the poll period and --timeout gives up after a while; by default it waits indefinitely.

Every command below accepts a run selector. latest is usually what you want, and is the default where the argument is optional.

Every command takes --json for scripting.

Reading the results

haystack verify findings  # every problem found, with the readings that moved
haystack verify findings --full  # include the judge's full explanation
haystack verify show latest 3  # one test in full: finding, readings, timeline, log

A test is selected by its number, its id, or a substring of its title, so haystack verify show latest sort works if the title is unambiguous.

Looking at artifacts

Every test leaves artifacts: a screenshot after each step, a video of the replay, and the final DOM. You can open any of them, replay what the test did, jump into the live universe, or compare base against head.

haystack verify open latest 3 --step 4  # the screenshot after replay step 4
haystack verify open latest 3 --live  # the test's live environment
haystack verify open latest 3 --side base  # the base universe instead of head
haystack verify diff latest 3  # side-by-side screenshot diff page
haystack verify artifacts latest  # pull screenshots, DOM and video locally

open takes --kind screenshot|video|dom; it shows the video if there is one, otherwise the last screenshot. diff builds a comparison page and opens it; --out controls where it is built and --no-open leaves it on disk.

Getting inside a failure

haystack verify shell latest 3                 # terminal into that test's live universe
haystack verify reopen latest 3                # fresh compute against the retained state
haystack verify refork latest 3 attempt:fix-2  # checkpoint a durable child
haystack verify reopen archil:…:branch:…       # open that child later

This connects to the sandbox the test ran in, so you can reproduce the failure by hand in the exact environment that produced it.

reopen attaches new compute to state that was retained after the original sandbox went away. refork quiesces that exact environment and checkpoints its current application state into a child that can be reopened or nested again by its returned fork ID. cleanup shows exact retained-state references, provider deletion and cleanup-SLO status.

Where runs are stored

Runs persist under .haystack/verify/ in the repository, so status, findings and show work after the process that started the run has exited. --repo-root points at that directory explicitly if you are running from somewhere else.