The deploy webhook
In Settings, open Deploys and press Create webhook URL. Add that URL to your deploy provider: on Railway and Vercel it goes under Settings, Webhooks, and any provider that can send a POST when it deploys will work. The URL is shown once.
From then on, every deploy runs the free checks against the new version. When the webhook names a pull request’s preview, on your product’s domain or a known preview host, the preview is what gets checked. The same pane has two more options:
- Pull requests: connect the repository with a fine-grained token that has one permission, commit statuses (read and write). Each pull request checked against its preview then gets a commit status from nth Labs: pass or fail, with the reason.
- Slack alerts: only when something changes, or on every deploy. Slack alerts come with Team and Agency.
The GitHub Action
The Action lives in the nth Labs repository, which is private for now, so the step below works once you have access to it: write to hello@nthlabs.dev. The deploy webhook above needs no repository and works on every plan.
Add one step after your deploy step. It needs no API key:
- uses: dannykwonyj/nthlabs@v1
with:
url: ${{ steps.deploy.outputs.preview-url }}A real browser checks the deployed address for accessibility, speed, dead links and console errors, and the job fails when the change brings in a ship-blocker, such as a server key served to the browser or a button that does nothing. The verdict and the findings appear on the job’s summary page.
| Input | Default | What it does |
|---|---|---|
url | required | The deployed address to check. If nth Labs cannot get into the preview, the job fails rather than passing. |
fail-on | new-critical | When to fail: new-critical (this change brought in a ship-blocker), any-critical (any ship-blocker is open), readiness:N (the launch score is under N out of 100, which only a check with a key measures) or never. |
check | vitals | vitals is the free layer. changes compares against baselines carried between runs. pr is the pull-request lane, and needs your repo checked out with full history (fetch-depth: 0) and a profile. |
profile | none | A product profile committed to your repo. With one, the check covers the pages it names; without one, the front page only. |
bypass-token | none | Vercel’s Protection Bypass for Automation secret, from a repository secret. |
basic-auth | none | user:password for a preview behind HTTP Basic auth, such as Netlify password protection. |
test-email, test-password | none | The test account for pages behind your sign-in, from repository secrets. |
base | the pull request’s base branch | The branch a pull request merges into, for check: pr. |
node-version | 20 | The Node version to run on. 20 or newer. |
Three outputs come back: pass (true or false), findings (how many are open after the run, empty when nothing was measured) and report (the path of the Markdown summary).
Set fail-on: never while you evaluate it, and the job reports without failing. An address that does not answer, or asks for a password, still fails the job under every policy, never included, because a check that could not see the page cannot pass.
Previews behind a password
You do not have to switch deployment protection off. On Vercel, generate a Protection Bypass for Automation secret (Project Settings, Deployment Protection) and pass it as bypass-token. For Netlify password protection, or any preview behind HTTP Basic auth, pass basic-auth.
The secret goes only to the preview’s own address, never to a font, script or API on another domain, and never over plain http except to localhost. The job log masks it, and the report, the findings and anything else the run prints leave it out.
Pages behind your sign-in
To check more than the front page, commit a copy of your product profile with its password line deleted, and pass the test account from two repository secrets:
- uses: dannykwonyj/nthlabs@v1
with:
url: ${{ steps.deploy.outputs.preview-url }}
profile: .nthlabs/product.json
test-email: ${{ secrets.LABS_TEST_EMAIL }}
test-password: ${{ secrets.LABS_TEST_PASSWORD }}They are used for that run only, and never written to the profile, the report or the log. Use a test account, never a real person’s.
Failing only on new problems
The default, new-critical, fails only on ship-blockers this change introduced, so problems that were already there do not fail every build. A GitHub runner keeps nothing between runs, though, so on a fresh one every ship-blocker counts as new until you carry nth Labs’ ledger (its state/ folder) from run to run. The workflow in ci/labs-pr.yml, in the repository, shows how.
From the command line
node cli.cjs ci --target https://preview-123.example.devThe gate checks the setup first, runs its checks and every regression check you have kept, and writes out/ci-summary.md (ready to paste into a pull request) and out/ci-summary.json. The exit code is 1 when the gate fails.
By default ci runs Site review and the launch score, which call a model and need a key. For a free gate, run the free layer only. Your kept regression checks still run.
node cli.cjs ci --ci-modules vitals --target "$PREVIEW_URL"--fail-on takes the same policies as the Action, and --soft reports without ever exiting non-zero. A few things fail the gate under every policy: a setup that no longer matches the product, a check that crashed, and a bug you fixed coming back.
For pull requests, node cli.cjs pr --base main --target <preview> compares only the pages the diff can reach against the base branch’s baselines. It needs no key.
Reading the result
The Markdown summary is written to be pasted into a pull request as it is: the verdict, the question the gate asked, what it did not check, the four Readiness answers as they stand, the new ship-blockers and the top findings with their fix prompts. The JSON carries the same answer for a script: whether it passed, the counts, and the new ship-blockers with their ids.