Notes on testing the AI way.
How AI exploration, Playwright, and deterministic CI fit together — and the design decisions behind Hover.
- ·vibe-codingai-testingregression-testingopinion
Your coding agent doesn't know what it broke
AI coding agents optimize for the prompt in front of them. They hold no durable model of your app's invariants — so they can't catch the regressions they cause. A bigger model won't fix that. An external contract will.
Read → - ·vibe-codingregression-testingai-testingcihow-to
How to test a vibe-coded app before you ship it
Vibe-coding gets you to a working demo fast — and to silent regressions just as fast. Here's a concrete, five-step way to lock in the behavior that matters before launch, so the next AI edit can't quietly break it.
Read → - ·multi-type-testingvisual-regressionaccessibilityapi-testingplaywrightai-testing
Four kinds of test, none with an AI in the loop
Multi-type testing — E2E, visual, API, accessibility in one suite — is a good idea. Doing it with a model running at test time is not. Hover crystallizes all four as plain Playwright: pixel diffs, rule engines, and recorded flows that pass CI deterministically, with zero AI at run time.
Read → - ·claude-codeagent-loopai-testingregression-testingverify
Your coding agent can't tell if it broke the app. Hover can.
The Claude Code loop is guess, check, repeat. It rises or falls on the check. For code logic the agent has npm test; for the flows a user clicks it has nothing, so it edits with its eyes closed. Hover is the verify step for your app's behavior: a recorded flow, replayed with no model in the loop.
Read → - ·claude-codehooksai-testingregression-testingmcpci
Five Claude Code hooks that keep your regression green
The MCP is what the agent calls; hooks are what fire automatically around the loop. Here are five Claude Code hooks — SessionStart orientation, a Stop gate that blocks until specs pass, a crystallize replay, a guard-first nudge, and a push guard — that make an agent start oriented and finish green. hover-hook install wires the first ones for you.
Read → - ·ai-testingllm-as-judgeciself-healingregression-testingplaywright
Deterministic verdicts and an LLM judge: how Hover Cloud evaluates a test run
A red CI run needs a verdict, not just a red X. Hover Cloud judges in two layers: a deterministic assertion layer (status, record, coverage, determinism) and an LLM judge for the calls with no single right answer. Together they turn your suite into a regression net that guards the system you already shipped — and into the loop where you define the business, not the code.
Read → - ·playwrightvs-codeextensionai-testingmcpself-healing
The Hover VS Code extension: a local-first Playwright cockpit
A Playwright test cockpit inside VS Code: Overview (run matrix), Heal (the CI drift queue), Env (targets + test accounts), and Map (coverage). Local-first, sign-in optional, no chat. Authoring runs in your own coding agent through the Hover MCP.
Read → - ·playwrightmcpmodel-context-protocolclaude-codecursorai-testing
Hover MCP: a Playwright MCP server for your coding agent
Add the Hover MCP to Claude Code or Cursor and /mcp__hover__test_app turns your coding agent into a Playwright test writer. Grounded tools give record == replay, and specs crystallize to plain @playwright/test with no model in CI.
Read → - ·playwrightcidashboardflakinessself-healinggithub-actions
Hover Cloud: a Playwright CI dashboard with a heal queue
Connect a GitHub repo and Hover Cloud watches every CI run: pass rate, a flakiness score, a Heal Queue that triages drift vs bug, a business map, and regression alerts. It runs no browsers; execution stays in your own CI.
Read → - ·playwrightauthenticationstorageState
Log in once and reuse the session across every Playwright test
The naive suite logs in at the top of every spec and pastes credentials into ten files. Hover lifts the login into an auth setup project, wires storageState, and keeps the password in an env var.
Read → - ·playwrightapi-testingcdp
Generate API tests from the traffic your app already makes
Your app fires real requests as you click through it. Hover reads them off the browser over CDP, no proxy or MITM, and crystallizes selected calls into a plain Playwright API spec, session tokens stripped.
Read → - ·playwrightpage-object-modelrefactoring
Auto-extract Page Objects from Playwright specs you already recorded
Five specs share the same opening navigation. Hover finds the shared prefix, generates a Page Object and fixtures, and folds the specs to call it. No model, and it leaves your login alone.
Read → - ·playwrightself-healmaintenance
Self-healing Playwright tests, without a model running in CI
Rename a button and six specs go red. Instead of hand-patching selectors or re-recording from scratch, replay the recorded steps, find the one that drifted, and re-ground only that step.
Read → - ·vibe-codingvibe-testingaiplaywrighttesting
Vibe coding vs. vibe testing: what's the difference, and why it matters
Vibe coding ships features in minutes. A Veracode study found AI-generated code carried a security flaw in 45% of cases. Vibe testing is the answer the term hasn't settled on yet. Here's what both mean and where they're going.
Read → - ·vibe-codingtestingplaywrightai
You vibe-coded the feature. Is it actually tested?
Vibe-coding makes shipping a feature easy and verifying it hard. Clicking through by hand doesn't scale, and you won't hand-write specs for code you didn't hand-write. Vibe-test it instead: describe the flow, get a real Playwright spec.
Read → - ·vibe-codingsecurityidorapi-testing
Vibe-coded apps ship security holes by default
An AI optimizing for 'make it work' writes the happy path, not the authorization check. IDOR and broken access control are the holes your click-through tests never touch. Point your agent at Hover and it finds them from your dev browser.
Read → - ·vibe-codingmaintenanceplaywrightregression
Your vibe-coded app keeps breaking and you can't tell why
Vibe-coding piles up code you never read, so the next prompt quietly breaks a screen three steps away. A suite of plain Playwright specs is the regression net. Hover authors and maintains them from your real flows.
Read → - ·pricingbyokaiopen-source
Stop renting access to the models you already pay for
Most AI tools charge you monthly to resell model access you could buy directly. Hover holds no key and resells no tokens: it runs the Claude or Codex CLI you already have, on your own subscription. Bring your own key, own the tool.
Read → - ·environmentscredentialsciplaywright
I stopped hardcoding test logins: multi-environment accounts with @account
Every E2E suite starts with a login, and that login is where credentials leak into the repo. I got the agent to sign in for me across local, staging, and prod, without a single password landing in a committed spec.
Read → - ·securityplaywrightsecretsregression
Your recorded test might be committing a session cookie
Record a real request, save it as a test, and the cookie, bearer token, and SSN ride along into git. Hover strips credentials out of the request before it writes the spec, and tells you what it dropped.
Read → - ·engineeringcode-reviewaisecurity
We ran an audit pass over our own AI-built code
Most of Hover was written fast, with an agent in the loop. So we stopped and audited the whole thing in one pass: dead code, latent bugs, the works. It found around forty real issues. The interesting ones weren't where you'd guess.
Read → - ·playwrightflaky-testspromise-all
Fix flaky Playwright download, popup, and upload tests with Promise.all
Downloads, OAuth popups, and file uploads flake in Playwright when the event listener registers after the click. Pair them with Promise.all, and let Hover write the pairing for you.
Read → - ·playwrightai-testingci
AI-authored Playwright tests, without an AI in your CI
Most AI testing tools keep a model in the loop at runtime, so every CI run pays for LLM calls. The alternative: let AI author the test once, then ship plain Playwright that runs deterministically.
Read → - ·playwrightcodegencomparison
Playwright codegen vs. AI exploration: when each one wins
Playwright's built-in recorder is free, local, and writes clean code, but it only transcribes clicks. Where codegen is the right tool, and where authoring from intent changes the job.
Read → - ·architecturebyo-clipricing
Bring your own CLI: AI testing with no API key and no token resale
Hover bundles no AI runtime. As an MCP server it rides the coding agent you already run (claude, codex, …), so you reuse the subscription you already pay for. Why that architecture matters.
Read → - ·playwrightmaintenanceselectors
How to keep AI-authored Playwright tests from breaking when your UI changes
Selector rot breaks end-to-end suites. Here is a model-free way to handle it: semantic locators, deliberate re-recording, and treating real flow breaks as the bugs they catch.
Read → - ·cdpchromearchitecture
Why Hover drives your real Chrome over CDP instead of a fresh headless context
An agent that tests your app should see what you see. Why Hover connects to an already-running debug Chrome over the Chrome DevTools Protocol, and why it never touches your primary browser profile.
Read →