Testpulse is static analysis for Playwright, Karate and Cucumber suites. It finds tests that pass but verify nothing, and shows which security behaviours your tests never exercise: no 401 test, no 403 test, no test of another user's data. It runs on your machine and needs no changes to your CI.
✗ [tautological-assertion] line 7: expect(true).toBe(…) can never fail ✗ [tautological-assertion] line 8: expect(null).toBeNull(…) can never fail ⚠ [swallowed-error] line 24: Empty catch block with no explanation — failures are silently ignored ⚠ [swallowed-error] line 30: .catch(() => {}) swallows promise rejections silently → Assert the actual computed value against the expected result, not a constant against itself
ENDPOINT AUTH 401 403 BOLA INPUT GET /bookings 🔒 ✓ · · · POST /bookings 🔒 ✓ ✗ · ✓ GET /bookings/:id 🔒 ✓ ✗ ✗ · DELETE /bookings/:id 🔒 ✓ ✗ ✗ ✗ Security coverage: 45% (5/11 applicable checks) ✗ [missing-object-level-authz-test] DELETE /bookings/:id no test accesses another user's resource (BOLA/IDOR)
Real output from the CLI on the fixtures in the repository (an AI-written test suite for a sample bookings API).
The failure mode of AI-generated tests: green, and meaningless.
Hygiene rules plus coverage measured against the spec.
Do the tests cover what OWASP says goes wrong with LLM apps?
audit-mcp: tool poisoning, invisible characters, over-broad tools| Finding | What happened |
|---|---|
| Real suites are noisy ground | The first version of several rules produced mostly false positives on Rocket.Chat, Grafana, Superset, Supabase and Karate. We narrowed each rule and recorded the before/after. |
| AI tests are not empty, they are shallow | Of 120 AI-written tests only 7 were slop, almost all "asserts status 200, never looks at the data". The classic tautologies were absent. |
| The gaps are security tests | The generated suites covered the happy path and many 401s, but most "another user's resource" and invalid-input cases were missing. |
These are small samples. Rules were tuned on the same files they were measured on, the labels come from an AI reviewer rather than a person, and the security specs were written for the exercise. Treat the numbers as a regression check, not a benchmark. The full history, including a first run that was wrong, is in docs/VALIDATION.md.
# analyse a suite (exit code 1 on critical issues) npx testpulse ./e2e # only what changed in this branch npx testpulse ./e2e --changed-since origin/main # which security tests are missing? npx testpulse security-coverage ./tests \ --openapi openapi.yaml --min-coverage 60 # SARIF for GitHub code scanning npx testpulse ./e2e --format sarif --output testpulse.sarif
# .github/workflows/quality.yml
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- uses: barurb1996/testpulse/packages/github-action@v0.2.0
with:
path: ./e2e
project: my-project
min-score: 60
Node 20 or newer. Quality score drops 10 points per critical issue and 3 per warning. A self-hosted dashboard (history, flaky detection, Slack digests) lives in the same repository.
--push with the address of a server you run yourself.testpulse judge is opt-in and uses your Anthropic API key. Test code is sent only when you run that command, with credentials redacted first.audit-mcp --stdio starts the command you give it on your machine, so only audit servers you would run anyway.