Test quality analyser for QA teams

Your tests are green.
Do they check anything?

Testpulse is static analysis for Playwright, Karate and Cucumber suites. It finds tests that pass but verify nothing, and shows which security behaviours your tests never exercise: no 401 test, no 403 test, no test of another user's data. It runs on your machine and needs no changes to your CI.

Get started View on GitHub

Tests that cannot fail

✗ [tautological-assertion] line 7: expect(true).toBe(…) can never fail
✗ [tautological-assertion] line 8: expect(null).toBeNull(…) can never fail
⚠ [swallowed-error] line 24: Empty catch block with no
  explanation — failures are silently ignored
⚠ [swallowed-error] line 30: .catch(() => {}) swallows
  promise rejections silently
→ Assert the actual computed value against the expected
  result, not a constant against itself

Security coverage from your OpenAPI spec

ENDPOINT              AUTH  401  403  BOLA  INPUT
GET /bookings          🔒   ✓   ·    ·     ·
POST /bookings         🔒   ✓   ✗    ·     ✓
GET /bookings/:id      🔒   ✓   ✗   ✗     ·
DELETE /bookings/:id   🔒   ✓   ✗   ✗    ✗

Security coverage: 45% (5/11 applicable checks)
✗ [missing-object-level-authz-test] DELETE /bookings/:id
  no test accesses another user's resource (BOLA/IDOR)

Real output from the CLI on the fixtures in the repository (an AI-written test suite for a sample bookings API).

✓

Tests that pass but verify nothing

The failure mode of AI-generated tests: green, and meaningless.

  • missing, tautological and weak assertions
  • swallowed errors, commented-out assertions
  • 2xx status checked, response never inspected
  • duplicated scenarios, flaky waits
🔒

Security in and around your tests

Hygiene rules plus coverage measured against the spec.

  • hardcoded secrets, disabled TLS, production URLs, real-looking PII
  • missing 401, 403, BOLA/IDOR and input-validation tests per endpoint
  • works for Karate and Playwright API tests
  • SARIF output for GitHub code scanning
★

Tests of AI apps and MCP servers

Do the tests cover what OWASP says goes wrong with LLM apps?

  • prompt injection, system-prompt leak, unsafe output handling, tool permissions
  • audit-mcp: tool poisoning, invisible characters, over-broad tools
  • optional LLM review of a test's quality, with your own API key

What we measured, and how far to trust it

602
test files from 5 public repositories
120
AI-written API tests, labelled by an independent reviewer
25–67%
security coverage of five AI-written suites
FindingWhat happened
Real suites are noisy groundThe first version of several rules produced mostly false positives on Rocket.Chat, Grafana, Superset, Supabase and Karate. We narrowed each rule and recorded the before/after.
AI tests are not empty, they are shallowOf 120 AI-written tests only 7 were slop, almost all "asserts status 200, never looks at the data". The classic tautologies were absent.
The gaps are security testsThe generated suites covered the happy path and many 401s, but most "another user's resource" and invalid-input cases were missing.

These are small samples. Rules were tuned on the same files they were measured on, the labels come from an AI reviewer rather than a person, and the security specs were written for the exercise. Treat the numbers as a regression check, not a benchmark. The full history, including a first run that was wrong, is in docs/VALIDATION.md.

Run it

# analyse a suite (exit code 1 on critical issues)
npx testpulse ./e2e

# only what changed in this branch
npx testpulse ./e2e --changed-since origin/main

# which security tests are missing?
npx testpulse security-coverage ./tests \
  --openapi openapi.yaml --min-coverage 60

# SARIF for GitHub code scanning
npx testpulse ./e2e --format sarif --output testpulse.sarif

In CI

# .github/workflows/quality.yml
- uses: actions/checkout@v4
  with: { fetch-depth: 0 }
- uses: barurb1996/testpulse/packages/github-action@v0.2.0
  with:
    path: ./e2e
    project: my-project
    min-score: 60

Node 20 or newer. Quality score drops 10 points per critical issue and 3 per warning. A self-hosted dashboard (history, flaky detection, Slack digests) lives in the same repository.

Privacy