An API test suite should fail on the vulnerable build

Oct 12, 2026 · api-qa-suite on GitHub

A lot of API test suites check the happy path and a few 4xx codes. They miss the bugs that matter: reading someone else's object by changing an ID, forged tokens, leaked password hashes, mass assignment, missing rate limits. How do you know your suite would catch those? I built it against an API with the bugs planted in it.

Setup

api/app.py is a small Tasks API. With APP_MODE=vulnerable it ships 6 seeded defects that mirror OWASP API Security Top 10 findings; APP_MODE=fixed corrects them. The suite has three layers:

• Functional: 13 tests for CRUD, auth, validation boundaries and Unicode.
• Security: 9 tests mapped to the OWASP API Top 10 (2023).
• Contract: schemathesis property-based fuzzing from the OpenAPI schema.

CI runs a matrix: the fixed build must pass, and the vulnerable build must fail exactly the 9 security tests.

Results

• Vulnerable build: 9 failed, 19 passed. All 6 seeded defects caught.
• Fixed build: 28 passed, stable over 3 repeated runs.
• Schemathesis found 3 bugs I hadn't seeded, including lax boolean coercion and a wrong Allow header on 405 responses.

The vulnerable-mode check is the part I'd keep in any project. A green suite only tells you something if you've seen it go red for the right reasons.

← All posts