CodeValid Testing · July 15, 2026
We test CodeValid with CodeValid.
CodeValid started as a simple idea. Today it's a distributed system: a frontend, backend services, background workers, schedulers, and infrastructure that scales on demand. That complexity is exactly the problem we built CodeValid to solve — so we pointed it at ourselves. Here's a real verification run on our authentication and onboarding flows, defects and all.
43 auto-generated cases found 2 high-severity auth defects (IDOR + fail-soft masking) in ~18 minutes of wall clock — work we estimate at roughly an engineer-week by hand.
EditorService · Auth & onboarding · 43 cases · tests not modified to force green
Every sprint made CodeValid harder to test by hand
Each part of our stack multiplies the testing surface. A change to authentication touches the frontend, three backend services, and every async worker that acts on a user's behalf. Manually maintaining security-grade coverage across all of it stopped being realistic a long time ago.
Frontend
Editor UI, OAuth redirects, session handling
Backend services
EditorService APIs, workspace & PR endpoints
Distributed compute
Workflow engine orchestrating every run
Async jobs
Test generation, execution, defect detection workers
Schedulers
Recurring verification and re-run triggers
Scalable infra
Elastic workers spun up per customer suite
User authentication with Azure AD — simple to describe, brutal to test
When someone signs in to CodeValid, they're redirected to Azure Active Directory. Azure AD verifies who they are and sends them back with a signed token. From that point, every API call must prove three things: the token is genuine and unexpired, the user is who the token says, and — critically — they can only touch data in their own workspace. Behind that flow sit real external dependencies: Azure AD, GitHub, and our workspace-mapping layer.
Azure AD (identity)
OAuth login, CSRF state, redirect & error paths
Signed tokens (JWT)
Must reject missing, invalid, and expired tokens
GitHub (data)
PR & repo data that must fail loudly when GitHub is down
Workspace mapping
Every request checked against the user's own workspace
Concurrency
Parallel logins must never share or collide OAuth state
Failure paths
Failed identity flows must fail closed, never open
The happy path is one scenario, but security lives in the dozens of unhappy ones — expired tokens, tampered tokens, a user guessing another workspace's ID, Azure AD errors mid-login, GitHub going down, two logins racing each other. Engineers reliably write the happy path; the long tail is where both of our high-severity defects were hiding.
43 test cases from acceptance criteria — zero written by hand
We fed CodeValid the acceptance criteria for user authentication and protected access: Azure AD login, token validation, unauthorized-access restriction, and workspace data protection. It generated a security-heavy suite — negative and rejection paths dominant by design — and ran it end to end, fully automatically.
43
API cases generated
37
Passed
6
Failed
2
High-severity defects
Incomplete workspace authorization (IDOR)
The pull-requests endpoint authenticated the user but skipped the user-to-workspace mapping check. Any logged-in user with a workspace ID could read another workspace's PR metadata.
Why it matters: Classic cross-tenant data exposure — the kind of bug unit tests rarely catch, because every individual component “works.”
Silent empty success on dependency failure
When GitHub was unavailable, the API returned HTTP 200 with an empty pull-request list instead of 503 — masking outages from monitoring and users alike.
Why it matters: Fail-soft bugs inflate time-to-detect in production. CodeValid found it by simulating the dependency outage directly.
The numbers behind the story
Both P0 defects went straight into our remediation backlog — and once the fixes land, the same suite re-runs to prove they're closed. One automated run found what mattered; keeping it costs minutes.
12.8 min
To set up infra, resolve dependencies, and generate a 43-case security-grade auth suite — roughly an engineer-week by hand.
3.6 min
To execute the full suite and detect the bugs — including two high-severity security defects with linked evidence.
2 P0s
High-severity security defects (cross-workspace exposure, silent failure masking) caught before customers could hit them.
What CodeValid saved CodeValid
This isn't a hypothetical customer benefit — it's our own engineering time back. Here's what building this exact security-grade coverage would have cost by hand, next to what our product actually spent.
| Phase | By hand (est.) | CodeValid |
|---|---|---|
| Test infrastructure & environmentHarness, fake identity provider, outage simulation, parallel-request rig | ~12 h | included |
| Dependency setup & mocksAzure AD flows, GitHub API, token fixtures | ~6 h | included |
| Writing 43 security-grade casesNegative paths, authorization, CSRF, concurrency — ~30 min each | ~21 h | 12.8 min |
| Execution, triage & evidenceRun suite, investigate failures, document defects | ~4 h | 3.6 min |
| Total | ~43 hours (≈ 1 engineer-week) | ~18 minutes |
~42 hrs
An engineer-week back on this feature area alone — roughly 99% of the effort returned as automation. Every re-check after a fix costs under 4 minutes instead of hours of manual regression.
Assumptions: hand-built estimates from an experienced engineer building comparable auth coverage · CodeValid times are measured wall clock from our Jul 15, 2026 run (1.8 min analysis + 12.8 min infra/dependencies/test generation + 3.6 min execution & detection).
If it works on us, it works on you.
Our stack has a frontend, distributed backends, async workers, and schedulers — and CodeValid still turned acceptance criteria into evidence-linked defects in one run. Point it at your repo and see what it finds.