AI-assisted security review is faster than manual review, but it introduces a new risk: a confident-sounding output that misses the real issue. The process we use on ManOps, our internal operations product, pairs automated AI-assisted runs with human sign-off at every point where a finding matters. This post describes what that looks like in practice, what it caught, and where the limits are.
What ManOps Is and Why Security Matters
ManOps is the operations tool we built to run our own delivery. It handles projects, tasks, chat, in-app calls, push notifications, and a credential vault - all on a shared API used by the web app, the iOS app, and the Android app. See the ManOps project page for the full scope.
Because the product touches auth tokens, push device tokens, a credential vault, and team communications, a security gap is not hypothetical. A misconfigured Supabase table or a missing auth check on an API route can expose data to any user who knows the URL. We discovered exactly that early in development: two tables with RLS disabled and anon-role access were flagged by Supabase's own database linter as an ERROR (rls_disabled_in_public: anyone with your project URL can read, edit and delete all data in the table). Both were locked down with a migration on 29 September 2026, after the discovery audit found them.
How the AI-Assisted Review Process Works
During the ManOps build, we ran approximately 130 automated security-review runs and 52 sub-agent runs using AI coding tools, working from a structured audit plan. The plan divided the codebase into areas - API authorisation, database access, authentication, calls and push, vault, notifications - and assigned a specific scope to each review pass.
Each automated run followed the same rules. The agent read the code in its assigned area, traced each path from API route to database, identified any gap between what the documentation described and what the code actually did, and produced a finding report with a severity label, a root cause, and a recommended fix. Nothing was changed in the code or database during these runs. Every finding was a read-only report.
Human review happened at the end of each area. A developer read the finding report, verified the cited files and lines, reproduced the issue in a local check where possible, and decided whether to accept, dismiss, or escalate the finding. Fixes were drafted with AI assistance where useful, then written, reviewed, and committed under a developer's direction.
What This Caught That Might Have Been Missed
The most useful finding from this process was a systematic pattern rather than a one-off bug. ManOps uses the Supabase service-role client on the server, which bypasses RLS entirely. That is a legitimate design choice when every route has a proper application-layer auth check. The review read the route handlers and checked how each one authenticates and authorises. It found authorisation gaps, which were then fixed or tracked. A few routes are secured by a signature check or a cron secret instead of a user session (Slack and cron); the audit confirmed which.
A second finding came from the vault. The credential store encrypts secrets server-side before persisting them. The review traced the encryption path and found that production uses a key from the environment, but that a key derived from a source constant is used if the variable is unset. A fix that made this fail closed had been put back by a later deploy change, so the finding was logged with a recommendation to restore fail-closed behaviour. That kind of end-to-end trace is exactly what automated scanning tools miss because they do not understand business logic.
The review also surfaced a known pattern from our own prior work: a fix marked resolved in a commit message whose root cause was not addressed. Push notifications stayed undecryptable for about three weeks while being marked fixed three times, and the vault fail-closed fix was silently reverted by a later deploy change. The review process made these regressions visible by tracing from first principles rather than trusting commit history.
Where AI Review Falls Short
There is a class of issue that code review - AI or human - cannot verify without a running system. Whether an APNs VoIP push actually rings a locked iPhone, whether FCM data messages survive Android battery optimisation on a specific OEM, and whether a production JWT validation rejects a token correctly are all runtime questions. We documented these explicitly as NOT VERIFIED in our audit output and wrote test scripts to run on real devices separately.
The AI-assisted runs also produced some false positives - findings that looked like gaps in the code but were actually intentional design choices documented elsewhere. The human review step is where those were filtered out. Without it, acting on every AI finding would waste development time and, in the worst case, break working functionality.
We also saw that the AI tools were better at tracing data flow than at assessing risk. Identifying that a table had no RLS was straightforward. Deciding whether the application-layer auth check compensated for that, given the specific access patterns, required human judgment about the system as a whole.
The Structure of Our Audit Output
Each area produced a structured document with a scope statement, a feature inventory, lifecycle traces for each feature (create, read, update, delete), and a findings register. Findings were labelled CONFIRMED ISSUE, POSSIBLE RISK, NOT VERIFIED, or WORKING AS EXPECTED. Severity was P0 to P3 following a written definition: P0 for auth bypass or data exposure, P1 for major feature broken, P2 for edge-case failure, P3 for minor issues.
Findings labelled NOT VERIFIED included the specific runtime test needed to verify them: the exact steps, the SQL query to check the result, or the device behaviour to observe. This meant the review left a clear handoff rather than an open question.
The test suite supports the review process rather than replacing it, and the counts below come from our own test run rather than a third party. ManOps has 3,298 vitest tests on the web app and 2,266 Flutter tests on the native apps, verified on 9 October 2026. Those tests catch regressions in known behaviour. The security review catches gaps in the assumptions behind that behaviour - which is a different question.
What the Process Requires From the Team
The AI-assisted runs produced useful output because the audit plan was specific. General prompts like "review this file for security issues" produce general outputs that are harder to act on. The value came from structured scope (one area at a time), a shared evidence standard (cite file and line, not just concept), and a severity definition agreed in advance.
The human review step is non-negotiable. A finding that is accepted without verification can create a false sense of security worse than no review at all. P0 and P1 findings were checked against the code by a developer before anyone acted on them.
If you are considering a similar process for a product you are building or have inherited, the starting question is: do you have a written audit plan that names the areas, defines severity, and sets an evidence standard? Without that, an AI-assisted run produces output you cannot confidently act on. See how a code audit report is structured before you commission one.
If you want an independent review of a product you did not build, or want a second opinion on one you did, get in touch. We can scope what a review covers and what it does not before any work starts.
Common questions
Can AI do a security review of my code?
AI can speed up security review but needs human sign-off. On ManOps the AI-assisted runs traced data flow well but produced some false positives and were weaker at assessing risk, so a developer checked P0 and P1 findings against the code before anyone acted.
What should a security audit plan include?
A written plan that names the areas to review, defines severity in advance, and sets an evidence standard such as citing file and line. Without that, an AI-assisted run produces output you cannot confidently act on. ManOps used a P0 to P3 severity scale agreed before the runs.
Can I get an independent security review of an app I did not build?
Yes. Zovintra offers an independent review of a product you did not build, or a second opinion on one you did, and can scope what a review covers and what it does not before any work starts.
Sources
- Database Advisors: 0013 rls_disabled_in_public - Supabase documentation, accessed 19 Oct 2026
