A test suite of 5,564 automated tests covering an internal ops app's API routes, push notifications and Flutter UI screens will catch almost every regression in the logic you have already modelled. It will not catch whether a real Android device actually rings when an FCM data message arrives, whether CallKit presents a lock-screen answer button on an iPhone, or how a specific phone manufacturer's battery optimiser kills a background app mid-call. The number is meaningful. It is not a safety certificate.
The Numbers and What They Come From
ManOps - our own internal ops platform for project management, team chat and voice calls - reached 3,298 Vitest tests across 246 files and 2,266 Flutter tests across 146 test files as of 9 October 2026, verified in a clean worktree run. Of those, 3,296 web tests pass (2 skipped, none failing) and 2,261 native tests pass (5 skipped, none failing). The test suite grew from zero to 5,564 between the end of September and early October 2026, after the app had been running in production for weeks with no automated tests at all.
The web tests cover specific subsystems: ring-timeout logic (8 tests), call-push routing (18 tests), device push registration (8 tests), push API routes (17 tests), do-not-disturb status propagation (5 tests) and a project-switcher UI component (26 tests). The Flutter side covers push message parsing, the ring-once deduplication ledger, call accept/decline/cancel flows, security properties (a dedicated Flutter test file) and accessibility semantics. Every subsystem has a test file with named cases for the specific behaviour it is expected to enforce.
Together these tests enforce a tight contract. If a code change breaks the 60-second ring-window constant, the CI pipeline fails before anyone notices on a device. If a push payload changes shape, the parsing tests catch it. If a new status variant bypasses the do-not-disturb silence logic, the status tests fail. That is exactly what automated tests are supposed to do.
What No Automated Test Can Verify
The state document for ManOps includes a column titled 'Not covered by that verification', and it reveals the precise shape of the gap. The calls and push backend tests confirm that the server correctly identifies which device tokens to target, calculates ring windows and routes push payloads. They do not confirm that a real FCM data message travels from the Vercel serverless function to a physical Android phone.
The Flutter tests confirm that the app handles push parsing and ring deduplication correctly. They do not confirm that the push arrives through Apple Push Notification service on a real iPhone, that CallKit presents a lock-screen ringing UI, that one phone's answer stops the other phone's ring, or that the user's OEM battery-saving policy does not terminate the background handler before the 60-second window closes. The accessibility tests confirm that every screen passes programmatic label and semantics checks. They do not confirm that TalkBack or VoiceOver on real hardware reads elements in the right order.
This is not a criticism of the test suite. These are the inherent limits of any automated approach. A test running in a CI container cannot impersonate FCM's delivery queue, Apple's push infrastructure or an OEM kernel scheduler. The gap is documented clearly in the project state. A real-device test script exists in the repo and is ready to run. It had not been executed on real phones as of 9 October 2026.
What Zero Tests Looked Like Before This
The ManOps state documents record what the zero-test period looked like, and it is instructive. Web push encryption was broken for three weeks after launch. Every push message had been undecryptable since the feature shipped because the HKDF key derivation used Expand where Expand-only is correct. The bug was 'fixed' three times based on an AI assistant concluding push 'works' before delivery was actually confirmed on a device. The real fix landed in a specific commit when an independent decrypt test, run in a scratchpad outside the repo, proved the payload was unreadable. That test was never committed.
The vault master key fix was silently reverted by a later deployment. The call restart loop took five commits to resolve, and the push bug was marked fixed three times before delivery was actually confirmed. A stored cross-site scripting risk in the email reader stayed open while automated security reviewers dismissed reachable issues as pre-existing because they sat outside the diff. None of these failures involved bad developers. They were caused by the absence of a machine-enforced contract. A test that asserts 'this push payload decrypts correctly' would have caught the crypto bug on the first commit. A test that asserts 'the vault key fails closed if missing' would have caught the revert.
The pattern shows up in other fast-shipped apps as well, not just internal products. The mechanism is always the same: behaviour that is never encoded in a test can be broken by any future change, and no alarm sounds.
Three Lessons for Tech Leads
- Test what breaks silently. Crypto, push delivery routing and configuration-dependent logic are bad candidates for 'we checked it in production'. They are good candidates for unit tests with known inputs and expected outputs. The ManOps push crypto bug was only found by a test that lived in a scratchpad; the lesson is to put that kind of test in the repository, where it runs on every change.
- Label what the tests do not reach. The ManOps state document includes a 'Not covered' column alongside each verified item. That discipline matters more than the total count. A team that knows 'we have tested the logic but not the real device' makes better deployment decisions than one that sees 5,564 passing and assumes everything is covered.
- Write the real-device test script before you need it. The script exists in the ManOps repo and was written during the pre-launch audit phase. The value is not just in running it once; it is in having a repeatable procedure any team member can follow when the app is updated or runs on a new OS version.
What a Test Count Tells You and What It Does Not
5,557 passing tests means the team has encoded 5,557 specific hypotheses about how the app should behave. That is a real, auditable asset. It tells you the regression surface is covered. It tells you the team is thinking systematically about failure cases. It does not tell you the app works end to end on real hardware under real network conditions with real push infrastructure.
For an internal ops app used by a small team, that is the right trade-off. The cost of a CI-only verification pipeline is much lower than maintaining a device lab. The acceptable risk of an untested real-device call path is much lower for a five-person internal team than for a consumer app with tens of thousands of users. The right question is not 'how many tests do you have?' but 'what have you tested, what have you deliberately not tested, and do the people making launch decisions know the difference?' Our ManOps project state answers all three. That transparency is more valuable than the headline number.
If you want to map the same kind of gap in your own test suite before a launch or handover, a structured technical review can produce the same kind of 'tested / not tested / not testable in CI' breakdown for your specific stack. The ManOps case study has more detail on how the ops platform is structured.
Common questions
Does a high automated test count mean my app works?
No. A test suite covers the logic you have modelled, not real-world conditions. ManOps has 5,564 automated tests, yet they cannot confirm that a real FCM message reaches an Android phone or that CallKit shows a lock-screen answer button. The better question is what you tested and what you deliberately did not.
What can automated tests not verify in a mobile app?
Anything that depends on real devices and infrastructure: push delivery through FCM or Apple's push service, CallKit lock-screen behaviour, an OEM battery optimiser killing a background handler, and how TalkBack or VoiceOver reads elements on real hardware. Those need a real-device test script.
Can someone map what my test suite does and does not cover?
A structured technical review can produce a tested, not tested and not testable in CI breakdown for your specific stack before a launch or handover. Zovintra offers this kind of review, and the ManOps case study shows how its own platform is structured.
Sources
- ManOps project state document v1.4 (internal), 9 Oct 2026
- ManOps native test plan (internal), 9 Oct 2026
- ManOps calls and push requirements (internal), 9 Oct 2026
