100 vibe apps, one scanner: cross-platform sweep, real leaks, honest gaps

82

Lovable

79

Bolt

78

Deno

78

Firebase

78

Netlify

74

Fly.io

64

Railway

54

Replit

We spent a day running Sekrd against 100 real apps deployed on 12 different vibe-coding platforms. Not our handpicked test corpus. Not synthetic fixtures. Fresh URLs pulled straight from the Wayback Machine Certificate Transparency firehose — a random sample of *.lovable.app, *.bolt.host, *.replit.app, *.vercel.app, *.netlify.app, *.pages.dev, *.web.app, *.up.railway.app, *.onrender.com, *.fly.dev, *.glitch.me, *.deno.dev. Plus 10 worst-scoring reference URLs from our prior-day sweeps as a non-regression check.

Every scan was a deep scan — all 15 providers firing, no integration credentials passed, which is the view an anonymous attacker would get. Rate-limited 2 submits/minute per key, total run time ~90 minutes.

The ugly honest numbers

The 12 failures are a signal by themselves. Render had 100% fail rate (10 of 10 URLs dead). Vercel had 100% fresh-URL fail rate. Railway 90%. These platforms host huge volumes of preview deploys that churn fast — Wayback archived them a few months ago, by today they're gone. Lovable had 20% fail rate — apps deployed there stick around.

Platform fingerprints are very real

Read the per-platform score table carefully:

Platform	Fresh N	Avg	Range	Notes
Lovable	8	82.0	82/82	All 8 identical
Bolt	10	79.0	76/90	9/10 at 78
Deno Deploy	6	78.0	78/78	All identical
Firebase	4	78.0	78/78	All identical
Fly.io	8	74.2	62/80	More user-controlled
Cloudflare Pages	4	58.5	0/82	One scored 0 (13 webhook bugs)
Replit	6	54.3	0/78	Real Stripe-webhook leak

Read it twice. Nine of ten bolt apps score 78. All eight lovable apps score 82. All six deno apps score 78.

This isn't the scanner being lazy. It's the scanner correctly observing that apps deployed on a given vibe platform inherit that platform's default response headers — same Strict-Transport-Security, same Permissions-Policy, same X-Frame-Options, same server-version disclosure. The app-level code differs wildly (todo apps, chatbots, landing pages, crypto trackers). The platform-level security posture is identical.

If you want to know how secure your freshly-generated Lovable app is before writing a single line of custom code, the answer is: 82 out of 100. Not because Lovable is great at security, but because they've made specific header tradeoffs. That's the floor you start at. Anything you add can only move from there.

The five real leaks we found

Most findings were the expected medium-severity "missing X-Frame-Options" / "no privacy policy" kind. But five sites had real critical issues a scanner should flag. All five were live when we hit them.

1. Stripe live secret key in a Bolt demo

A sk_live_ key in client JavaScript is a full-compromise situation: anyone who opens DevTools can copy it and make Stripe API calls as you — refund themselves, create charges, read payment history. Sample/demo that got deployed publicly. Scored 0 last week, scored 0 this week.

2. Stripe webhook accepting unsigned requests (tattoo studio SaaS)

Attacker sends a forged payment_intent.succeeded event. Gets a free tattoo appointment marked paid. Zero dollars transferred. Stripe's docs literally begin the webhook section with "always verify signatures." Vibe-built apps skip this because the happy-path template doesn't emphasize it.

3–5. Supabase RLS bypass × 3 live projects

All three projects ship the Supabase anon key in the browser (intended, per Supabase's architecture). The problem is the tables: RLS isn't enabled, or the policy is USING (true). The anon key is supposed to honor RLS; without RLS there's nothing to honor. Result: any visitor can GET the entire profiles table, including user IDs, heights, sex, session tokens.

These are concrete examples of why the scanner exists. The authors used Supabase "the vibe way" — connect, create tables, deploy — without opening the RLS policy editor.

Scanner non-regression: did we go blind?

A week ago we shipped 15 false-positive-class fixes in one day. A reasonable worry: did any of that cleanup also silence real detection?

The reference set tells us. Ten URLs from prior-day runs, each with a known prior score:

URL	Prior	Now	Δ
partner-developer-po-ud7p.bolt.host	0	0	0
auto-co-landing…up.railway.app	0	0	0
litrixdemo.lovable.app	35	40	+5
ownex.replit.app	21	53	+32
mobile-buildathon.replit.app	21	74	+53
boltfaq.bolt.host	32	70	+38
goldmineai.vercel.app	48	70	+22
responsible-ai.netlify.app	56	78	+22

Every site with real critical findings stayed at the same score. Every site whose prior score was FP-inflated moved up — in direct proportion to how many false positives we killed. The scanner didn't go blind; it got honest.

Score-drift without context reads like regression. If you saw 21 → 74 in isolation you'd think we dialed the scanner down. What you're actually seeing is what each site was all along, minus the false positives we used to count against them. Proof lives in internal/provider/secrets/scanner_test.go — 9 anti-suppression gates, each proving a real leak still fires through the new guards.

What the corpus says about CSP

The most-hit finding across 100 scans was missing-csp: 57 of 100 apps had no Content-Security-Policy header at all. That's not a scanner quirk — it's the default on every platform we tested except Netlify.

Second-most-hit: missing-permissions-policy at 57. Third: server-version-disclosed at 53 — Server: Vercel / Server: cloudflare headers leaking CDN fingerprints.

Defaults win. Whatever the platform sends you, your app ships. Stronger security requires going out of your way.

What this tells vibe-coders

1. Know your platform floor. Lovable=82, Bolt=78, Deno=78. That's the ceiling from platform alone. Custom CSP only goes up from there.
2. Webhook signatures. If your site accepts POST /api/webhooks/stripe, you MUST verify Stripe-Signature. Not optional. Sekrd's active-probe will find you.
3. Supabase RLS is not a setting — it's a practice. Every table you create needs an explicit policy. If your profiles table is queryable with the anon key alone: that's a critical.
4. CSP is free. Add it. Four header lines move your score ~10 points on most vibe platforms.
5. Run a scan before you ship. We're going to keep doing these sweeps. Better to find it first.

Try it yourself

Free scan — no signup — at sekrd.com/scan. Score + top 3 findings + evidence. Sign up free to see all findings + fix prompts for Cursor / Claude Code / Lovable. Pro plan gets unlimited scans, Deep Inspect (AI re-reads source), CI badges, scheduled rescans.

Raw corpus for this post: 100 JSON files + summary.md + top_leaks.md + corpus.txt + runner.py reproducible from the repo under research-data/2026-04-22__big-qa-100/.