Benchmark · 20 November 2026

BuildWithThis open lab: our local run

Sekrd run locally on the open lab from the published 16 October 2026 test Is my vibe-coded app safe? A security check of 6 scanners. Their results come from their method; ours are a vendor's self-run and are not comparable.

Who was in their test

The six tools were Supabase Security Advisor, Aikido Security (free Developer plan), Semgrep Community Edition 1.181.0, GitHub code scanning + secret scanning + Dependabot, Prodable's free URL scan, and VibeScan's free URL scan. Lovable's built-in scan, CheckVibe, and Vibe App Scanner were not part of it. Their page says Lovable was not tested. Those products are compared separately on /compare, from their own public pages, and those pages are not this lab.

Lab

Public source: pathakarchit/bwt-vuln-lab, branch vulnerable, commit 996d7c4059333c43780992276587df90490e8c18 (the tree the test script calls the scored vulnerable copy). Ten planted issues, V1–V10, and four decoys, D1–D4. Every key in that repo is a dummy.

We did not scan https://bwt-vuln-lab.pages.dev or https://mqdhisrirpesywdzoosj.supabase.co. The script says that Supabase project is paused or deleted after the test. Our copy:

  1. Clone the commit above.
  2. Replace the project host in the local tree with bwtlocal.supabase.co, which /etc/hosts points at 127.0.0.1. The dummy JWTs stay as published. The scanner never dials the hostname from the repo.
  3. npm ci && npm run build, then serve dist/ on http://127.0.0.1:4173. That is the Cloudflare Pages shape: the built JS, not .env and not package-lock.json.
  4. Load supabase/migrations/20261016060000_vuln_lab.sql into a local Postgres, with stub auth and storage schemas so the migration applies.
  5. Serve a local HTTPS stand-in on bwtlocal.supabase.co that answers the read-only calls the migration implies: anon can read notes and user_emails, anon cannot read profiles or todos, anon can list uploads and cannot list public-assets, and GET /functions/v1/admin-users with the anon key returns a fake profile. No writes.

Sekrd's HTTP client refuses loopback addresses except inside a test binary. The scan harness is internal/bwtbench (//go:build bwtlab). It registers the same providers as internal/scan/engine.go and pre-fetches JS the same way.

go test -tags bwtlab -count=1 -timeout 8m -run TestLocalLab ./internal/bwtbench/
  • Quick is ScanType=free (secrets, dependencies, Nuclei, auth, payments, cost exposure, compliance, JS, IaC).
  • Deep, URL only is ScanType=deep with no integration. That adds infra, active probes, and the network port check.
  • Deep + database is the same deep scan plus the Supabase integration the product accepts on a deep scan: project ref, the dummy service-role key read from the local src/admin.js, and a database URL pointed at the local database.

Nuclei was not installed on this machine, so that provider contributed no findings. These planted issues are not Nuclei template checks.cmd/compliance-eval needs GEMINI_API_KEY and was not run. In the harness, quick skips the providers named supabase, firebase, mcp, infra, activedast, and network.

What each planted issue is

IDPlanted issueWhere it is visible
V1public.notes has RLS offAnon read of /rest/v1/notes; also pg_tables
V2public.profiles policy is TO authenticated USING (true) for select and updatePolicy SQL. Anon reads return no rows
V3Service-role JWT in src/admin.jsProduction JS bundle
V4.env committed with dummy Stripe and OpenAI keysGit only. Not in the bundle, not served
V5Bucket uploads is public and anon can list and insertAnon list; storage.objects policies
V6Edge Function admin-users, verify_jwt = false, returns profilesGET /functions/v1/admin-users with the anon key
V7View public.user_emails selects auth.users and is granted to anonAnon read; view definition
V8Note text rendered with dangerouslySetInnerHTMLProduction bundle: dangerouslySetInnerHTML:{__html:R.text}
V9lodash@4.17.20Bundle header next to __lodash_hash_undefined__. Lockfile is not served
V10public.delete_note is SECURITY DEFINER, no auth.uid() check, executable by anonFunction definition. We do not call it

Decoys: D1 anon key in the bundle (meant to be public), D2 todos with an owner policy, D3 .env.example placeholders, D4 bucket public-assets public-read for one logo and no list or insert policy.

Before the rule changes

Sekrd quick 1 of 10, deep URL only 2 of 10, deep + database 4 of 10.

IDSekrd quickSekrd deep, URL onlySekrd deep + database
V1noyesyes
V2nonoyes
V3yesyesyes
V4nonono
V5nonoyes
V6nonono
V7nonono
V8nonono
V9nonono
V10nonono
Caught124

BuildWithThis's own results for six tools are on their page. We don't reproduce them next to ours because the methods differ.

Before, deep + database also flagged decoy D4 (public-assets) at high severity because every public: true bucket was reported as world-readable. That is the one false positive against their decoy list. D1, D2, and D3 were not flagged. The handle_new_user trigger was not flagged.

After the rule changes

Sekrd quick 3 of 10, deep URL only 7 of 10, deep + database 9 of 10. These rules were written after the first run against this same lab. The after numbers are on a lab we had already seen.

IDSekrd quickSekrd deep, URL onlySekrd deep + database
V1noyesyes
V2nonoyes
V3yesyesyes
V4nonono
V5noyes, anonymous listyes, list and insert policies
V6noyesyes
V7noyesyes
V8yesyesyes
V9yesyesyes
V10nonoyes
Caught379

BuildWithThis's own results for six tools are on their page. We don't reproduce them next to ours because the methods differ.

V2 and V6 are the two items no tool caught in BuildWithThis's test. In our self-run, deep + database flagged V2 on the first run and V6 only after we added a rule for it. That is not a like-for-like comparison with their results.

V4 is still a miss. The keys are not in the served bundle, and dist/ does not serve .env. Sekrd's deep scan does not clone the app repository (the GitHub input on a deep scan is the MCP configuration scanner). A rule that only matches this repo's .env would not be a general detector.

V2 stays a miss on URL-only scans. Anonymous reads of profiles return no rows, which is what the policy says. The owner check needs the policy SQL from the database connection, which the deep scan already did before this change.

V10 is not invoked. Calling delete_note would delete a row. The deep database scan reads pg_proc instead. Trigger functions are skipped, so handle_new_user stays quiet.

Rules added after the first run

The list is the note's "What changed" section. Rule ids are the catalog ids those detectors set. The harness writes the finding list to a temp file that is not kept in the repo, so a title below is the string the detector emits, not a line copied out of a saved log.

  • js-dangerous-html — flag dangerouslySetInnerHTML when __html is not a string literal and not a DOMPurify / sanitize call. The production bundle keeps this property name. React's own runtime checks (=== "dangerouslySetInnerHTML") do not match.
  • js-lodash-vulnerable — if the lodash build markers sit next to a version older than 4.18.0, flag it. OSV, queried 20 November 2026: GHSA-35jh-r3h4-6jhm and GHSA-29mw-wpgm-hmr9 fixed in 4.17.21, GHSA-xxjr-mmjv-4gpg fixed in 4.17.23, GHSA-r5fr-rjxr-66jc fixed in 4.18.0. A bare version string with no lodash marker does not fire. Severity is high, not a hard block.
  • Active probe: read the PostgREST OpenAPI document and probe those relations, instead of only a fixed name list. That is how user_emails is seen. If the document is missing, the previous name list still runs. A non-empty anon read still uses rule supabase-rls-bypass.
  • supabase-storage-anon-list — POST /storage/v1/object/list/<bucket> for bucket names in the bundle. A non-empty list is the finding. public-assets returns a denial and is not flagged.
  • supabase-edge-unauthenticated — GET /functions/v1/<name> with the anon key. A 200 that contains an email or phone field is the finding. A {"ok":true} body is not.
  • supabase-auth-users-exposed — views in public that select auth.users and are granted to anon, authenticated, or PUBLIC.
  • supabase-security-definer-exposed — SECURITY DEFINER functions granted to those roles whose body never mentions auth.uid() or auth.jwt(), excluding triggers. The database scan reads pg_proc.
  • supabase-storage-anon-list and supabase-storage-anon-insert from storage policies that let anon list or insert without an owner check.
  • Removed the "every public bucket is a vulnerability" finding. public = true is the documented way to serve one static file. D4 no longer fires. V5 still fires from the list probe and from the policies.

What Sekrd reported, per item

Pass and miss are the two tables above. Titles are from the detectors and from tests that use this lab's names (internal/provider/activedast/supabase_public_test.go, internal/provider/supabase/catalog_test.go, and the title map in internal/findings/catalog.go). Where the repo does not name a title, this section says so.

V1 — notes, RLS off

Quick no, deep URL yes, deep + database yes, before and after. Quick does not run activedast or the Supabase provider. The active-probe test built on this lab's OpenAPI paths expects a finding whose title contains notes and whose rule is supabase-rls-bypass. That probe's title format is Supabase table 'notes' readable without authentication when anon GET /rest/v1/notes returns rows. The same test expects profiles and todos to stay quiet when the body is []. The database path's title format is RLS disabled on 'notes' table, from pg_tables when rowsecurity is false. That finding does not set a rule id in the provider.

V2 — profiles policy is authenticated USING (true)

Quick no and deep URL no, before and after. Deep + database yes, before and after. URL scans miss it because anonymous reads of profiles return no rows. The owner check needs the policy SQL from the database connection. The rule is supabase-rls-auth-no-owner. Its title format is RLS <command> on '<table>' allows any authenticated user. For this policy the commands are SELECT and UPDATE on profiles. The catalog test uses that title shape with a different table name, not a saved harness log for profiles.

V3 — service-role JWT in the bundle

Yes on quick, deep URL, and deep + database, before and after. The secrets provider runs on a free scan. Its title for this pattern is Potential supabase_service_role_key detected in client code, rule secret-pattern-match. The pattern requires the service_role marker inside the JWT. The anon key is a different pattern and is not this finding. The note places the key in src/admin.js and in the production bundle. The harness reads the local admin.js only to pass the dummy key into the deep + database integration.

V4 — committed .env

Miss on every Sekrd mode, before and after. No finding title: the keys are not in the served bundle, dist/ does not serve .env, and a deep scan does not clone the repository. The GitHub input on a deep scan is the MCP configuration scanner.

V5 — uploads bucket, anon list and insert

Before: only deep + database, via the check that also flagged D4. Every bucket with public: true was reported as world-readable, at high severity. The note calls that finding "public bucket". The current code no longer emits it (CheckStorageBuckets returns nothing). After: deep URL is "yes, anonymous list". The catalog maps Storage bucket 'uploads' is listable without login to supabase-storage-anon-list. Deep + database is "yes, list and insert policies", which adds Storage bucket 'uploads' accepts uploads without login (supabase-storage-anon-insert). The active-probe test expects supabase-storage-anon-list and fails if a title contains public-assets.

V6 — admin-users without verify_jwt

Before: miss on every Sekrd mode. After: deep URL yes and deep + database yes. Quick still no, because the probe is activedast. The title in the catalog is Edge Function 'admin-users' returns personal data without login, rule supabase-edge-unauthenticated. The probe test requires that rule when GET /functions/v1/admin-users returns an email or phone, and a separate test checks that a {"ok":true} body does not fire it.

V7 — user_emails view over auth.users

Before: miss on every Sekrd mode. After: deep URL yes and deep + database yes. The OpenAPI probe is why the URL scan sees user_emails. The active-probe test expects a title containing user_emails. The title format for a non-empty anon read is Supabase table 'user_emails' readable without authentication (supabase-rls-bypass). The database check's title is View 'user_emails' exposes auth.users, rule supabase-auth-users-exposed, from a view that selects auth.users and is granted to the Data API.catalog_test.go builds that finding for a view named user_emails.

V8 — dangerouslySetInnerHTML

Before: miss on every Sekrd mode. After: yes on quick, deep URL, and deep + database. JS analysis runs on a free scan. Title dangerouslySetInnerHTML renders dynamic HTML, rule js-dangerous-html. The note's bundle snippet is dangerouslySetInnerHTML:{__html:R.text}. A string literal or a DOMPurify / sanitize call does not match, and React's own === "dangerouslySetInnerHTML" checks do not match.

V9 — lodash 4.17.20 in the bundle

Before: miss on every Sekrd mode. After: yes on quick, deep URL, and deep + database. The catalog maps Vulnerable lodash 4.17.20 shipped in client JavaScript to js-lodash-vulnerable. The detector fires when __lodash_hash_undefined__ or __lodash_placeholder__ sits next to a version older than 4.18.0. The lockfile is not served, so this is the bundle header, not package-lock.json. Severity is high, not a hard block.

V10 — delete_note, SECURITY DEFINER

Before: miss on every Sekrd mode. After: deep + database yes. Quick and deep URL stay no. The function is not called. Calling it would delete a row. The database scan reads pg_proc (proname, prosecdef, pg_get_functiondef).catalog_test.go expects rule supabase-security-definer-exposed for a function named delete_note whose body has no auth.uid() check. The catalog title is SECURITY DEFINER function 'delete_note' is executable without an ownership check. The same test expects the trigger handle_new_user to stay quiet.

Decoys and false positives

D1 is the anon key in the bundle, which is meant to be public. D2 is todos with an owner policy. D3 is .env.example placeholders. D4 is bucket public-assets, public-read for one logo, with no list or insert policy. Nuclei was not installed, so it did not add findings, and these planted issues are not Nuclei template checks.

CheckBeforeAfter
D1 anon key at high or criticalnono
D2 owner-only todosnono
D3 .env.example placeholdersnono
D4 public-assets described as listable or uploadableyes, high, "public bucket"no
handle_new_user triggernono
False all-clear (said a planted check passed)nono

Extra findings that are true for this local static server and are not planted holes: missing security headers, no privacy-policy link. The HTTP, SPF, and DMARC notes are because the harness URL is http://127.0.0.1:4173, not because the published Pages site failed TLS.

Source note: docs/benchmarks/buildwiththis.md (internal note, not public). Their article: buildwiththis.com/jobs/security-check-vibe-coded-app. Their method: buildwiththis.com/tests/security-check-vibe-coded-app.

All benchmarks · Comparisons from public pages