Skip to content

QA-Report-2026-04-06.md

Latest commit

 

History

History
265 lines (200 loc) · 11.5 KB

QA-Report-2026-04-06.md

File metadata and controls

265 lines (200 loc) · 11.5 KB

CertiPrepAI — Full QA Report

Date: 2026-04-06 Method: Live browser interaction + full source code audit + AWS infrastructure scan Build: Amplify Job #131 (SUCCEED) — latest Auditor: Claude (Cowork)

⚠️ EXECUTIVE SUMMARY

The site loads and renders. But 3 out of 4 paid premium features are permanently broken for all paying customers. The DB Lambda is returning 403 for logged-in free users causing infinite loading. No mobile navigation exists. These are revenue-destroying bugs that must be fixed before engaging real testers or real users.

🔴 CRITICAL BUGS (fix immediately — breaks paid features)

BUG-01 — Visual Exam, Architecture Builder, and Diagrams NEVER unlock for paid users

Severity: 🔴 CRITICAL Files: src/pages/VisualExam.tsx:148, src/pages/ArchitectureBuilder.tsx:165, src/pages/Diagrams.tsx:472 Root cause: All three pages use leftover Supabase auth syntax:

const tier = (user as any)?.user_metadata?.tier  // WRONG — Supabase pattern

The Cognito user object has no user_metadata property. tier is always undefined. isPremium is always false. Every paying customer (Monthly, Yearly, Lifetime) sees the paywall even after they pay.

Fix:

// Replace in all 3 files:
const { tier } = useAuth()  // already available from AuthContext
const isPremium = tier === 'monthly' || tier === 'yearly' || tier === 'lifetime'

Impact: 100% of paid users cannot access 3 of 4 premium features they paid for.

BUG-02 — DB Lambda returns 403 for logged-in free users → infinite loading spinner

Severity: 🔴 CRITICAL Files: src/pages/CertDetail.tsx:82, src/lib/db.ts:18 Root cause: getFreeUsage() is called when a free-tier user opens any cert page. The awsprepai-db Lambda returns HTTP 403 (JWT verification failure). The .then() call has no .catch(), so:

  • setUsageLoaded(true) is never called
  • isLoading stays true forever: 
    const isLoading = ... || (tier === 'free' && !usageLoaded)
  • Free users see a permanent loading spinner and can never practice.

Fix: Two parts:

  1. Add .catch() to getFreeUsage() call so usageLoaded always resolves
  2. Investigate why the Lambda is rejecting valid Cognito access tokens (check JWT signature/expiry validation in Lambda code)

Same issue affects monthly-tier users: getMonthlyCert() has the same missing .catch() — monthly users also get stuck on infinite loading. (src/pages/CertDetail.tsx:91, src/pages/Dashboard.tsx:55)

BUG-03 — Signup with paid plan sends user to Stripe before email is verified

Severity: 🔴 CRITICAL File: src/pages/Signup.tsx:52-67 Root cause: After signUp(), if plan is paid, the code immediately calls the Checkout API and redirects to Stripe. The user's Cognito account is not email-verified yet. After paying, they return to /payment-success, which calls the verify Lambda. But they cannot log in afterwards because Cognito requires email confirmation first.

Broken flow: SignUp → Stripe checkout (unverified account) → Payment success → Cannot log in → Rage churn

Fix: Complete email verification step first, then redirect to Stripe checkout from /login?verified=1&plan=monthly.

🟠 HIGH PRIORITY BUGS

BUG-04 — No mobile navigation at all

Severity: 🟠 HIGH File: src/components/Navbar.tsx (264 lines, zero responsive code) Root cause: No @media queries, no hamburger menu, no mobile state, no breakpoints anywhere in Navbar. At 390px (iPhone), all 7 nav items render horizontally — they visually overflow or collapse into each other. Impact: All mobile users (~50% of traffic) have a broken navigation experience. Fix: Add hamburger/drawer mobile nav with breakpoint at 768px.

BUG-05 — No SEO meta tags whatsoever

Severity: 🟠 HIGH File: index.html Problem: Only <title>CertiPrepAI</title> exists. Missing:

  • <meta name="description">
  • <meta property="og:title/description/image/url">
  • Twitter Card tags
  • Canonical URL Impact: Google ranking is severely hurt. Sharing on LinkedIn/Twitter/Slack shows no preview card.

BUG-06 — AI Coach: advertised in Lifetime plan but does not exist

Severity: 🟠 HIGH Evidence: /pricing page lists "AI Coach (exclusive — lifetime only)" as a feature. No /ai-coach route exists in App.tsx. No AI Coach component exists anywhere in the codebase. Impact: False advertising. Lifetime customers ($147) pay for a feature that doesn't exist. Fix: Build it or remove it from the pricing page immediately.

🟡 MEDIUM PRIORITY BUGS

BUG-07 — "Total Questions" stat uses ❓ icon — looks like a broken state

Severity: 🟡 MEDIUM File: src/pages/Dashboard.tsx:176 Code:

{ label: 'Total Questions', value: '3,120', icon: '❓' }

The ❓ emoji is used as the icon for the "Total Questions" stat card. To users it looks like the question count is unknown or broken. Should be 📝 or 🔢 or similar.

BUG-08 — Visual Exam and Architecture Builder: Monthly plan inconsistency

Severity: 🟡 MEDIUMProblem: The code in VisualExam.tsx and ArchitectureBuilder.tsx checks:

const isPremium = tier === 'monthly' || tier === 'yearly' || tier === 'lifetime'

This means Monthly ($7/mo) would unlock Visual Exam and Architecture Builder. But the Pricing page only lists "Visual Exam + Architecture Builder" under the Yearly plan features — Monthly plan features list does NOT include them. Impact: Either Monthly users are being under-sold (they get more than advertised) or Yearly users are being misled about exclusivity. Decide which is correct and make both the code and pricing page consistent.

BUG-09 — Footer "FAQ" link goes nowhere

Severity: 🟡 MEDIUM File: Footer in all pages Problem: href="/pricing#faq" — there is no element with id="faq" anywhere on the Pricing page. Fix: Add a FAQ section to Pricing page with id="faq", or remove the footer link.

BUG-10 — Footer "Mock Exams" link goes to practice mode, not mock exam mode

Severity: 🟡 MEDIUM Problem: Footer "Mock Exams" → /certifications. To trigger mock exam mode, the URL must be /certifications?mode=mock. Fix: Change footer href to /certifications?mode=mock.

BUG-11 — Dashboard: getMonthlyCert fails silently for monthly users

Severity: 🟡 MEDIUM (related to BUG-02) File: src/pages/Dashboard.tsx:55 Code: getMonthlyCert(user.accessToken).then(...) — no .catch(). If the DB Lambda is broken, the monthly cert selection widget either shows nothing or hangs. Same root cause as BUG-02.

🔵 LOW PRIORITY / POLISH

BUG-12 — Footer GitHub link points to wrong repo

File: Footer Problem: https://github.com/Isaloum/CertiPrepAI → repo name is AWSPrepAI. Fix: Change to https://github.com/Isaloum/AWSPrepAI.

BUG-13 — No robots.txt or sitemap.xml

Fix: Add public/robots.txt (allow all) and public/sitemap.xml listing all public routes.

BUG-14 — No "Resend verification code" button on signup confirmation step

File: src/pages/Signup.tsx Fix: Add a resend button that calls cognitoUser.resendConfirmationCode().

BUG-15 — Certifications page filter has "Foundation" not "Foundational"

File: src/pages/Certifications.tsx Problem: The filter button says "Foundation" but certification level is labeled "Foundational" on cert cards. Inconsistent labeling.

BUG-16 — No OG image for social sharing

Fix: Create public/og-image.png (1200×630px) and add <meta property="og:image"> to index.html.

4. CONTENT ACCURACY

Item Status
All 12 cert JSON files present ✅ All 260 questions each, confirmed
Total = 3,120 questions ✅ 12 × 260 = 3,120
Glossary ⚡ Caching category ✅ Live and working
Keywords page ✅ 14 categories, working
Pricing prices consistent ✅ $7/$67/$147 consistent across pages
"50+ terms" claim on homepage ✅ Glossary has far more than 50 terms
AI Coach feature claim ❌ Does not exist (BUG-06)

5. INFRASTRUCTURE

Item Status Notes
Amplify hosting ✅ Job #131 SUCCEED
CloudFront CDN ✅ Distribution active, cache invalidated
SPA rewrite rules ✅ Fixed — 200 status, assets excluded
awsprepai-db Lambda ⚠️ AuthType: NONE but returns 403 — JWT verification broken
awsprepai-checkout API GW ❓ Needs live test — couldn't verify from sandbox
Cognito User Pool ✅ us-east-1_bqEVRsi2b — env vars configured
HTTPS / SSL ✅ Valid
404 page ✅ Custom NotFound renders correctly

6. PRIORITY FIX ORDER FOR TESTERS

# Bug Severity Effort Impact
1 BUG-01 — 3 premium pages never unlock 🔴 Critical Trivial (3-line fix each) All paid users
2 BUG-02 — Free users: infinite loading spinner 🔴 Critical Low All free users
3 BUG-03 — Payment before email verification 🔴 Critical Medium All new paid signups
4 BUG-04 — No mobile nav 🟠 High Medium ~50% of all users
5 BUG-05 — No SEO meta tags 🟠 High Low Growth/SEO
6 BUG-06 — AI Coach doesn't exist 🟠 High High (build) or Trivial (remove) Lifetime buyers
7 BUG-07 — ❓ icon on Total Questions 🟡 Medium Trivial Cosmetic trust
8 BUG-08 — Monthly/Yearly plan feature mismatch 🟡 Medium Low Pricing clarity
9 BUG-09 — FAQ link broken 🟡 Medium Low Support UX
10 BUG-10 — Mock Exams footer link wrong 🟡 Medium Trivial Navigation
11 BUG-12 — Wrong GitHub link 🔵 Low Trivial Credibility
12 BUG-13 — No robots/sitemap 🔵 Low Low SEO
13 BUG-14 — No resend verification code 🔵 Low Low Auth UX
14 BUG-15 — Foundation vs Foundational 🔵 Low Trivial Copy consistency
15 BUG-16 — No OG image 🔵 Low Low Social sharing

7. TESTER SCENARIOS (after BUG-01/02/03 are fixed)

Auth

  • New signup (free) → verify email → log in → dashboard shows Free plan
  • New signup with ?plan=monthly → verify email → log in → Stripe checkout → payment → access unlocked
  • Login with wrong password → error shown
  • Forgot password → reset email → new password → login
  • Sign out → confirm redirect

Free tier

  • Open any cert page → question loads without spinner (BUG-02 fix verification)
  • Answer 20 questions → paywall appears at question 21
  • Mock exam → paywall shown
  • Visual Exam → paywall shown
  • Architecture Builder → paywall shown
  • Diagrams → paywall shown

Paid tier (Monthly)

  • Open any cert page → all 260 questions accessible
  • Mock exam → 65 questions, 130-min timer, score on submit
  • Visual Exam → 20 diagram questions accessible (BUG-01 fix verification)
  • Architecture Builder → 10 challenges accessible (BUG-01 fix verification)
  • Diagrams → 14 diagrams accessible (BUG-01 fix verification)

Mobile (390px width)

  • Hamburger menu opens/closes
  • All pages scroll correctly
  • Buttons are tappable

Report v2 — generated after full browser interaction + code audit + AWS infra scan. Previous report missed BUG-01, BUG-02, BUG-03 — all are revenue-critical. Fix before any tester onboarding.