Problem
ToolPotion is a comprehensive AI tools directory indexing over 28,000 listings across categories, pricing models, and functional tags. At this catalog scale, data integrity drift, incorrect AI-generated taxonomies, broken hyperlinks, and malicious third-party submissions threaten directory authority, SEO rankings, and user safety.
The platform required rigorous multi-round QA to audit directory taxonomies, verify listing integrity, and intercept security threats.
Context
Quality Assurance Lead governing data accuracy, search rankings, link health, and submission safety across 10 structured QA rounds on a 28,000+ listing production directory.
My Role
Quality Assurance Lead: established classification standards, conducted manual and automated directory sweeps, flagged abuse and security risks, and tracked defect remediation and retest closures.
Architecture & Coverage Scope
Testing covered 10 QA rounds spanning directory search, category taxonomies, ranking algorithms, metadata integrity, outbound link health, and third-party developer submissions:
- •Categorization & Taxonomies: AI domain mapping, sub-category routing, and multi-tag consistency
- •Search & Ranking: Relevancy scoring, keyword match accuracy, and featured listing precedence
- •Listing Data Integrity: Developer URLs, pricing tiers, logo assets, feature summaries, and duplicate suppression
- •Safety & Malicious Submissions: Phishing detection, malware scanners, and redirection hygiene
Reliability & Key Findings
Over 10 QA rounds, testing logged 1,300+ defects (1,398 total logged entries across all rounds). Defect severity breakdown recorded 172 critical failures, 1,003 major issues, and 223 minor items. Systematic retesting verified and closed 316 fixes.
Top defect categories included:
- •Wrong category / miscategorization: 544 entries where AI or submitters assigned tools to incorrect functional categories.
- •Ranking and relevancy failures: 457 entries with inaccurate search ranking, keyword drift, or suppressed relevancy scores.
- •Data integrity errors: 214 entries with broken metadata, truncated pricing, or mismatched platform fields.
- •Duplicate listings: 122 entries containing duplicated tool records across different submission batches.
- •Dead or fake tools: 71 logged entries identifying 190+ dead, abandoned, or counterfeit tools.
- •Broken listing links: 22 entries with 404 dead links or expired target domains.
Critical security findings uncovered during sweeps included:
- •Phishing-flagged listing: Identified and removed a counterfeit listing designed to steal user credentials.
- •Malware-flagged listing: Intercepted and quarantined a submitted download link pointing to malicious payloads.
- •Gambling redirect clone: Flagged and purged a clone listing configured with a hidden redirect to an external gambling website.
Results & Impact
- •1,300+ defects documented across 10 rigorous QA rounds (1,398 total entries)
- •316 verified and closed fixes through regression sweeps
- •Purged 190+ dead or fraudulent tools and blocked active phishing, malware, and gambling redirects
- •Restored search relevancy and categorization accuracy across 28,000+ directory records