Problem
An AI tools directory with more than 28,000 listings needed every record collected, categorized, described, given an FAQ, translated into 9 languages, and published, then kept current. Doing that with LLMs at this scale raises problems a prototype never meets: model calls time out or return malformed output, anti-bot protection blocks scraping, listing URLs break, and a failure halfway through a run can leave a record looking finished when it isn't.
Context
Production work at TechPotion.ai, where I built and maintained the directory end-to-end as Backend & Automation Engineer. The pipeline ran on AWS and fed a multilingual Next.js frontend.
My Role
I owned the AI pipeline and the data layer around it: extraction, status-driven processing, the LLM stages, failure handling, URL healing, and the cloud automation.
Architecture
The pipeline is status-driven. Every record carries a status in PostgreSQL, and each stage only picks up records that the previous stage completed:
- 1.Extraction. Listings scraped from 10 platforms, then social links extracted.
- 2.LLM stage 1: labels and categorization.
- 3.LLM stage 2: description enrichment.
- 4.LLM stage 3: FAQ generation.
- 5.LLM stage 4: translation into 9 languages.
- 6.Thumbnail generation.
- 7.Complete: ready for the frontend.
Because progress lives in the database rather than in a running process, any stage can be stopped, resumed, or rerun without redoing finished work.
Technology Stack
Python, OpenRouter with Gemini Flash Lite, PostgreSQL, Playwright, Selenium, BeautifulSoup, AWS (EC2, Lambda, EventBridge, SQS, RDS PostgreSQL, API Gateway, S3), Next.js, FastAPI.
Implementation
Model choice. The LLM stages used Gemini Flash Lite through OpenRouter. It was chosen for its large context window (crawled pages are long), low cost per token at this volume, and its fit for the translation stage, which is by far the largest workload.
Structured outputs. LLM calls run in JSON mode, so every response has to match the expected schema before it's written.
Status-gated processing. A stage only advances a record's status when its output is valid. If a call fails because of a timeout, malformed output, a JSON parsing error, or a model/API failure, the failure is recorded and the record stays at its current stage for later retry or reprocessing. A failed LLM stage never silently marks a record as done.
Controlled runs. Stages can run on a limited batch first, which made it possible to test prompt or model changes on a small sample before running them across all records.
Parallel, resilient extraction. Scrapers with 20+ extraction methods run as a fault-tolerant parallel system: 50 concurrent workers, dynamic rate limiting, circuit breakers, and anti-bot countermeasures. A URL-healing service detects broken listing links and diagnoses or recovers them.
Reliability / QA
The design goal was that no failure could corrupt a record or hide itself. Status gating means a bad model response becomes a visible, retryable error rather than a broken listing. I also ran structured QA on the platform across authentication, XP tracking, certificates, and Content Studio components.
Challenges
- •LLM calls failing or returning invalid output across tens of thousands of records
- •A translation workload many times larger than the other stages
- •Anti-bot blocking and rate limits across many external sources
- •Broken or moved listing URLs at scale
Solution
Treat the pipeline as a state machine: explicit per-record status, one job per stage, schema-checked model output, failures recorded instead of skipped, and a cost-efficient model for the heaviest stage. Around it, concurrency with guardrails and a dedicated URL-healing service.
Results
- •28,000+ production records processed and maintained
- •4 sequential LLM stages, including translation into 9 languages
- •5.6x processing speedup across 28K+ URLs, as reported
- •95% reported extraction success rate across 28K+ URLs
- •The same platform's monthly AWS costs were later reduced by 34% (see AWS Infrastructure & Cost Optimization)
Lessons Learned
In production, the hard part of an LLM pipeline isn't the prompt. It's making sure a failed call is noticed, recorded, and retried. Status-gated stages gave the pipeline that property, and they also made it cheap to rerun any single stage after changing a prompt or model.
This pipeline is also where my research started. In the translation and enrichment stage, roughly 2 in 10 outputs needed a retry or came back as malformed JSON. That was a rough, fluctuating observation from this pipeline, not a measured rate. Status gating caught those failures, but it raised a harder question: where a wrong output is a security risk rather than a broken listing, can model output be checked formally instead of only validated against a schema? That question became Sentinel-Mesh, a research framework that accepts an LLM-generated cloud-security fix only when a Z3 solver verifies it.