Hira Ahmed
Case Study · buildClient: Independent build·Role: Designed and built both versions end-to-end·Practice Area: AI Agent Development, AI Automation, n8n Automation

Autonomous Cloud Compliance & AI Auditing Engine

LangGraph Corrective RAG auditing engine answering 250+ question reviews from live AWS evidence with verified citations. Read the case study.

~90%
Reported cut in questionnaire turnaround
250+
Questions per enterprise review
2,000+
Questions processed, no timeout failures (reported)
0.95
Reported semantic relevance score
System Architecture & Execution Flow
Evidence-based AI auditing engine, second versionQuestionnaire questions go through an n8n and PostgreSQL queue to a FastAPI service running a LangGraph Corrective RAG state machine. It retrieves live AWS evidence and policy passages from pgvector, grades relevance with Llama 3.1 8B, corrects weak retrievals, generates an answer, and a Critic Agent rejects any answer without a verified ARN or document citation.Security questionnaireCAIQ · SOC 2 · SIGQueue workern8n + PostgreSQL · row locksFastAPI serviceone question per runLangGraph state machine · Corrective RAGRetrieveAWS evidence +pgvector passagesGrade relevanceLlama 3.1 8B on GroqGenerate answerfrom graded passagesCritic Agentverify ARN ordocument sectionrelevantCorrect retrievalre-query weak passagesweakno verified citationAWS account50+ CIS checks · CLI sidecarPolicy PDFsPyMuPDF · FastEmbedCited answerARN or document sectionverified
Simplified view of the Corrective RAG loop. An answer is only accepted when the Critic Agent confirms it cites a real AWS ARN or document section. It answers questionnaires with evidence; it is not a certification or attestation.

Problem

Enterprise customers send long security questionnaires, such as CAIQ, SOC 2, and SIG reviews, often with 250 or more questions. Answering them by hand means checking live cloud configuration, searching policy documents, and writing up each answer with evidence. It is slow, and a generic AI answer is worse than none, because a confident answer with no real evidence behind it is a liability.

Context

A self-built project that went through two versions. The first version automated AWS infrastructure auditing and reporting. The second turned it into an evidence-based engine that answers security questionnaires from live AWS evidence and policy manuals. It supports security reviews; it is not a formal compliance certification or an independent attestation.

My Role

I designed and built both versions end-to-end.

Architecture

First version: automated AWS audit

  1. 1.A Python and Flask service in an isolated Docker container runs read-only AWS CLI audit commands and normalizes the JSON output.
  2. 2.n8n orchestrates extraction, evaluation, and storage.
  3. 3.A context-injected Langflow agent, using Groq, evaluates the collected configuration against security checkpoints.
  4. 4.Findings, evidence, and remediation roadmaps are stored in Supabase.

Second version: evidence-based questionnaire engine

  1. 1.Evidence collection. 50+ live AWS CIS Benchmark checks, run through the AWS CLI sidecar.
  2. 2.Document retrieval. Policy manuals parsed with PyMuPDF, embedded locally with FastEmbed (all-MiniLM-L6-v2), and stored in Supabase with pgvector.
  3. 3.Answering. A 6-node LangGraph state machine using Corrective RAG (CRAG). Retrieved passages are scored for relevance by Llama 3.1 8B on Groq, and weak retrievals are corrected before an answer is written.
  4. 4.Verification. A Critic Agent checks that every answer cites a real AWS ARN or a real document section before it's accepted.
  5. 5.Queueing. An asynchronous worker in n8n and PostgreSQL feeds questions through the engine, served by FastAPI.

Related case study: Autonomous AI Email Agent, another n8n-based system.

Technology Stack

LangGraph, LangChain, CRAG, LlamaIndex, Groq (Llama 3.1 8B), Supabase pgvector, FastEmbed, PyMuPDF, FastAPI, n8n, PostgreSQL, Docker, AWS CLI. The first version also used Langflow and Flask.

Implementation

  • •Every answer is required to carry a citation: a verified AWS ARN from the live account or a specific section of a policy document
  • •Relevance scoring decides whether retrieved context is good enough, or whether retrieval needs to be corrected first
  • •Embeddings run locally, so there's no paid embedding API
  • •The queue worker uses row-level processing locks so two workers never take the same question, 3-second delays to respect rate limits, and exponential backoff on failures

Reliability / QA

The design separates "collect" from "judge" and "judge" from "accept". The model only sees evidence that was actually collected or retrieved. The Critic Agent rejects answers whose citations don't check out. Queue locks and backoff keep long runs stable instead of failing on rate limits.

Challenges

  • •Making AI answers traceable to real evidence rather than plausible text
  • •Retrieving the right policy passage out of long PDF manuals
  • •Running thousands of questions without hitting model rate limits or timeouts

Solution

Evidence first, then retrieval with correction, then a critic that verifies citations, all driven by a queue that respects rate limits.

Results

  • •Automates answering of 250+ question enterprise security reviews (CAIQ, SOC 2, SIG)
  • •~90% reduction in questionnaire turnaround time, as reported
  • •0.95 semantic relevance score for retrieval, as reported, with no embedding API cost
  • •2,000+ questions processed with no timeout failures, as reported
  • •The first version produces a 100-point infrastructure assessment with a 0–100 risk score and a remediation roadmap

A recorded walkthrough of the first version is linked below. The second version doesn't have a public demo yet.

Lessons Learned

For compliance work, an AI answer is only useful if someone can check it. Requiring a verifiable citation for every answer, and rejecting the ones that fail, turned the model from a writer into a researcher whose work can be reviewed.

Author & System Architect

Hira Ahmed

AI Engineer & Systems QA Lead specializing in autonomous LLM pipelines, formal verification with Z3 SMT solvers, and mission-critical SaaS reliability.

GitHub →LinkedIn →ResearchGate →Preprint DOI →Topmate →Medium →

Technologies & Tools Used

LangGraphLangChainCRAGLlamaIndexGroq (Llama 3.1 8B)Supabase pgvectorFastEmbedPyMuPDFFastAPIn8nPostgreSQLDockerAWS CLI

External References

n8n-production-resilience-patterns →Repository →Demo of the first version (Loom) →
← Back to all Case StudiesReturn to Homepage