01 · Pillar
LLM Evaluation & Quality
How to measure whether an LLM feature is good enough to ship — metrics, judges, golden sets, and the evidence that a change did not quietly regress.
All 19 in LLM Evaluation & Quality →Home / Learn
The reference library
Everything I know about shipping AI you can trust.
A working reference on evaluating, testing, and shipping AI features — organized into eight pillars, from LLM evaluation to workflow automation. Built from real engagements, and every claim links to a receipt. New material lands here continuously; the docs cover how engagements run.
02 · Pillar
RAG & Retrieval Quality
Evaluating retrieval-augmented generation end to end: faithfulness, context precision and recall, chunking, reranking, and citation accuracy.
All 9 in RAG & Retrieval Quality →03 · Pillar
AI Agent Reliability
Making multi-step, tool-using agents trustworthy: trajectory evaluation, tool-call accuracy, multi-turn testing, and the non-determinism problem.
GuideAI agent testingField noteEighteen agents audited a live curriculumField noteFive pages, five agents, zero conflicts
All 9 in AI Agent Reliability →
04 · Pillar
AI Testing in CI/CD
Wiring evaluation into the pipeline — eval gates, quality ratchets, regression suites, canary releases, and "no fake green" as an actual workflow.
ReferenceEval gates: the missing CI step for AIField noteLLM regression testing with Promptfoo in CIField noteYour LLM feature needs a regression suite
All 15 in AI Testing in CI/CD →
05 · Pillar
Test Automation & QA
Classic software quality engineering, sharpened for AI-powered products: flaky tests, Playwright, visual regression, coverage that means something.
All 9 in Test Automation & QA →06 · Pillar
Shipping AI Safely
The product- and leadership-level view: launch checklists, "passing tests isn't proof," build-vs-buy, cost, rollback plans, and hiring for AI quality.
All 10 in Shipping AI Safely →07 · Pillar
AI Workflow Automation
Business-process automation with AI in the loop: support triage, lead qualification, human-approval steps, failure modes, and honest ROI.
All 9 in AI Workflow Automation →08 · Pillar
Tools & Landscape
Straight comparisons and landscape maps across the eval, RAG, and testing tool ecosystems. Written from hands-on use, not SEO farming.
ReferenceStack & integrations (reference)ReferenceProduct & platform (reference)CompareTool comparisons
All 10 in Tools & Landscape →
Want this applied to your product?
Ask my AI associate on any page, or book a 15-minute call and we'll scope it.