I'm who you bring in when you're about to ship an LLM feature that could embarrass you in front of a customer. I build it, and the gate that catches it the moment it drifts. see why →
LLM apps and agents, plus workflow automation (RAG, LangGraph, n8n/Make), paired with the regression and eval harness that keeps them honest in production: Playwright, Pytest, k6, Promptfoo, DeepEval, LLM-as-judge. Most people build the AI or test it. I do both, and every claim below links to an artifact.
✓ Quoted in writing before work starts. The audit fee credits into the build. No fit, no charge.
LLM & RAG evaluation, test automation & CI, and workflow automation — the AI feature and the evidence it holds up.
13 years in software quality at The Home Depot and in fintech. Today I build and run two live AI products solo — and this site runs its own QA in public.
A one-week AI Quality Audit. You leave with a plan either way, and the fee credits into the build if we continue.
Everyone ships the demo. I ship the proof it works.
Sixty seconds on how I build an AI feature and the gate that keeps it honest in production — in my own voice.
▸ every claim in the film links to a verbatim run below. no fake green — not even mine.
Everyone can build an AI demo now. Almost nobody can prove it works in production.
That gap is the whole job, and it's the part I have receipts for. Four, each linked to the verbatim run.
Everyone builds AI demos. Here's the work — with the numbers to back it.
▸ real systems, real numbers — every claim links to the run that produced it.
the sales assistant on this site gets red-teamed too. see the 10/10 results →
▸ 141 unit tests, the site's own 128-check QA gate, and an LLM eval — real commands, real numbers, in my voice.
Scroll it. The same five-stage flow, from manual-and-unprovable to automated-and-eval-gated. The shape of what I build for you.
The strongest proof I can offer isn't a testimonial — it's software I ship and operate solo, every day. Two products on one engineering system. Open the front door of either.
A live SaaS platform — member cockpit, admin ops, Stripe billing, real market-data engines, and a companion Discord AI engine. Production scale, one operator.
A live learning product — AI tutor on a real model, interactive labs, Stripe billing, and cryptographically verifiable certificates with a public verify page.
Agent warehouse monorepo: voice agents, outbound SDR, RAG support chat, and workflow orchestrators on Mastra + LangGraph — with a multi-provider LLM router, cost tracking per run, Langfuse tracing, and a hard TCPA consent gate on every outbound call.
Proof-first MCP engineering OS: 140 tools an AI agent drives through policy, signed approvals, and a hash-chained proof ledger. Nothing is "done" because a model said so — a claim-firewall rejects unproven success language.
RAG knowledge base where every answer is extractive and cited. Durable ingestion worker, pgvector retrieval, persisted eval runs, and a full audit trail from source to answer.
Release-critical e-commerce flows under regression with the evidence a release manager would ask for: traces, screenshots, four reporters, and a written risk model. CI uploads artifacts on every push.
85 quality runners under one CLI — including hallucination, jailbreak, prompt-injection, toxicity, and PII-leak evals for LLM features. Every score is computed from a real command and packaged as ed25519-signed evidence.
Answer 3 questions or talk to Nadine, my AI associate — you'll get an itemized plan with a price range in about 2 minutes. No signup, no sales call required.
✓ Prefer to start small? Begin with a fixed-price $497 audit week — credited straight into the build if you continue.
Outcome first; the full eight-section spec — including the parts most case studies hide — is one click deeper. Every headline number states how it was measured.
Negative feedback now surfaces in seconds — not at the Friday review.
Customer feedback arrived through a form and sat unread. Negative signals surfaced days late, after the customer was already gone.
Negative feedback surfaces in real time instead of at end-of-week review — measured as webhook-to-alert latency of seconds vs a manual weekly pass.
Nobody wants to converse with their own feedback queue. The value is deterministic: classify, summarize, route, alert. A chat interface would add latency and remove auditability.
Make scenario, four steps end to end — see fig. 1. Each step re-runnable in isolation.
None needed — each message is classified independently. Deliberately avoided: statelessness keeps the pipeline debuggable.
The AI never replies to a customer. It routes to a human with a summary; the alert email is the handoff, not the resolution.
Structured output schema on the LLM step, fallback label on parse failure, and the raw message always logged alongside the classification.
A golden set of labeled messages run nightly through Promptfoo to catch classification drift when the model version changes.
100% of answers cite their source. The pipeline has no uncited path.
Research teams need answers from their own corpus — but a RAG system that paraphrases confidently without sources is a liability, not a tool.
100% of answers citation-backed — verified by design: there is no uncited generation path. Quality, safety, latency, and cost tracked per query on the analytics endpoint.
A chat wrapper over the corpus was the obvious build. The actual need was an operations dashboard: sources, ingestion jobs, eval runs, and feedback all inspectable — the Q&A is one feature inside it.
FastAPI API + durable ingestion worker feeding fig. 1; provider gateway swaps local deterministic embeddings for Gemini per environment. React dashboard on top.
Persisted queries, answers, and citations. Deleting a source cascades: chunks, embeddings, jobs, payloads, citations — no orphaned memory.
Answer-level feedback endpoint feeds a review loop; eval runs are triggered and persisted by API so a human signs off before provider or threshold changes ship.
Extractive-only answers (cite or abstain), full audit trail on source/query/eval/feedback events, deterministic local providers for reproducible dev, Gemini fallback-to-local on missing keys.
Promote eval thresholds into deployment gates so a retrieval regression blocks the release, not the retro.
3,759 tests. On publish day the gate caught real CVE drift, blocked me, and was green again by afternoon.
Quality dashboards state numbers they can’t defend. For LLM features it’s worse: teams ship prompt changes with no regression signal at all.
3,759 tests, 91.12% line coverage, 13/13 gates. On 2026-08-15 the CVE gate went honestly red (15 high/critical in deps) and blocked its own release — patched and PROVEN again the same afternoon, both runs published verbatim. One command regenerates the scorecard.
An "AI QA assistant" that opines on quality is exactly the failure mode. The verdict must be computed by code from command exit codes — an LLM never grades its own homework here.
One CLI driving fig. 1: every runner is a plugin (detect → plan → run → collectEvidence); one hung runner can’t stall a run. Two gates: <15ms pre-commit, full-repo merge gate.
Evidence ledger instead of memory: every run’s exact command, exit code, and parsed metric recorded to a proof ledger, re-verifiable offline.
The autonomous fix loop is bounded. It fixes only what the harness can verify, commits on improvement, reverts on regression, stops on an honest stall for human review.
Anti-hallucination contract: no score without a backing artifact. Ratcheting coverage floors, honest-skip discipline, ed25519-signed redacted evidence.
Ten dedicated AI-safety runners: bias, consistency, hallucination, jailbreak, prompt-injection, refusal, toxicity, PII-leak, cost, latency — the harness I bring to client LLM features.
The market filled up fast with shops wrapping a rented Zapier flow in a markup. So don't take my word for it — here's the checklist I'd hand you to vet anyone, me included. Run every line on me.
See the eval battery block a release — before a customer would.
Ten checks, scored by code. One fails, the gate stops the ship. Then the real caught-blocked-proven arc from my own repo.
Fixed scope, fixed deliverables, evidence included. Each offer is a productized version of something in the briefs above.
Hiring full-time? These are the three capabilities I'd bring to your team on day one, each backed by a shipped system above.
Your LLM feature gets a regression suite: golden traces, LLM-as-judge scoring for faithfulness and safety, and a CI gate that blocks the deploy when quality drops.
A risk-scoped Playwright/Pytest suite with the evidence discipline of a release manager: traces, reports, artifacts — wired into GitHub Actions from day one.
A working automation — intake, triage, routing, RAG-backed answers — built on n8n/Make or LangGraph, with human approval points where they belong and logs you can audit.
One highest-leverage bottleneck, mapped. You leave with a prioritized plan and a concrete quote you own.
Start a conversation →One visible, production-grade improvement: shipped, measured, in your repo.
The full system: suite, gate, automation, runbook, owned by your team at handoff.
Measure, improve, publish: the system compounds instead of quietly decaying.
About a week, fixed price.
I point my evaluation and QA tooling at your live AI feature, map the highest-leverage failure surface — correctness, safety, the test and automation gaps — and hand you a prioritized plan, the findings that matter, and a real quote for the build you own. The $497 credits straight into the build if you continue. The build itself is quoted from what the audit finds, so you pay for your problem, not a package.
Your QA team almost certainly covers the deterministic surface. The gap is the LLM feature: no golden set, no judge, no gate. Prompt changes ship on vibes. I add the eval layer to your existing pipeline; your team owns it when I leave.
Yours. Your repo, your CI, your conventions. Every engagement ends with a runbook and a walkthrough. The deliverable is a system your team runs without me. There's no dependency on me.
Common needs have a fixed price — a $497 AI Quality Audit, packaged builds from $4,997, and monthly care from $299/mo. The entry audit and the monthly retainers have set prices; project work is quoted in writing after a short scoping call — fixed scope, never open-ended hours. The 15-minute intro call is free and ends with a concrete plan and a real number either way. Start with the audit and you get the quote for the larger build baked in.
Booking one engagement at a time (see the availability chip at the top of the page). Week 1 is scoping, so the calendar risk is low: you'll know exactly what you're getting before the build starts.
The audit is one week. Pilot-sized sprints run 2–3 weeks; full builds 4–8 depending on scope. Every engagement ships something visible inside the first two weeks. No long silent phases.
NDA-friendly. Access is scoped to the engagement, credentials stay in your systems, code stays in your repos, and nothing is retained after handoff. The data-handling one-pager is available before we start.
Often you should — and at handoff you will be. The difference is role: an in-house hire maintains the system day to day; I'm brought in to design it once, with the pattern from many of these builds already baked in — the eval battery, the failure modes, the gate. The build hours are the cheap part. The real cost of doing it in-house is the ongoing one: keeping evals honest, chasing model and API drift, and the regression nobody catches until a customer does. I stand that up so it holds, then hand your team the keys.
Everything lands in your repo and CI, with a runbook and a walkthrough — no dependency on me, nothing locked to my platform. If something drifts, the gate is built to catch it before it ships and the runbook says exactly what to do. Access, credentials, and a written handoff plan exist from day one, so the work outlives me being unavailable. The optional operate window is there if you want a hand, not because you'll need one.
Then you ship with evidence instead of hope. That's the win. The suite stays in CI catching the regression that would have landed six weeks from now.
Open to the right one. The "Hire me" toggle up top reframes this page for recruiters, and the résumé PDF is one click away.
For thirteen years my job was making sure software actually worked. The kind of QA where a missed bug costs real money. Then AI features started shipping on a demo and a prayer, and I watched good products break in ways a customer found first. So now I build both halves: the AI, and the proof it works.
— Jason Teixeira · founder, Sage Ideas
I run Sage Ideas, an AI-native studio, and I've spent years building the same thing at every layer: systems that refuse to say "done" without evidence.
That thesis shows up as an engineering OS with a hash-chained proof ledger, a QA platform where every score is computed from a real command, and agent pipelines with golden-trace evals wired into CI. The pattern is always the same: ship the feature, then ship the machinery that catches it when it regresses.
If your team is shipping LLM features on vibes (no eval suite, no regression gate, no idea whether last week's prompt change made things worse), that's exactly the gap I close.
| capability | typical AI builder | typical QA engineer | me |
|---|---|---|---|
| Ships LLM features (RAG, agents, automation) | ✓ | — | ✓ |
| Builds eval suites & LLM-as-judge gates | rare | rare | ✓ shipped |
| Regression discipline in CI (Playwright/Pytest) | — | ✓ | ✓ 13 yrs |
| AI-safety battery (injection · PII · toxicity) | — | — | ✓ 10 runners |
| Evidence you can audit (signed, reproducible) | — | rare | ✓ by design |
| Certified for it (ISTQB CT-AI + TAE) | — | rare | ✓ both |
Thirteen years across enterprise retail and fintech, plus an AI-native studio. Every role in the same discipline: automation that proves software works. Figures from prior roles are self-reported. Full résumé (PDF) ↓ · Company overview · See the proof →
longer-form writing lives at sageafterdark.com · free artifact: the LLM pre-launch eval checklist →
15 minutes. Bring the feature that scares you. Leave with a concrete eval-and-automation plan.
The engineer who ships the LLM feature and the harness that guards it. Let's talk about your team.
Send me your AI feature. I’ll run a free mini-eval against it and send you the findings: pass/fail, failure examples, and what I’d gate before your next release. No call required.
15 minutes, free. Bring the feature that scares you.
Week 1: we agree what "working" means and how it's measured.
Price attached to deliverables instead of open-ended hours. Either way, the plan is yours.
The hero film: who Jason is and the core promise — build the AI feature and the proof it works.
AI features & agents, RAG, automation, web apps, and the eval + QA layer that keeps them honest.
