Senior PM · 15 Years · CFA · Building multi-agent AI on Google Cloud

Enterprise AI has a proof problem.
I build the proof.

Pilots stall because nobody trusts the evaluations. I'm a senior PM (Societe Generale, CFA) who has spent two years building that missing proof discipline: eval gates that catch what demos hide, regression suites that make model upgrades an afternoon instead of a quarter, and agents engineered to run unattended — on Google ADK, Gemini, and BigQuery, shipped end to end.

The problem map → See my work Résumé ↗ LinkedIn ↗
PM Philosophy
Ground every claim, then measure it

AI that can't cite its source is a liability. I build with provenance first, then put an eval suite around it so quality is a number, not a vibe.

Determinism where correctness matters

Workflow when the order is known; agent when the order depends on findings. LLMs only where judgment is genuinely required — the cheapest, most reliable parts of a system have no model at all.

PMs who build earn trust

I write code, ship prototypes, and stand up multi-agent pipelines. Engineers respect PMs who understand what they're asking for.

New · Newsletter No Black Boxes — building AI you can actually trust, in public.
Subscribe ↗
1,900+
Filings scored across the Nasdaq-100
700+
Grounded, cited sources in production RAG
33→90%
Retrieval precision, via eval-driven tuning
56×
Warm-search latency cut (11.2s → 0.2s)
The Problem Map

I build against what stalls enterprise AI

In 2026 surveys, about seven in ten enterprises are piloting AI agents. Roughly one in ten has put a single agent into live production. Proof is what separates the two groups — four named gaps, and what I've shipped against each.

01The Evaluation Gap
The pain

Teams can't verify their AI works, so it stalls before production. In surveys, half of enterprises shipped something that passed internal evals and then failed a customer — because a passing eval is not the same as a working product.

What I've shipped against it

Binary PASS/FAIL evals against human labels — a 4.9/5 self-grade was hiding a 42% failure rate. Judges structurally separated from generators. An auditor that caught its own passing pipeline lying. And a documented NO-GO on an A/B the judge had passed clean — because the real failure was one the eval didn't measure yet.

Lived in: Pain Signal Intelligence · RedInk

02The Model Upgrade Tax
The pain

Every model release invalidates every customer's validation — thresholds, judge calibrations, and golden sets are properties of a specific model, and that model has a retirement date. Without eval infrastructure, the choice is pin-and-break or upgrade blind.

What I've shipped against it

Version the judgment, not the model. A formula version on every score row so retuning never rewrites history, and a release gate that went 5/13 → 12/13 golden cases across four rubric versions with the model held constant. When a platform release deprecated my pipeline's agent family mid-build, the migration took an afternoon — because the decision records were separate from the primitives.

Lived in: Pain Signal Intelligence · RedInk

03The Agent Operations Gap
The pain

Evaluation asks whether the output is good. Operations asks whether the agent stays alive, on budget, and observable when nobody is watching. Gartner predicts 40% of enterprises will demote or decommission autonomous agents by 2027 — over governance gaps found only after production incidents.

What I've shipped against it

Checkpointing, idempotency, and liveness — each learned from a real failure: a silent seven-day outage (now silence itself is the alarm), a day-one crash mid-run (state persists before the LLM tail, so a crash resumes instead of restarting), and a 21.4-hour hang on a single filing (fixed with an out-of-band 240-second deadline).

Lived in: two unattended agents in production · see Now Building

04The Activation Gap
The pain

Seats get bought; usage stays shallow. Enterprise AI adoption fails on voice and trust before it ever fails on capability — and renewal risk hides in idle seats, not in headline penetration numbers.

What I've shipped against it

At Societe Generale, a working summary agent was rejected by its own first user for sounding fluent but generic. Few-shot prompting on the user's own previously accepted artifacts fixed it: the agent learned the house voice from work the user had already blessed. Adoption gets fixed one skeptical user at a time.

Lived in: Societe Generale · production AI in a regulated bank

All four gaps trace back to missing proof. I've spent two years building that discipline, in public, with the receipts on this page.
Featured Project · Live on Google Cloud

Pain Signal Intelligence

A multi-agent system on Google's Agent Development Kit that mines real enterprise-AI pain from five live public sources, scores it deterministically, and has a cross-family judge verify every claim before it writes a PRD — gated on evals, with the receipts in BigQuery.

Pain Signal Intelligence overview: deterministic pipeline (collectors, classifier, scorer, synthesizer, cross-family Claude judge, PRD), the frequency × intensity × addressability² scoring formula, and the ADK 2.2 / Gemini / Claude / BigQuery / Cloud Run stack.
Cross-model judge verdict · per-criterion · before any PRD
● GROUNDED Every pain traces to a real scored row; every cited number matches the data. No fabrication, no inflation.
● ACTIONABLE Each pain's rating is justified with a specific product reason an engineer could push back on.
● RETRY FIRES When the writer invented a pain, grounded failed and the gate forced a rewrite — which has happened on a live run.
Live · Multi-agent · Google ADK 2.2
From raw signal to a judge-verified PRD

ProblemTeams decide what to build from anecdotes — so the loudest pain wins, not the most buildable one.

InsightA pain you can't act on is a backlog, not a signal. Score it frequency × intensity × addressability² in deterministic Python — a multiplicative veto, so one near-zero pillar kills the score instead of being averaged away. Addressability carries the highest exponent because it decides whether a pain belongs on the roadmap at all.

TradeoffA fixed-order ADK graph, not an autonomous LLM coordinator. The pipeline has exactly one correct order, so determinism owns the control flow and every stage persists to BigQuery — a crash resumes only the stage that failed. The model is reserved for judgment, never routing.

ResultA cross-family Claude judge verifies grounding before any PRD is written. I planted two lies in a synthesis — a count inflated from 12 to 80, and a fabricated "users want dark mode" with no signal behind it — and Claude failed both on the first run with the row-level evidence, then the gate forced a rewrite. A same-family control flagged nothing.

The pipeline — a deterministic graph, LLMs only where judgment is needed
5 sources → classify · Flash → score · Python → BigQuery → synthesize · Pro → judge · Claude → PRD
Google ADK 2.2 Gemini 2.5 Flash + Pro · Vertex AI Claude Sonnet (Judge) BigQuery Cloud Run · Scheduler · Trace
Open app ↗ GitHub ↗
5
Live sources
GitHub · HN · Stack Overflow · Discourse · G2
Cross-family
LLM judge
Caught a planted hallucination 1st try
0
Ungrounded PRDs
Retry gate blocks unverified output
Monthly
On GCP
Cloud Run · Cloud Trace · BigQuery
Also Featured · Live

RedInk

Eval-first financial anomaly detection across the Nasdaq-100. Three independent signals — statistical outliers, earnings-quality red flags, and narrative spin — roll into one conviction score, and an LLM-as-judge suite gates every explanation before an analyst sees it. The signal it hunts is the gap between the numbers and the story management tells about them.

RedInk architecture: eval-gated pipeline from EDGAR ingest to anomaly scorer (MCD, Beneish M) to narrative divergence (Claude reads the MD&A) to conviction synthesis to a binary eval gate (Layer 0 to 1 to 2) to the analyst UI; the conviction score is statistical 40 + earnings quality 30 + narrative 30, split into ALERT / FLAG / WATCH; Claude Sonnet 4.6 / BigQuery / Python / SEC EDGAR / Cloud Run / scikit-learn stack.
Narrative-divergence labels · cited MD&A
● CONTRADICTS Revenue +12% YoY, yet MD&A omits the margin compression the cash-flow statement implies.
● NEUTRAL Inventory build flagged, but framed as seasonal — no clear stance for or against the anomaly.
● CORROBORATES Management explicitly attributes the guidance cut to the same demand softness the data shows.
Live · Eval-first
Narrative-divergence detection

ProblemAn analyst covering 90+ names can't read every 10-Q line by line — and the filings that matter most are the ones where the numbers turn but the narrative doesn't.

InsightThe signal worth an analyst's time is the gap between the numbers and the story management tells about them. When three independent signals agree, a false positive becomes unlikely.

TradeoffGraded explanations on a hard binary PASS/FAIL, not a 1–5 scale — a 4.9/5 average was hiding a 42% fail rate. Failure visibility over a flattering number.

Result32 ALERT-tier filings surfaced from 1,352 scored — each one a cited MD&A passage behind an LLM-as-judge gate that lifted the champion prompt to 4.72/5 with 100% label match.

The architecture — built across 4 repos
qqq-anomaly-lab → scoring-pipeline → qqq-eval-suite → redink-ui
Python Claude Sonnet 4.6 LLM-as-judge BigQuery GCP Telemetry
Open app ↗
How a conviction score is built · 0–100
40
Statistical anomaly
Robust multivariate outlier vs. the company's own history — MCD / Mahalanobis on median & IQR, so one-off spikes don't dominate.
30
Earnings quality
Beneish M-Score across 8 accrual & margin components — the classic accounting-manipulation screen.
30
Narrative transparency
An LLM reads the MD&A and scores whether management names the signal — or quietly steps around it.
Three independent methods on three different data sources roll into one score, then split into ALERT · FLAG · WATCH. The highest-conviction flags are where all three fire together — weak numbers, a manipulation signal, and a management narrative that looks the other way.
1,352
Filings scored
94 Nasdaq-100 companies
4.72/5
Champion eval score
+0.62 vs v1 · 100% label match
93.2%
Judge–human agreement
LLM judge vs. hand-labeled golden set
Next
Earnings-call agent
Transcript vs. filing divergence
More Work

Also Building

Each one a rung on the same ladder — every link below is live.

PM Confessional architecture: 298 Lenny's Podcast episodes chunked to 800 words, a Claude extractor that keeps only first-person confessions, a Claude audit gate keeping 701 of 776, pgvector 768-dim HNSW cosine retrieval, then a confidence router on top-1 cosine that routes to the grounded Decision Coach; the router skips the rerank above 0.70 (~200 ms, no LLM cost), uses Gemini Flash Lite between 0.45 and 0.70, and refuses below 0.55; Claude Sonnet 4.6 / Gemini Embed / pgvector / PostgreSQL / Cloud Run / TypeScript stack.
Grounded RAG · Live on Cloud Run
PM Confessional

A grounded "Decision Coach" over 700+ verbatim PM regrets from Lenny's Podcast. Precision 33%→90% via prompt iteration; confidence threshold tuned to refuse rather than guess (top-1 relevance 30%→80%).

RAGEvals
Open app ↗
0→1 Web App · Live on Cloud Run
Strategic Fit Canvas

Résumé + JD → an AI-scored candidate-fit radar, with auth, batch processing, an analytics dashboard, and a feedback loop. Kept deliberately raw — the "before" of the arc.

GCPFull-stack
Open app ↗
Open Source · Python · MCP
Product Management OS

An open-source AI system that reads your goals + backlog and surfaces what to work on next. The meta-tool that runs this whole portfolio — building the system that builds the products.

MCPClaude API
GitHub ↗
The Arc

A deliberate progression

Each product taught the next. Read top to bottom: the capability ladder from shipping to systems to self-verifying agents.

0→1 · Raw
Strategic Fit Canvas
Shipping a real web app end to end — auth, uploads, deploy.
Grounding + Evals
PM Confessional
RAG with provenance; precision auditing; threshold tuning; a 56× latency fix.
Eval-first Systems
RedInk
Multi-repo architecture; LLM-judge eval suite; narrative divergence; telemetry.
Multi-agent · Self-verifying
Pain Signal IntelligenceFlagship
Fixed-order ADK graph; deterministic scoring with an addressability veto; a cross-family judge that gates the output — monthly on Google Cloud.
Unattended · Long-running
Two agents, no supervisor
A daily Cloud Run job watching hiring signals and a weekly agent reconciling five repos against last week's report — checkpointing, idempotency, and silence-is-alarm reporting, running without me.
Meta / Infra
Product Management OS
Building the system that builds the products — and learning in public.

Learned to ship → to measure → to build systems → to make them agentic, self-verifying, and unattended — in public.

Now Building · The Agent Frontier

The three problems everyone building agents is circling

Memory and context, continuous learning, and planning over long horizons — the open problems of the agent era. A position on each, earned from shipped systems, and what I'm building against it right now.

Memory & context management

I split memory by grain: session state for the run, an append-only history table for trends, and a persisted registry as institutional memory — deciding per stage whether context gets injected or fetched. And I've hit the classic failure: storing volatile upstream state as a durable fact.

In progress

An unattended weekly agent that reads git facts across five repos, compares against its own prior report to compute deltas, and reports drift — its memory-access failures logged and carried as open issues. Plus conversation memory in a routed LangGraph chat agent.

Continuous learning

Version the judgment, not the model. Golden sets, per-attempt per-criterion judge verdicts logged to BigQuery, and a formula version on every score row so retuning never rewrites history. The release gate improved 5/13 → 12/13 with the model held constant — the learning lives in the rubric, so it survives model churn.

In progress

An error-discovery loop for content: an agent drafts, a human edits, and the deltas between draft and accepted version become the next round's few-shot examples — the same accepted-artifact pattern that fixed adoption at the bank.

Planning & long horizons

Workflow when the order is known; agent when the order depends on findings. My flagship is a fixed-order graph by deliberate, ADR-defended choice — its pipeline has exactly one correct order. And long-running means checkpointing, idempotency, and liveness, each learned from a real production failure.

In progress

Crossing from workflow to agent: a coordinator that owns data-dependent control flow — "this pain looks big → spawn a deeper sub-investigation; this week's signal is weak → skip the PRD." In parallel: routed chat agents in LangGraph, deciding who owns the tool decision — a classifier route in code, or the model's own tool calls.

Platform depth: Google ADK 2.2 in production — Sequential, Parallel and Loop agents, with a cross-family Claude judge reached over the LiteLLM bridge · LangGraph for routed conversational agents · deterministic Python wherever correctness has to be provable.

The AI-PM Bar

What each product proves

The capabilities a senior AI PM is expected to own — and where each shipped product demonstrates them.

Product Grounding / RAG Evals Agentic Telemetry Shipped live
Pain Signal Intelligence
RedInk
PM Confessional
Strategic Fit Canvas
Product Management OS
Demonstrated Partial / in progress Not the focus
Newsletter · On LinkedIn

No Black Boxes

Building AI you can actually trust — grounding, evals, and cross-model verification, with the receipts. The how and the why behind every product on this page, written as I build it.

The decisions, not just the demos — ADRs, tradeoffs, and the calls that would survive a design review.
What broke and how I caught it — the planted hallucination, the routing bug, the 403 mid-run.
Evals over vibes — turning "it feels better" into a number you can defend.
Subscribe to No Black Boxes ↗
Published on LinkedIn · new issues as the products ship
What's inside · recent threads
My cross-family LLM judge caught a planted hallucination on the first try
Pain Signal Intelligence · the same-family control caught nothing
The cheapest agent in my pipeline has no model at all
Determinism where correctness matters
A 4.9/5 became 42% FAIL the moment I switched to binary
RedInk · why numerical rubrics hide the failures you need
I was calling an LLM because I could — not because every query needed it
PM Confessional · a 56× latency cut by judgment, not a faster model
Build in Public

I ship the lessons, not just the products

A few from the feed — each one a real decision from building these tools, in the open.

Cross-model evals
"My cross-family judge caught a planted hallucination on the first try."

An LLM judging its own family shares its blind spots — correlated error. So Pain Signal Intelligence judges Gemini's synthesis with Claude, from a different model family. I planted a fabricated pain to test it; Claude failed it on grounded and forced a rewrite. A same-family control run flagged nothing.

Pain Signal Intelligence · No Black Boxes
Architecture
"The cheapest agent in my pipeline has no model at all."

Collectors hit source APIs that already return structured data, and scoring is a fixed formula. Neither needs inference. LLMs only where judgment is genuinely required — classify with Flash, synthesize with Pro, judge with Claude — and deterministic Python everywhere correctness has to be provable.

Pain Signal Intelligence · No Black Boxes
Evals
"A 4.9/5 became 42% FAIL the moment I switched to binary."

Numerical rubrics smooth over the failures you most need to see. Moving RedInk's eval suite to binary PASS/FAIL (per Hamel Husain's method) surfaced a routing bug — the model pointing analysts to the wrong filing artifact — and cut the failure rate 41% in one iteration, with no prompt change.

RedInk · 3,585 impressions
Latency
"I was calling an LLM because I could — not because every query needed it."

PM Confessional's search took 11 seconds. Dropped it to under 0.1s by skipping the expensive rerank when internal confidence is already high, falling back to Gemini Flash-Lite when it isn't, and caching embeddings. The fix was judgment about when to spend the call, not a faster model.

PM Confessional · 1,084 impressions
Subscribe to No Black Boxes ↗ Read all posts on LinkedIn ↗
Product Philosophy

How I Think

The principles that shape every product decision I make.

01
User problems over feature requests

Features are solutions in search of problems. I spend more time understanding why users behave the way they do than cataloguing what they ask for. The best product insights live one question deeper — "why does that matter to you?" is where the real brief is.

02
Quality is a number, not a vibe

For AI products especially, "it feels better" isn't shippable. I put an eval harness around the thing — precision, TPR/TNR, an LLM-as-judge with a golden set, a cross-family judge for the calls that matter — so I can tell whether a prompt change helped, regressed, or just moved the demo. Metrics inform; judgment decides; but I refuse to fly blind.

03
Determinism where correctness matters

An LLM is the most expensive, least predictable component in any system — so I reach for it last, not first. Fixed formulas, fixed order, and plain code own the parts that have to be provable; the model is reserved for genuine judgment. Scoring a pain as frequency × intensity × addressability² in Python — not asking a model to "rate it" — is exactly this call.

Skills

What I Bring

Product
Product Strategy
Prioritisation
User Research
PRDs & Specs
Go-to-Market
AI & Evals
Multi-agent Systems
Evals & LLM-as-judge
Agent Memory & Context
Continuous Learning · Golden Sets
Grounding / RAG
Build & Cloud
Google Cloud (GCP)
Google ADK · Gemini
LangGraph
Claude API · MCP
BigQuery · Python
Get in touch

Let's build something worth trusting.

I'm looking for senior AI PM roles that own the hard problems on this page — evaluation, model churn, agent reliability, adoption. If that's the roadmap you're hiring for, let's talk.

Email me ↗ Résumé ↗ LinkedIn ↗