Sheldon Gomes — Senior PM building multi-agent, eval-first AI products

Featured Project · Live on Google Cloud

Pain Signal Intelligence

A multi-agent system on Google's Agent Development Kit that mines real enterprise-AI pain from five live public sources, scores it deterministically, and has a cross-family judge verify every claim before it writes a PRD — gated on evals, with the receipts in BigQuery.

Cross-model judge verdict · per-criterion · before any PRD

● GROUNDED Every pain traces to a real scored row; every cited number matches the data. No fabrication, no inflation.

● ACTIONABLE Each pain's rating is justified with a specific product reason — not boilerplate, not vibes.

● RETRY FIRES When the writer invented a pain, grounded failed and the gate forced a rewrite — which has happened on a live run.

Live · Multi-agent · Google ADK 2.2

From raw signal to a judge-verified PRD

ProblemTeams decide what to build from anecdotes — so the loudest pain wins, not the most buildable one.

InsightA pain you can't act on is a backlog, not a signal. Score it frequency × intensity × addressability² in deterministic Python — a multiplicative veto, so one near-zero pillar kills the score instead of being averaged away. Addressability carries the highest exponent because it decides whether a pain belongs on the roadmap at all.

TradeoffA fixed-order ADK graph, not an autonomous LLM coordinator. The pipeline has exactly one correct order, so determinism owns the control flow and every stage persists to BigQuery — a crash resumes only the stage that failed. The model is reserved for judgment, never routing.

ResultA cross-family Claude judge verifies grounding before any PRD is written. I planted two lies in a synthesis — a count inflated from 12 to 80, and a fabricated "users want dark mode" with no signal behind it — and Claude failed both on the first run with the row-level evidence, then the gate forced a rewrite. A same-family control flagged nothing.

The pipeline — a deterministic graph, LLMs only where judgment is needed

5 sources → classify · Flash → score · Python → BigQuery → synthesize · Pro → judge · Claude → PRD

Google ADK 2.2 Gemini 2.5 Flash + Pro · Vertex AI Claude Sonnet (Judge) BigQuery Cloud Run · Scheduler · Trace

Open app ↗ GitHub ↗

5

Live sources

GitHub · HN · Stack Overflow · Discourse · G2

Cross-family

LLM judge

Caught a planted hallucination 1st try

0

Ungrounded PRDs

Retry gate blocks unverified output

Monthly

On GCP

Cloud Run · Cloud Trace · BigQuery

Also Featured · Live

RedInk

Eval-first financial anomaly detection across the Nasdaq-100. Three independent signals — statistical outliers, earnings-quality red flags, and narrative spin — roll into one conviction score, and an LLM-as-judge suite gates every explanation before an analyst sees it. The signal isn't the anomaly; it's the gap between the numbers and the story management tells about them.

Narrative-divergence labels · cited MD&A

● CONTRADICTS Revenue +12% YoY, yet MD&A omits the margin compression the cash-flow statement implies.

● NEUTRAL Inventory build flagged, but framed as seasonal — no clear stance for or against the anomaly.

● CORROBORATES Management explicitly attributes the guidance cut to the same demand softness the data shows.

Live · Eval-first

Narrative-divergence detection

ProblemAn analyst covering 90+ names can't read every 10-Q line by line — and the filings that matter most are the ones where the numbers turn but the narrative doesn't.

InsightThe signal isn't the anomaly — it's the gap between the numbers and the story management tells about them. When three independent signals agree, a false positive becomes unlikely.

TradeoffGraded explanations on a hard binary PASS/FAIL, not a 1–5 scale — a 4.9/5 average was hiding a 42% fail rate. Failure visibility over a flattering number.

Result32 ALERT-tier filings surfaced from 1,352 scored — each one a cited MD&A passage behind an LLM-as-judge gate that lifted the champion prompt to 4.72/5 with 100% label match.

The architecture — built across 4 repos

qqq-anomaly-lab → scoring-pipeline → qqq-eval-suite → redink-ui

Python Claude Sonnet 4.6 LLM-as-judge BigQuery GCP Telemetry

Open app ↗

How a conviction score is built · 0–100

40

Statistical anomaly

Robust multivariate outlier vs. the company's own history — MCD / Mahalanobis on median & IQR, so one-off spikes don't dominate.

30

Earnings quality

Beneish M-Score across 8 accrual & margin components — the classic accounting-manipulation screen.

30

Narrative transparency

An LLM reads the MD&A and scores whether management names the signal — or quietly steps around it.

Three independent methods on three different data sources roll into one score, then split into ALERT · FLAG · WATCH. The highest-conviction flags are where all three fire together — weak numbers, a manipulation signal, and a management narrative that looks the other way.

1,352

Filings scored

94 Nasdaq-100 companies

4.72/5

Champion eval score

+0.62 vs v1 · 100% label match

93.2%

Judge–human agreement

LLM judge vs. hand-labeled golden set

Transcript vs. filing divergence

More Work

Also Building

Each one a rung on the same ladder — every link below is live.

Grounded RAG · Live on Cloud Run

PM Confessional

A grounded "Decision Coach" over 700+ verbatim PM regrets from Lenny's Podcast. Precision 33%→90% via prompt iteration; confidence threshold tuned to refuse rather than guess (top-1 relevance 30%→80%).

0→1 Web App · Live on Cloud Run

Strategic Fit Canvas

Résumé + JD → an AI-scored candidate-fit radar, with auth, batch processing, an analytics dashboard, and a feedback loop. Kept deliberately raw — the "before" of the arc.

Open Source · Python · MCP

Product Management OS

An open-source AI system that reads your goals + backlog and surfaces what to work on next. The meta-tool that runs this whole portfolio — building the system that builds the products.

The Arc

A deliberate progression

Each product taught the next. Read top to bottom: the capability ladder from shipping to systems to self-verifying agents.

0→1 · Raw

Strategic Fit Canvas

Shipping a real web app end to end — auth, uploads, deploy.

Grounding + Evals

PM Confessional

RAG with provenance; precision auditing; threshold tuning; a 56× latency fix.

Eval-first Systems

RedInk

Multi-repo architecture; LLM-judge eval suite; narrative divergence; telemetry.

Multi-agent · Self-verifying

Pain Signal IntelligenceFlagship

Fixed-order ADK graph; deterministic scoring with an addressability veto; a cross-family judge that gates the output — monthly on Google Cloud.

Meta / Infra

Product Management OS

Building the system that builds the products — and learning in public.

Learned to ship → to measure → to build systems → to make them agentic and self-verifying — in public.

The AI-PM Bar

What each product proves

The capabilities a senior AI PM is expected to own — and where each shipped product demonstrates them.

Product	Grounding / RAG	Evals	Agentic	Telemetry	Shipped live
Pain Signal Intelligence
RedInk
PM Confessional
Strategic Fit Canvas
Product Management OS

Demonstrated Partial / in progress Not the focus

Build in Public

I ship the lessons, not just the products

A few from the feed — each one a real decision from building these tools, in the open.

Cross-model evals

"My cross-family judge caught a planted hallucination on the first try."

An LLM judging its own family shares its blind spots — correlated error. So Pain Signal Intelligence judges Gemini's synthesis with Claude, from a different model family. I planted a fabricated pain to test it; Claude failed it on grounded and forced a rewrite. A same-family control run flagged nothing.

Pain Signal Intelligence · No Black Boxes

Architecture

"The cheapest agent in my pipeline has no model at all."

Collectors hit source APIs that already return structured data, and scoring is a fixed formula. Neither needs inference. LLMs only where judgment is genuinely required — classify with Flash, synthesize with Pro, judge with Claude — and deterministic Python everywhere correctness has to be provable.

Pain Signal Intelligence · No Black Boxes

Evals

"A 4.9/5 became 42% FAIL the moment I switched to binary."

Numerical rubrics smooth over the failures you most need to see. Moving RedInk's eval suite to binary PASS/FAIL (per Hamel Husain's method) surfaced a routing bug — the model pointing analysts to the wrong filing artifact — and cut the failure rate 41% in one iteration, with no prompt change.

RedInk · 3,585 impressions

Latency

"I was calling an LLM because I could — not because every query needed it."

PM Confessional's search took 11 seconds. Dropped it to under 0.1s by skipping the expensive rerank when internal confidence is already high, falling back to Gemini Flash-Lite when it isn't, and caching embeddings. The fix was judgment about when to spend the call, not a faster model.

PM Confessional · 1,084 impressions

Subscribe to No Black Boxes ↗ Read all posts on LinkedIn ↗

Product Philosophy

How I Think

The principles that shape every product decision I make.

01

User problems over feature requests

Features are solutions in search of problems. I spend more time understanding why users behave the way they do than cataloguing what they ask for. The best product insights live one question deeper — "why does that matter to you?" is where the real brief is.

02

Quality is a number, not a vibe

For AI products especially, "it feels better" isn't shippable. I put an eval harness around the thing — precision, TPR/TNR, an LLM-as-judge with a golden set, a cross-family judge for the calls that matter — so I can tell whether a prompt change helped, regressed, or just moved the demo. Metrics inform; judgment decides; but I refuse to fly blind.

03

Determinism where correctness matters

An LLM is the most expensive, least predictable component in any system — so I reach for it last, not first. Fixed formulas, fixed order, and plain code own the parts that have to be provable; the model is reserved for genuine judgment. Scoring a pain as frequency × intensity × addressability² in Python — not asking a model to "rate it" — is exactly this call.

Skills

What I Bring

Product

Product Strategy

Prioritisation

User Research

PRDs & Specs

Go-to-Market

AI & Evals

Multi-agent Systems

Evals & LLM-as-judge

Grounding / RAG

Prompt Engineering

Build & Cloud

Google Cloud (GCP)

Google ADK · Gemini

Claude API · MCP

BigQuery · Python

Get in touch

Let's build something worth trusting.

I'm looking for senior AI PM roles where grounding, evals, and shipping matter. If that's the bar you're hiring for, let's talk.

Email me ↗ Résumé ↗ LinkedIn ↗

From user problem to shipped product.

Pain Signal Intelligence

RedInk

Also Building

A deliberate progression

What each product proves

No Black Boxes

I ship the lessons, not just the products

How I Think

What I Bring

Let's build something worth trusting.

From user problem
to shipped product.