At a glance
The problem
I worked on this initiative as a Senior Product Manager through Tiger Analytics, embedded with Equifax’s data platform organization.
As AI agents spread across Equifax, individual teams built and maintained their own tracing. There was no shared or aggregate view of how those agents were performing or what they cost to run — just fragmented, team-by-team visibility. That mattered more here than it might elsewhere: Equifax handles sensitive consumer credit, employment, and identity data under real regulatory scrutiny, FCRA and CCPA among it. In an environment handling sensitive, regulated data, agent quality, safety, and cost visibility carried implications beyond product performance alone.
Early monitoring attempts were hard to sustain for a second reason: agent technology itself kept changing, so anything built against one team’s implementation went stale quickly. That shifted once teams building agents across the business converged on Google’s Agent Development Kit (ADK) as a shared standard — a change that happened on the organization’s own timeline, not something I drove. Once that foundation existed, we built our observability framework on it directly, using ADK and OpenTelemetry, rather than a bespoke layer chasing each team’s separate approach.
Conceptual flow, reconstructed for clarity — not a depiction of Equifax's internal architecture.
The approach
Two groups wanted visibility, for different reasons. Agent owners wanted to track their own usage, cost, and trace-level detail. AI governance wanted something else: aggregate quality and performance — coherence, groundedness, safety — across every onboarded agent, not just cost. Rather than build two disconnected tools, I led requirements with our VP of Product, a senior PM colleague, and Equifax’s global AI governance team to define a single evaluation foundation that could serve both from one data layer.
I partnered with our VP of Engineering to select DeepEval as the evaluation framework — an existing, purpose-built tool rather than something bespoke. DeepEval scores agent behavior using an LLM-as-a-judge approach across roughly twenty possible metrics, and its tracing layer is itself built on OpenTelemetry. Working with AI governance, we set around ten of those metrics as industry-standard defaults, available immediately; agent owners can define their own custom metrics and rubrics for anything beyond that.
Both views draw from one shared evaluation & observability foundation.
Decisions and trade-offs
- Built the framework on the shared ADK/OpenTelemetry foundation rather than a bespoke layer tracking each team separately.
- Chose DeepEval with Engineering rather than building an evaluation engine from scratch.
- Designed one shared data layer to serve two structurally different views, rather than two disconnected tools.
- Set a smaller, governance-aligned default metric set, with custom metrics available beyond it — trading some initial completeness for a system teams could actually adopt without being overwhelmed on day one.
The one technical boundary worth naming directly: I didn’t design the trace-collection and processing pipeline feeding the evaluation engine — that was our architects’ and engineering leadership’s work. My role was shaping the requirements and pressure-testing the proposed design against them: asking about latency, data structure, and whether it could actually deliver what governance and Finance needed.
Both feed configurable, per-metric thresholds — anything crossing one surfaces to the agent owner as an anomaly.
Execution
Agent owners can set a threshold on any metric — for instance, treating an answer-relevancy score below a certain level as worth flagging immediately. Anything crossing that line surfaces as an anomaly, rather than waiting to be noticed in a dashboard on a day nobody’s looking. That loop, from measurement to actionable alert, is what makes the two views useful day to day rather than just informative.
Outcome
More than 40 agents are onboarded and evaluated in production today, with real-time cost and quality visibility that governance and Finance didn’t have before. The initiative is still early in its adoption curve.
Learning
The clearest lesson: shared technical standards create leverage for platform products. While agent implementations were fragmented across teams, a common observability layer would have meant continually adapting to each team’s own instrumentation rather than building on solid ground. The organization’s convergence on Google’s Agent Development Kit (ADK) wasn’t something I drove — but once that shared standard existed, we could build directly on it, which freed the product work to focus on the higher-order questions: what to evaluate, for whom, and how evaluation becomes actionable rather than just visible.