Challenge
MasterClass Executive uses four collaborating AI agents to personalize learning across sessions that can last hours and learning journeys that span months. To understand where the experience falls short, the team needs to see how those agents plan, teach, remember, and interact. Prelaunch evaluations alone couldn’t answer whether the system was working as intended for learners in production.
Solution
MasterClass uses Weights & Biases Weave on CoreWeave Forge to trace agent inputs, outputs, tool use, and interactions across its teaching system. Building on those traces, the team created an agent-powered search skill that investigates learner experiences and finds recurring failure patterns. Those findings become datasets for testing fixes and feed monitoring workflows that flag issues for the team to review.
Benefits
- Clearer insight into learner problems: The team can investigate what happened across agents and identify possible causes of confusing or ineffective teaching.
- More efficient testing and iteration: Trace searches uncover similar failures and collect the agent inputs needed to replay scenarios, test fixes, and improve the experience.
- More proactive issue detection: Automated monitoring surfaces trends, while a broader workflow connects learner feedback with traces, GitHub, and Jira to flag issues that need attention.
