This course is for engineers who can already call an LLM API, build a basic retrieval-augmented pipeline, and wire together a simple tool-calling agent, and who now need to answer a harder question: how do you know your agent is actually working once it leaves your laptop? It moves you from treating an agent as a black box you test by eyeballing outputs, to treating it as a system you can trace, measure, and hold to a defined standard of reliability in production.

The course pairs a mandatory external course with original material built specifically to extend past where that source stops. Section 3 is built entirely around LangChain Academy's free, self-paced "Foundation: Introduction to Agent Observability & Evaluations" course, which teaches tracing, evaluation, prompt engineering, human feedback, and production monitoring on the LangSmith platform. The sections around it, all original, deliberately avoid repeating what that course already covers well. Instead they widen the lens: comparing LangSmith against other observability platforms on the dimensions that actually drive cost and lock-in, and teaching OpenTelemetry, the open, vendor-neutral standard for tracing AI agents that no single vendor's course would teach you.

The first half of the course builds the conceptual and platform-specific foundation. It starts with why LLM and agent systems fail in ways that traditional software does not, non-deterministically, gradually, and across long chains of calls that make it hard to see where a failure actually started, and builds the vocabulary of traces, spans, metrics, and logs you need before any tracing tool makes sense. From there it works through LangSmith Academy's full curriculum on tracing, evaluation, and production monitoring, and adds a short synthesis unit that maps those five modules back onto the vocabulary you just learned.

The second half moves outward from any single vendor. It compares LangSmith against Logfire, Arize AX Pro, and Langfuse on billing models and the incentives those models create, then sends you to research each platform's self-hosting and data ownership posture yourself, since no single source covers that comparison. It closes with a full treatment of OpenTelemetry for AI agents: the three signal types, the GenAI semantic conventions that are stabilizing across the industry, the instrumentation libraries you would actually choose between, and the collector architecture and sampling strategy decisions that come up once you are running this in production.

Prerequisites

This course assumes you have completed the prerequisite modules covering LLM APIs, prompt engineering, retrieval-augmented generation, and basic agent construction, or have equivalent hands-on experience. It does not assume any prior exposure to tracing or observability tooling, and it does not assume you have used the Model Context Protocol before the capstone.

Course structure

The course runs across six sections: a short welcome, two original units on foundations, the mandatory LangSmith Academy course plus a synthesis unit, two original units comparing observability platforms, four original units on OpenTelemetry, and a two-day capstone mission that has you build two MCP servers and a LangGraph agent, then instrument the whole system end to end. Outside of the mission and the LangSmith course itself, most sections take under two hours to work through. A learner moving at a steady pace should expect roughly a day and a half of instructional content plus the two-day mission. The course has no fixed deadline; work through it at whatever pace fits your schedule, and the mission at the end is where everything before it gets put into practice.