Unlocking the Black Box: Using LangSmith to Understand and Debug Your AI Agents

Building AI agents that can perform complex tasks is an exciting frontier, but it also comes with challenges. These systems, often involving multiple steps, tools, and intricate interactions, can be difficult to understand and debug. That’s where LangSmith steps in as an invaluable tool. Developed by the creators of LangChain, LangSmith provides the ability to monitor, trace, and debug your AI agents, offering a clear view of their behavior and helping you resolve issues effectively.

This is a summary of a section from a recent Ai+ Live Training session, “AI Agents: From LLMs to Multi-Agent Systems.” You can watch the full presentation and learn more about LangSmith by subscribing to Ai+ Training here!

Why Do We Need Tools Like LangSmith?

As AI agents get more complex, it’s easy to lose sight of what’s going on “under the hood”. Traditional debugging methods often fall short when dealing with these dynamic, multi-step processes. LangSmith bridges that gap, allowing developers to:

• Follow the breadcrumbs: See each step your agent takes, from initial input to final output. This includes tool calls, model responses, and all intermediate steps.
• Dive into the details: Examine the specific inputs, outputs, and parameters used at each step to understand why a certain decision was made or a particular output was generated.
• Pinpoint problems: Identify errors or unexpected behaviors by tracing the execution path, making it easier to resolve issues whether they stem from the model, a tool, or the overall logic.
• Evaluate performance: Assess the overall quality and efficacy of your agent’s output, and identify places for improvement.

Getting Started with LangSmith

Integrating LangSmith into your agentic system is straightforward:

1. Get Your API Key: Sign up for a LangChain account and get your LangSmith API key.
2. Enable Tracing: In your code, enable tracing by setting the appropriate environment variables. This usually involves providing your API key and setting a tracing parameter to True. You can also organize your traces using a project name.
3. Run Your Agent: Simply run your agent as you normally would. LangSmith will automatically capture the trace data.
4. View Your Traces: Open the LangSmith user interface in your browser to view your projects, including detailed step-by-step traces of each agent execution.

Real-World Example: Tracing a Financial Analysis Agent

Let’s explore how LangSmith can help you understand an AI agent designed for financial analysis. This agent, described in the workshop, is capable of:

• Retrieval Augmented Generation (RAG): Accessing a vector database to look up internal research on a company.
• Web Search: Gathering real-time data and information from the web.
• Code Execution: Generating and running code to create data visualizations.

Imagine we ask this agent: “What is the market cap and what are the buy/sell recommendations for Nvidia?”

With LangSmith, we can trace the execution path, including:

• The Initial Request: LangSmith shows the user’s initial query.
• Agent Reasoning: The agent uses the LLM to decide on the best course of action, which in this case is to call the tool to get the market cap and the buy/sell recommendations for Nvidia.
• Web Search: The agent uses Tavily to perform web searches. LangSmith displays the exact search queries used and the content that’s returned including the sources.
• LLM Response: You can then see the model’s output, after processing the information from the web.
• Completion: The agent concludes its process and presents the final response.

LangSmith’s detailed view of each step helps you analyze the flow, evaluate the quality of the results, and identify bottlenecks or errors.

LangSmith: Your Window Into AI Agents

LangSmith is an incredibly valuable tool for anyone building AI agents. By allowing you to see inside the agent and observe all the steps, tool calls, responses, and decision points of the execution, it empowers you to build more reliable and efficient agentic systems. It’s a must-have for debugging and evaluation. With LangSmith, you are no longer in the dark.