LangSmith Tracking Setup

• Author: JeongGi Park

• Peer Review: MinJi Kang, Wooseok Jeong

• Proofread : Q0211

• This is a part of LangChain Open Tutorial

Overview

This tutorial covers how to set up and use LangSmith, a powerful platform for developing, monitoring, and testing LLM applications.LangSmith provides comprehensive tracking capabilities that are essential for understanding and optimizing your LLM applications.

LangSmith tracking helps you monitor:

• Token usage and associated costs

• Execution time and performance metrics

• Error rates and unexpected behaviors

• Agent interactions and chain operations

In this tutorial, we'll walk through the process of setting up LangSmith tracking and integrating it with your LangChain applications.

Table of Contents

• Overview

• Setting up a LangSmith trace

• Using LangSmith tracking

• Enable tracking in your Jupyter notebook or code

References

• OpenAI API Pricing

• Token Usage Guide

• LangChain Python API Reference

---

Setting up a LangSmith trace

LangSmith is a platform for developing, monitoring, and testing LLM applications. If you're starting a project or learning LangChain, LangSmith is a must-have to get set up and running.

Project-Level Tracking

At the project level, you can check execution counts, error rates, token usage, and billing information.

project-level-tracking

When you click on a project, all executed Runs appear.

project-level-tracking-detail

Detailed Step-by-Step Tracking for a Single Execution

detailed-step-by-step-tracking

After a single execution, it records not only the search results of retrieved documents but also detailed logs of GPT's input and output content. Therefore, it helps you determine whether to change the search algorithm or modify prompts after reviewing the searched content.

Moreover, at the top, it shows the time taken for a single Run (about 30 seconds) and tokens used (5,104), and when you hover over the tokens, it displays the billing amount.

Using LangSmith tracking

Using traces is very simple.

Get a LangSmith API Key

1. Go to https://smith.langchain.com/ and sign up.

2. After signing up, you will need to verify your email.

3. Click the left cog (Setting) - center "Personal" - "Create API Key" to get an API key.

get-api-key

Set environment variables is in the .env file.

Copy the contents of .env_sample and load it into your .env with the key you set.

from dotenv import load_dotenv

load_dotenv(override=True)

True

In Description, enter a description that makes sense to you and click the Create API Key button.

create-api-key

Copy the generated key and proceed to the next step.

(Caution!) Copy the generated key somewhere safe so that it doesn't leak.

copy-api-key

Setting the LangSmith key in .env

First, enter the key you received from LangSmith and your project information in the .env file.

• LANGCHAIN_TRACING_V2: Set to "true" to start tracking.

• LANGCHAIN_ENDPOINT: https://api.smith.langchain.com (Do not modify this value).

• LANGCHAIN_API_KEY: Enter the key you received in the previous step.

• LANGCHAIN_PROJECT: Specify a project name to group and trace all runs under that project group.

setting-api-key

Enable tracking in your Jupyter notebook or code

Enabling tracking is very simple. All you need to do is set an environment variable.

Copy the contents of .env_sample and load it into your .env with the key you set

%%capture --no-stderr
%pip install python-dotenv

from dotenv import load_dotenv

load_dotenv(override=True)

True

As long as your traces are enabled and your API key and project name are set correctly, tracking will work properly.

However, if you want to change the project name or change the tracking, you can do so with the code below.

import os

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGCHAIN_PROJECT"] = "<LangChain Project Name>"
os.environ["LANGCHAIN_API_KEY"] = "<LangChain API KEY>"