Caching

Overview

LangChain provides optional caching layer for LLMs.

This is useful for two reasons:

  • When requesting the same completions multiple times, it can reduce the number of API calls to the LLM provider and thus save costs.

  • By reduing the number of API calls to the LLM provider, it can improve the running time of the application.

In this tutorial, we will use gpt-4o-mini OpenAI API and utilize two kinds of cache, InMemoryCache and SQLiteCache. At end of each section we will compare wall times between before and after caching.

Table of Contents

References

Environment Setup

Set up the environment. You may refer to Environment Setup for more details.

[Note]

  • langchain-opentutorial is a package that provides a set of easy-to-use environment setup, useful functions and utilities for tutorials.

  • You can checkout the langchain-opentutorial for more details.

%%capture --no-stderr
%pip install langchain-opentutorial
# Install required packages
from langchain_opentutorial import package

package.install(
    [
        "langsmith",
        "langchain",
        "langchain_core",
        "langchain_community",
        "langchain_openai",
        # "vllm", # this is for optional section
    ],
    verbose=False,
    upgrade=False,
)
# Set environment variables
from langchain_opentutorial import set_env

set_env(
    {
        "OPENAI_API_KEY": "You OpenAI API KEY",
        "LANGCHAIN_API_KEY": "LangChain API KEY",
        "LANGCHAIN_TRACING_V2": "true",
        "LANGCHAIN_ENDPOINT": "https://api.smith.langchain.com",
        "LANGCHAIN_PROJECT": "Caching",
    }
)
Environment variables have been set successfully.
# Alternatively, one can set environmental variables with load_dotenv
from dotenv import load_dotenv


load_dotenv(override=True)
False
from langchain_openai import ChatOpenAI
from langchain_core.prompts import PromptTemplate

# Create model
llm = ChatOpenAI(model_name="gpt-4o-mini")

# Generate prompt
prompt = PromptTemplate.from_template(
    "Sumarize about the {country} in about 200 characters"
)

# Create chain
chain = prompt | llm
%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea, located on the Korean Peninsula, is known for its rich culture, advanced technology, and vibrant economy. It features bustling cities like Seoul, renowned cuisine, and historic landmarks.
    CPU times: total: 93.8 ms
    Wall time: 1.54 s

InMemoryCache

First, cache the answer to the same question using InMemoryCache.

from langchain_core.globals import set_llm_cache
from langchain_core.caches import InMemoryCache

# Set InMemoryCache
set_llm_cache(InMemoryCache())
%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea is a technologically advanced country known for its fast-paced lifestyle, vibrant culture, and delicious cuisine. It is a leader in industries such as electronics, automotive, and entertainment. The country also has a rich history and beautiful landscapes, making it a popular destination for tourists.
    CPU times: total: 0 ns
    Wall time: 996 ms

Now we invoke the chain with the same question.

%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea is a technologically advanced country known for its fast-paced lifestyle, vibrant culture, and delicious cuisine. It is a leader in industries such as electronics, automotive, and entertainment. The country also has a rich history and beautiful landscapes, making it a popular destination for tourists.
    CPU times: total: 0 ns
    Wall time: 3 ms

Note that if we set InMemoryCache again, the cache will be lost and the wall time will increase.

set_llm_cache(InMemoryCache())
%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea is a tech-savvy, modern country known for its vibrant culture, delicious cuisine, and booming economy. It is a highly developed nation with advanced infrastructure, high standards of living, and a strong emphasis on education. The country also has a rich history and is famous for its K-pop music and entertainment industry.
    CPU times: total: 0 ns
    Wall time: 972 ms

SQLiteCache

Now, we cache the answer to the same question by using SQLiteCache.

from langchain_community.cache import SQLiteCache
from langchain_core.globals import set_llm_cache
import os

# Create cache directory
if not os.path.exists("cache"):
    os.makedirs("cache")

# Set SQLiteCache
set_llm_cache(SQLiteCache(database_path="cache/llm_cache.db"))
%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea is a technologically advanced country in East Asia, known for its booming economy, vibrant pop culture, and rich history. It is home to K-pop, Samsung, and delicious cuisine like kimchi. The country also faces tensions with North Korea and strives for reunification.
    CPU times: total: 31.2 ms
    Wall time: 953 ms

Now we invoke the chain with the same question.

%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea is a technologically advanced country in East Asia, known for its booming economy, vibrant pop culture, and rich history. It is home to K-pop, Samsung, and delicious cuisine like kimchi. The country also faces tensions with North Korea and strives for reunification.
    CPU times: total: 375 ms
    Wall time: 375 ms

Note that if we use SQLiteCache, setting caching again does not delete stored cache.

set_llm_cache(SQLiteCache(database_path="cache/llm_cache.db"))
%%time
# Invoke chain
response = chain.invoke({"country": "South Korea"})
print(response.content)
South Korea is a technologically advanced country in East Asia, known for its booming economy, vibrant pop culture, and rich history. It is home to K-pop, Samsung, and delicious cuisine like kimchi. The country also faces tensions with North Korea and strives for reunification.
    CPU times: total: 0 ns
    Wall time: 4.01 ms
PreviousChat ModelsNextCaching VLLM

Last updated