This tutorial provides a comprehensive guide on how to use the CSVLoader utility in LangChain to seamlessly integrate data from CSV files into your applications. The CSVLoader is a powerful tool for processing structured data, enabling developers to extract, parse, and utilize information from CSV files within the LangChain framework.
CSVLoader simplifies the process of loading, parsing, and extracting data from CSV files, allowing developers to seamlessly incorporate this information into LangChain workflows.
# Set environment variables
from langchain_opentutorial import set_env
from dotenv import load_dotenv
if not load_dotenv():
set_env(
{
"OPENAI_API_KEY": "",
"LANGCHAIN_API_KEY": "",
"LANGCHAIN_TRACING_V2": "true",
"LANGCHAIN_ENDPOINT": "https://api.smith.langchain.com",
"LANGCHAIN_PROJECT": "04-CSV-Loader",
}
)
You can alternatively set OPENAI_API_KEY in .env file and load it.
[Note] This is not necessary if you've already set OPENAI_API_KEY in previous steps.
from dotenv import load_dotenv
load_dotenv()
True
How to load CSVs
A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. LangChain can help you load CSV files easily—just import CSVLoader to get started.
Each line of the file is a data record, and each record consists of one or more fields, separated by commas.
We use a sample CSV file for the example.
from langchain_community.document_loaders.csv_loader import CSVLoader
# Create CSVLoader instance
loader = CSVLoader(file_path="./data/titanic.csv")
# Load documents
docs = loader.load()
for record in docs[:2]:
print(record)
CSVLoader accepts a csv_args keyword argument that supports customization of the parameters passed to Python's csv.DictReader. This allows you to handle various CSV formats, such as custom delimiters, quote characters, or specific newline handling.
See Python's csv module documentation for more information on supported csv_args and how to tailor the parsing to your specific needs.
Passenger ID: 1
Survival (1: Survived, 0: Died): 0
Passenger Class: 3
Name: Braund, Mr. Owen Harris
Sex: male
Age: 22
Number of Siblings/Spouses Aboard: 1
Number of Parents/Children Aboard: 0
Ticket Number: A/5 21171
Fare: 7.25
Cabin:
Port of Embarkation: S
Specify a column to identify the document source
You should use the source_column argument to specify the source of the documents generated from each row. Otherwise file_path will be used as the source for all documents created from the CSV file.
This is particularly useful when using the documents loaded from a CSV file in a chain designed to answer questions based on their source.
This example shows how to generate XML Document format from CSVLoader. By processing data from a CSV file, you can convert its rows and columns into a structured XML representation.
Convert a row in the document.
row = docs[1].page_content.split("\n") # split by new line
row_str = "<row>"
for element in row:
splitted_element = element.split(":") # split by ":"
value = splitted_element[-1] # get value
col = ":".join(splitted_element[:-1]) # get column name
row_str += f"<{col}>{value.strip()}</{col}>"
row_str += "</row>"
print(row_str)
211Cumings, Mrs. John Bradley (Florence Briggs Thayer)female3810PC 1759971.2833C85C
Convert entire rows in the document.
for doc in docs[1:6]: # skip header
row = doc.page_content.split("\n")
row_str = "<row>"
for element in row:
splitted_element = element.split(":") # split by ":"
value = splitted_element[-1] # get value
col = ":".join(splitted_element[:-1]) # get column name
row_str += f"<{col}>{value.strip()}</{col}>"
row_str += "</row>"
print(row_str)
211Cumings, Mrs. John Bradley (Florence Briggs Thayer)female3810PC 1759971.2833C85C
313Heikkinen, Miss. Lainafemale2600STON/O2. 31012827.925S
411Futrelle, Mrs. Jacques Heath (Lily May Peel)female351011380353.1C123S
503Allen, Mr. William Henrymale35003734508.05S
603Moran, Mr. Jamesmale003308778.4583Q
UnstructuredCSVLoader
UnstructuredCSVLoader can be used in both single and elements mode. If you use the loader in “elements” mode, the CSV file will be a single Unstructured Table element. If you use the loader in elements” mode, an HTML representation of the table will be available in the text_as_html key in the document metadata.
from langchain_community.document_loaders.csv_loader import UnstructuredCSVLoader
# Generate UnstructuredCSVLoader instance with elements mode
loader = UnstructuredCSVLoader(file_path="./data/titanic.csv", mode="elements")
docs = loader.load()
html_content = docs[0].metadata["text_as_html"]
# Partial output due to space constraints
print(html_content[:810])
DataFrameLoader
Pandas is an open-source data analysis and manipulation tool for the Python programming language. This library is widely used in data science, machine learning, and various fields for working with data.
LangChain's DataFrameLoader is a powerful utility designed to seamlessly integrate Pandas DataFrames into LangChain workflows.
import pandas as pddf = pd.read_csv("./data/titanic.csv")
Search the first 5 rows.
df.head(n=5)
Parameters page_content_column (str) – Name of the column containing the page content. Defaults to “text”.
from langchain_community.document_loaders import DataFrameLoader# The Name column of the DataFrame is specified to be used as the content of each document.loader = DataFrameLoader(df, page_content_column="Name")docs = loader.load()print(docs[0].page_content)
Braund, Mr. Owen Harris
Lazy Loading for large tables. Avoid loading the entire table into memory
# Lazy load records from dataframe.for row in loader.lazy_load(): print(row) break # print only the first row
page_content='Braund, Mr. Owen Harris' metadata={'PassengerId': 1, 'Survived': 0, 'Pclass': 3, 'Sex': 'male', 'Age': 22.0, 'SibSp': 1, 'Parch': 0, 'Ticket': 'A/5 21171', 'Fare': 7.25, 'Cabin': nan, 'Embarked': 'S'}
