CSV Loader

• Author: JoonHo Kim

• Peer Review : syshin0116, forwardyoung

• Proofread : Q0211

• This is a part of LangChain Open Tutorial

Overview

This tutorial provides a comprehensive guide on how to use the CSVLoader utility in LangChain to seamlessly integrate data from CSV files into your applications. The CSVLoader is a powerful tool for processing structured data, enabling developers to extract, parse, and utilize information from CSV files within the LangChain framework.

Comma-Separated Values (CSV) is one of the most common formats for storing and exchanging data.

CSVLoader simplifies the process of loading, parsing, and extracting data from CSV files, allowing developers to seamlessly incorporate this information into LangChain workflows.

Table of Contents

• Overview

• Environment Setup

• How to load CSVs

• Customizing the CSV parsing and loading

• Specify a column to identify the document source

• Generating XML document format

• UnstructuredCSVLoader

• DataFrameLoader

References

• Langchain CSVLoader

• Langchain How to load CSVs

• Langchain DataFrameLoader

---

Environment Setup

Set up the environment. You may refer to Environment Setup for more details.

[Note]

• langchain-opentutorial is a package that provides a set of easy-to-use environment setup, useful functions and utilities for tutorials.

• You can check out the langchain-opentutorial for more details.

• unstructured package is a Python library for extracting text and metadata from various document formats like PDF and CSV

%%capture --no-stderr
%pip install langchain-opentutorial

# Install required packages
from langchain_opentutorial import package

package.install(
    [
        "langchain_community",
        "unstructured"
    ],
    verbose=False,
    upgrade=False,
)

# Set environment variables
from langchain_opentutorial import set_env
from dotenv import load_dotenv

if not load_dotenv():
    set_env(
        {
            "OPENAI_API_KEY": "",
            "LANGCHAIN_API_KEY": "",
            "LANGCHAIN_TRACING_V2": "true",
            "LANGCHAIN_ENDPOINT": "https://api.smith.langchain.com",
            "LANGCHAIN_PROJECT": "04-CSV-Loader",
        }
    )

You can alternatively set OPENAI_API_KEY in .env file and load it.

[Note] This is not necessary if you've already set OPENAI_API_KEY in previous steps.

from dotenv import load_dotenv

load_dotenv()

True

How to load CSVs

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. LangChain can help you load CSV files easily—just import CSVLoader to get started.

Each line of the file is a data record, and each record consists of one or more fields, separated by commas.

We use a sample CSV file for the example.

from langchain_community.document_loaders.csv_loader import CSVLoader

# Create CSVLoader instance
loader = CSVLoader(file_path="./data/titanic.csv")

# Load documents
docs = loader.load()

for record in docs[:2]:
    print(record)

page_content='PassengerId: 1
    Survived: 0
    Pclass: 3
    Name: Braund, Mr. Owen Harris
    Sex: male
    Age: 22
    SibSp: 1
    Parch: 0
    Ticket: A/5 21171
    Fare: 7.25
    Cabin: 
    Embarked: S' metadata={'source': './data/titanic.csv', 'row': 0}
    page_content='PassengerId: 2
    Survived: 1
    Pclass: 1
    Name: Cumings, Mrs. John Bradley (Florence Briggs Thayer)
    Sex: female
    Age: 38
    SibSp: 1
    Parch: 0
    Ticket: PC 17599
    Fare: 71.2833
    Cabin: C85
    Embarked: C' metadata={'source': './data/titanic.csv', 'row': 1}

print(docs[1].page_content)

PassengerId: 2
    Survived: 1
    Pclass: 1
    Name: Cumings, Mrs. John Bradley (Florence Briggs Thayer)
    Sex: female
    Age: 38
    SibSp: 1
    Parch: 0
    Ticket: PC 17599
    Fare: 71.2833
    Cabin: C85
    Embarked: C

Customizing the CSV parsing and loading

CSVLoader accepts a csv_args keyword argument that supports customization of the parameters passed to Python's csv.DictReader. This allows you to handle various CSV formats, such as custom delimiters, quote characters, or specific newline handling.

See Python's csv module documentation for more information on supported csv_args and how to tailor the parsing to your specific needs.

loader = CSVLoader(
    file_path="./data/titanic.csv",
    csv_args={
        "delimiter": ",",
        "quotechar": '"',
        "fieldnames": [
            "Passenger ID",
            "Survival (1: Survived, 0: Died)",
            "Passenger Class",
            "Name",
            "Sex",
            "Age",
            "Number of Siblings/Spouses Aboard",
            "Number of Parents/Children Aboard",
            "Ticket Number",
            "Fare",
            "Cabin",
            "Port of Embarkation",
        ],
    },
)

docs = loader.load()

print(docs[1].page_content)

Passenger ID: 1
    Survival (1: Survived, 0: Died): 0
    Passenger Class: 3
    Name: Braund, Mr. Owen Harris
    Sex: male
    Age: 22
    Number of Siblings/Spouses Aboard: 1
    Number of Parents/Children Aboard: 0
    Ticket Number: A/5 21171
    Fare: 7.25
    Cabin: 
    Port of Embarkation: S

Specify a column to identify the document source

You should use the source_column argument to specify the source of the documents generated from each row. Otherwise file_path will be used as the source for all documents created from the CSV file.

This is particularly useful when using the documents loaded from a CSV file in a chain designed to answer questions based on their source.

loader = CSVLoader(
    file_path="./data/titanic.csv",
    source_column="PassengerId",  # Specify the source column
)

docs = loader.load()  

print(docs[1])
print(docs[1].metadata)

page_content='PassengerId: 2
    Survived: 1
    Pclass: 1
    Name: Cumings, Mrs. John Bradley (Florence Briggs Thayer)
    Sex: female
    Age: 38
    SibSp: 1
    Parch: 0
    Ticket: PC 17599
    Fare: 71.2833
    Cabin: C85
    Embarked: C' metadata={'source': '2', 'row': 1}
    {'source': '2', 'row': 1}

Generating XML document format

This example shows how to generate XML Document format from CSVLoader. By processing data from a CSV file, you can convert its rows and columns into a structured XML representation.

Convert a row in the document.

row = docs[1].page_content.split("\n")  # split by new line
row_str = "<row>"
for element in row:
    splitted_element = element.split(":")  # split by ":"
    value = splitted_element[-1]  # get value
    col = ":".join(splitted_element[:-1])  # get column name

    row_str += f"<{col}>{value.strip()}</{col}>"
row_str += "</row>"
print(row_str)

211Cumings, Mrs. John Bradley (Florence Briggs Thayer)female3810PC 1759971.2833C85C

Convert entire rows in the document.

for doc in docs[1:6]:  # skip header
    row = doc.page_content.split("\n")
    row_str = "<row>"
    for element in row:
        splitted_element = element.split(":")  # split by ":"
        value = splitted_element[-1]  # get value
        col = ":".join(splitted_element[:-1])  # get column name
        row_str += f"<{col}>{value.strip()}</{col}>"
    row_str += "</row>"
    print(row_str)

211Cumings, Mrs. John Bradley (Florence Briggs Thayer)female3810PC 1759971.2833C85C
    313Heikkinen, Miss. Lainafemale2600STON/O2. 31012827.925S
    411Futrelle, Mrs. Jacques Heath (Lily May Peel)female351011380353.1C123S
    503Allen, Mr. William Henrymale35003734508.05S
    603Moran, Mr. Jamesmale003308778.4583Q

UnstructuredCSVLoader

UnstructuredCSVLoader can be used in both single and elements mode. If you use the loader in “elements” mode, the CSV file will be a single Unstructured Table element. If you use the loader in elements” mode, an HTML representation of the table will be available in the text_as_html key in the document metadata.

from langchain_community.document_loaders.csv_loader import UnstructuredCSVLoader

# Generate UnstructuredCSVLoader instance with elements mode
loader = UnstructuredCSVLoader(file_path="./data/titanic.csv", mode="elements")

docs = loader.load()

html_content = docs[0].metadata["text_as_html"]

# Partial output due to space constraints
print(html_content[:810]) 

DataFrameLoader
Pandas is an open-source data analysis and manipulation tool for the Python programming language. This library is widely used in data science, machine learning, and various fields for working with data.
LangChain's DataFrameLoader is a powerful utility designed to seamlessly integrate Pandas  DataFrames into LangChain workflows.
import pandas as pddf = pd.read_csv("./data/titanic.csv")
Search the first 5 rows.
df.head(n=5)





Parameters page_content_column (str) – Name of the column containing the page content. Defaults to “text”.
from langchain_community.document_loaders import DataFrameLoader# The Name column of the DataFrame is specified to be used as the content of each document.loader = DataFrameLoader(df, page_content_column="Name")docs = loader.load()print(docs[0].page_content)
Braund, Mr. Owen Harris
Lazy Loading for large tables. Avoid loading the entire table into memory
# Lazy load records from dataframe.for row in loader.lazy_load():    print(row)    break  # print only the first row
page_content='Braund, Mr. Owen Harris' metadata={'PassengerId': 1, 'Survived': 0, 'Pclass': 3, 'Sex': 'male', 'Age': 22.0, 'SibSp': 1, 'Parch': 0, 'Ticket': 'A/5 21171', 'Fare': 7.25, 'Cabin': nan, 'Embarked': 'S'}