The Hugging Face models can be run locally through the HuggingFacePipeline class.
The Hugging Face model Hub hosts over 120k models, 20k datasets, and 50k demo apps (Spaces) on its online platform, all of which are open-source and publicly available, allowing people to easily collaborate and build ML together.
These can be used in LangChain either by calling them through this local pipeline wrapper or by calling hosted inference endpoints through the HuggingFaseHub class. For more information on hosted pipelines, please refer to the HuggingFaseHub notebook.
# Set environment variables
from dotenv import load_dotenv
from langchain_opentutorial import set_env
# Attempt to load environment variables from a .env file; if unsuccessful, set them manually.
if not load_dotenv():
set_env(
{
"HUGGINGFACEHUB_API_TOKEN": "",
"LANGCHAIN_API_KEY": "",
"LANGCHAIN_TRACING_V2": "true",
"LANGCHAIN_ENDPOINT": "https://api.smith.langchain.com",
"LANGCHAIN_PROJECT": "Huggingface-Piplines",
}
)
[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip
[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip
# Path to download Hugging Face models/tokenizers
import os
# ./cache/ Set to download to the specified path
os.environ["HF_HOME"] = "./cache/"
from langchain_huggingface import HuggingFacePipeline
# Download the HuggingFace model.
hf = HuggingFacePipeline.from_model_id(
model_id="microsoft/Phi-3-mini-4k-instruct", # Specify the ID of the model to use.
task="text-generation", # Specify the task to perform. Here, it's text generation.
pipeline_kwargs={"max_new_tokens": 64}, # Set additional arguments to pass to the pipeline. Here, we limit the maximum number of new tokens to 64.
)
Loading checkpoint shards: 0%| | 0/2 [00:00
Device set to use cpu
You can also load by directly passing an existing transformers pipeline.
The text ageneration model is implemented using HuggingFacePipeline.
AutoTokenizer and AutoModelForCausalLM are used to load the microsoft/Phi-3-mini-4k-instruct model and tokenizer.
The pipeline function is used to create a "text-generation" pipeline, setting up the model and tokenizer. The maximum number of generated tokens is limited to 64.
The HuggingFacePipeline class is used to create an hf object, and the generated pipeline is passed to it.
Using this created hf object, you can perform text generation for a given prompt.
from langchain_huggingface import HuggingFacePipelinefrom transformers import AutoModelForCausalLM, AutoTokenizer, pipeline# Specify the ID of the model to use.model_id = "microsoft/Phi-3-mini-4k-instruct" # Load the tokenizer for the specified model. tokenizer = AutoTokenizer.from_pretrained(model_id) # Load the specified model.model = AutoModelForCausalLM.from_pretrained(model_id) # Create a text generation pipeline and set the maximum number of new tokens to be generated to 64.pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=64)# Create a HuggingFacePipeline object and pass the generated pipeline to it.hf = HuggingFacePipeline(pipeline=pipe)
Loading checkpoint shards: 0%| | 0/2 [00:00Device set to use cpuUsage of Gated ModelThe Gated Model is a model that can be used under a license agreement from Hugging Face.You must first visit the model page and agree to the terms before obtaining a Hugging Face token.Below is an example of how to use the Gated Model. You need to specify the Hugging Face token as shown below.from langchain_huggingface import HuggingFacePipelinefrom transformers import AutoModelForCausalLM, AutoTokenizer, pipeline# Specify the model ID registered in the Hugging Face repository.model_id = "microsoft/Phi-3-mini-4k-instruct" # Enter the Hugging Face token you received here.your_huggingface_token = ""# Load the tokenizer.tokenizer = AutoTokenizer.from_pretrained(model_id, token=your_huggingface_token)# Load the specified model.model = AutoModelForCausalLM.from_pretrained( model_id, token=your_huggingface_token, # load_in_4bit=True, # If bitsandbytes is installed (Linux) # attn_implementation="flash_attention_2", # If you have an Ampere GPU)# Create the pipeline.pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=64)# Create a HuggingFacePipeline object and pass the created pipeline.hf_llm = HuggingFacePipeline(pipeline=pipe)Loading checkpoint shards: 0%| | 0/2 [00:00Device set to use cpuExecute and check the resultsfor token in hf_llm.stream("What is the capital of France?"): print(token, end="", flush=True) # Answer The capital of France is Paris.Create ChainOnce the model is loaded into memory, you can configure it with prompts to form a chain.A prompt template defining the question and answer format is created using the PromptTemplate class.Create a chain object by connecting the prompt object and the hf object in a pipeline.Call the chain.invoke() method to generate and output an answer for the given question.from huggingface_hub import loginlogin()VBox(children=(HTML(value=' from langchain_huggingface import ChatHuggingFacellm = ChatHuggingFace(llm=hf)from langchain.prompts import PromptTemplatefrom langchain_core.output_parsers import StrOutputParsertemplate = """<|system|>You are a helpful assistant.<|end|><|user|>{question}<|end|><|assistant|>""" # Template for defining question and answer formatsprompt = PromptTemplate.from_template(template) # Create a prompt object using the template# Create a chain by connecting the prompt and the language modelchain = prompt |llm| StrOutputParser()question = "What is the capital of the United France?" # Define the questionprint( chain.invoke({"question": question})) # Call the chain to generate and output an answer to the question<|user|> <|system|>You are a helpful assistant.<|end|> <|user|>What is the capital of the United France?<|end|> <|assistant|><|end|> <|assistant|> The capital of France is Paris.GPU InferenceWhen running on a GPU, you can specify the device=n parameter to place the model on a specific device.The default value is -1, which means inference is performed on the CPU.If you have multiple GPUs or if the model is too large for a single GPU, you can specify device_map="auto".In this case, the library is required and is used to automatically determine how to load the model weights.Caution: device and device_map should not be specified together, as this can cause unexpected behavior.Load the gpt2 model using HuggingFacePipeline and set the device parameter to 0 to run it on the GPU.Limit the maximum number of tokens to be generated to 64 using the pipeline_kwargs parameter.Connect the prompt and gpu_llm in a pipeline to create the gpu_chain.Call the gpu_chain.invoke() method to generate and output an answer for the given question.gpu_llm = HuggingFacePipeline.from_model_id( model_id="microsoft/Phi-3-mini-4k-instruct", task="text-generation", device=-1, # Specifies the GPU device number. -1 stands for CPU. pipeline_kwargs={"max_new_tokens": 64}, # Set additional arguments to be passed to the pipeline. In this case, limit the maximum number of tokens to be generated to 64.)prompt = PromptTemplate.from_template(template) # Create a prompt object using the template# Create a chain by connecting the prompt and the language model.gpu_chain = prompt | gpu_llm | StrOutputParser()question = "What is the capital of France?" #Invoke the chain to generate and output the answer to the questionprint(gpu_chain.invoke({"question": question}))Loading checkpoint shards: 0%| | 0/2 [00:00Device set to use cpu<|system|>You are a helpful assistant.<|end|><|user|>What is the capital of France?<|end|><|assistant|> The capital of France is Paris. It is not only the largest city in France but also one of the most important cultural and commercial centers in Europe. Paris is known for its historical landmarks such as the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum, which is the worldBatch GPU InferenceWhen running on a GPU device, you can perform inference in batch mode on the GPU.Load the microsoft/Phi-3-mini-4k-instruct model using HuggingFacePipeline and set it to run on the GPU.When creating the gpu_llm, set the batch_size to 2, temperature to 0, and max_length to 64.Connect the prompt and gpu_llm in a pipeline to create the gpu_chain, and set the end token to "\n\n".Use gpu_chain.batch() to generate answers in parallel for the questions in the questions.Wrap each answer with tags and separate each answer with a line break.gpu_llm = HuggingFacePipeline.from_model_id( model_id="microsoft/Phi-3-mini-4k-instruct", task="text-generation", device=-1, # Specifies the GPU device number. -1 stands for CPU. batch_size=2, # Adjust the batch size. Set it appropriately based on GPU memory and model size. model_kwargs={ "temperature": 0, "max_length": 64, "do_sample": True }, # Set additional arguments to be passed to the model.)# Create a chain by connecting the prompt and the language model.gpu_chain = prompt | gpu_llm.bind(stop=["\n\n"]) questions = []for i in range(4): # Generate a list of questions questions.append({"question": f"What is the number {i} in English?"})answers = gpu_chain.batch(questions) for answer in answers: print(answer) print("")Loading checkpoint shards: 0%| | 0/2 [00:00Device set to use cpu<|system|>You are a helpful assistant.<|end|><|user|>What is the number 0 in English?<|end|><|assistant|> The number 0 in English is called "zero."<|system|>You are a helpful assistant.<|end|><|user|>What is the number 1 in English?<|end|><|assistant|> The number 1 in English is simply called "one."<|system|>You are a helpful assistant.<|end|><|user|>What is the number 2 in English?<|end|><|assistant|> The number 2 in English is spelled "two."<|system|>You are a helpful assistant.<|end|><|user|>What is the number 3 in English?<|end|><|assistant|> The number 3 in English is spelled "three."
