【LangChain系列】第一篇：文档加载

LangChain提供了一套强大的文档加载器，简化了从PDF、网站、YouTube视频和专有数据库（如Notion）等不同来源加载和标准化数据的过程。这篇博文我们将学习LangChain的文档加载功能，涵盖了各种加载器类型、实际应用和代码示例，使你能够轻松地将数据集成到你的机器学习工作流程中。

一、什么是文档加载器

文档加载器是LangChain生态系统提供的基本构建块，负责访问来自各种格式和来源的数据并将其转换为标准化格式，无论您的数据是 PDF、网站还是专有数据库，文档加载器都可以非常轻松地加载和处理数据。

文档加载器的主要用途是获取这些不同的数据源数据，并将它们加载到一个标准文档对象中，该对象由内容本身和关联的元数据组成。通过这样做，它们为处理数据提供了一致的接口，使您能够专注于构建智能应用程序的其他方面。

二、文档加载类型

LangChain拥有80多种不同类型的文档加载器，可满足各种数据源和格式的需求。简单分类如下：

• 非结构化数据加载器：这些加载器旨在处理原始非结构化形式的数据，例如文本文件、YouTube、Twitter 和 Hacker News 等公共数据源。

#! pip install langchain
# Loading data from a PDF
from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader("path/to/your/pdf_file.pdf")
docs = loader.load()

• 专有数据源的加载器：如果您依赖于 Figma 或 Notion 等专有数据源，LangChain 为您提供了专门设计用于处理这些格式的加载器。

# Loading data from Notion
from langchain.document_loaders import NotionDirectoryLoader

loader = NotionDirectoryLoader("path/to/notion/export")
docs = loader.load()

• 结构化数据加载器：虽然 LangChain 通常与非结构化数据相关联，但它也为 Airbyte、Stripe 和 Airtable 等结构化数据源提供加载器，允许您对这些结构化格式中包含的文本数据执行问答和语义搜索。

# Loading data from Airtable
from langchain_community.document_loaders import AirtableLoader

# Your airtable variables
api_key = "xxx"
base_id = "xxx"
table_id = "xxx"

loader = AirtableLoader(api_key, table_id, base_id)
docs = loader.load()

三、使用文档加载器

现在我们已经介绍了文档加载器的基础知识及其类型，让我们深入了解一些如何使用它们从各种来源加载数据的实际示例。

1.加载PDFs

让我们从一个常见场景开始：从 PDF 文件加载数据，以下是使用 LangChain 的 PyPDF 加载器实现此目的的方法：

from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader("docs/cs229_lectures/MachineLearning-Lecture01.pdf")
pages = loader.load()

# Access the content of the first page
page = pages[0]
print(page.metadata)
# Output
# {'page': 0, 'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf'}
print(page.page_content[:500])

>>>
MachineLearning-Lecture01  
Instructor (Andrew Ng):  Okay. Good morning. Welcome to CS229, the machine 
learning class. So what I wanna do today is ju st spend a little time going over the logistics 
of the class, and then we'll start to  talk a bit about machine learning.  
By way of introduction, my name's  Andrew Ng and I'll be instru ctor for this class. And so 
I personally work in machine learning, and I' ve worked on it for about 15 years now, and 
I actually think that machine learning is th e most exciting field of all the computer 
sciences. So I'm actually always excited about  teaching this class. Sometimes I actually 
think that machine learning is not only the most exciting thin g in computer science, but 
the most exciting thing in all of human e ndeavor, so maybe a little bias there.

在此示例中，我们将 PDF 脚本加载到 LangChain 中，这会产生一个对象列表，每个对象代表 PDF 的一个页面。然后，我们可以访问每个页面的内容和元数据。

2.加载视频

想象一下，你是一个喜欢参加在线讲座和会议的人，如果您可以与这些 YouTube 视频的内容聊天，那不是很神奇吗？LangChain通过组合 使这成为可能，您可以稍后将此数据加载到RAG应用程序中，代码如下：

# ! pip install yt_dlp
# ! pip install pydub
# ! pip install ffmpeg
# ! pip install ffprobe

from langchain.document_loaders.generic import GenericLoader
from langchain.document_loaders.parsers import OpenAIWhisperParser
from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader

url = "https://www.youtube.com/watch?v=jGwO_UgTS7I"
save_dir = "docs/youtube/"

loader = GenericLoader(YoutubeAudioLoader([url], save_dir), OpenAIWhisperParser())
docs = loader.load()
# [youtube] Extracting URL: https://www.youtube.com/watch?v=jGwO_UgTS7I
# [youtube] jGwO_UgTS7I: Downloading webpage
# [youtube] jGwO_UgTS7I: Downloading ios player API JSON
# [youtube] jGwO_UgTS7I: Downloading android player API JSON
# WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "jGwO_UgTS7I")
# [youtube] jGwO_UgTS7I: Downloading m3u8 information
# [info] jGwO_UgTS7I: Downloading 1 format(s): 140
# [download] docs/youtube//Stanford CS229： Machine Learning Course, Lecture 1 - Andrew Ng (Autumn 2018).m4a has already been downloaded
# [download] 100% of   69.76MiB
# [ExtractAudio] Not converting audio docs/youtube//Stanford CS229： Machine Learning Course, Lecture 1 - Andrew Ng (Autumn 2018).m4a; file is already in target format m4a
# Transcribing part 1!
# Transcribing part 2!
# Transcribing part 3!
# Transcribing part 4!
print(docs[0].page_content[:500])

>>>
Welcome to CS229 Machine Learning. Uh, some of you know that this is a class that's taught at Stanford for a long time. And this is often the class that, um, I most look forward to teaching each year because this is where we've helped, I think, several generations of Stanford students become experts in machine learning, got- built many of their products and services and startups that I'm sure, many of you or probably all of you are using, uh, uh, today. Um, so what I want to do today was spend some time talking over, uh, logistics and then, uh, spend some time, you know, giving you a beginning of an intro, talk a little bit about machine learning. So about 229, um, you know, all of you have been reading about AI in the news, uh, about machine learning in the news. Um, and you've probably heard me or others say AI is the new electricity.

在此示例中，我们加载了一个 YouTube 视频并使用 OpenAI 的 Whisper 模型转录其音频，从而可以与视频内容聊天。想象一下，能够就 Andrew Ng 的讲座或 YouTube 上的任何其他教育视频提出问题！

3.加载网站

互联网上到处充满着有用的信息，LangChain基于Web的加载器允许您利用这些丰富的信息。假设你遇到了一个有趣的 GitHub 存储库，其中包含一个您想与之聊天的 README 文件：

from langchain.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://raw.githubusercontent.com/RutamBhagat/code_wizard_frontend/main/README.md")
docs = loader.load()

print(docs[0].page_content[:500])

>>>
# Code Wizard: LangChain Documentation AI Chatbot

Code Wizard is a super cool AI chatbot that helps you learn and use the LangChain Documentation in an interactive way. Just ask it anything about LangChain concepts or code, and it'll break it down for you in an easy-to-understand way. Built with Next.js, FastAPI, LangChain, and a local LLaMA model.

**Link to project:** https://code-wizard-frontend.vercel.app/


https://github.com/RutamBhagat/code_wizard_frontend/assets/72187009/353ced90-f408-44ae-b633-c30f20dbd28f

在此示例中，我们将加载的内容存储在对象列表中，我们可以通过打印.code_wizard_frontendWebBaseLoaderDocumentdocs[0]来访问第一个文档的文本内容.page_content。虽然加载的内容可能包含一些格式或空格问题，但此示例演示了LangChain的多功能性，允许您加载和处理来自各种在线源的数据。

4.加载Notion

Notion 已成为个人和专业知识管理的流行工具，使其成为许多用户的宝贵数据来源。LangChain使您能够从Notion数据库加载数据并无缝地使用它。

首先，您需要以兼容的格式导出 Notion 数据。下面是一个如何使用LangChain从Notion数据库加载数据的示例：

from langchain.document_loaders import NotionDirectoryLoader

# Export your Notion data and save it in a directory
loader = NotionDirectoryLoader("path/to/your/notion/export")
docs = loader.load()

# Print the content of the first document
print(docs[0].metadata)
# {'source': "docs/Notion_DB/Blendle's Employee Handbook e367aa77e225482c849111687e114a56.md"}
print(docs[0].page_content[:500])

>>>
# Blendle's Employee Handbook

This is a living document with everything we've learned working with people while running a startup. And, of course, we continue to learn. Therefore it's a document that will continue to change. 

**Everything related to working at Blendle and the people of Blendle, made public.**

These are the lessons from three years of working with the people of Blendle. It contains everything from [how our leaders lead](https://www.notion.so/ecfb7e647136468a9a0a32f1771a8f52?pvs=21) to [how we increase salaries](https://www.notion.so/Salary-Review-e11b6161c6d34f5c9568bb3e83ed96b6?pvs=21), from [how we hire](https://www.notion.so/Hiring-451bbcfe8d9b49438c0633326bb7af0a?pvs=21) and [fire](https://www.notion.so/Firing-5567687a2000496b8412e53cd58eed9d?pvs=21) to [how we think people should give each other feedback](https://www.notion.so/Our-Feedback-Process-eb64f1de796b4350aeab3bc068e3801f?pvs=21) — and much more.

在这个例子中，我们从 Notion 数据库导出数据，并将加载的内容存储在对象列表中，我们可以通过打印来访问第一个文档的文本内容。NotionDirectoryLoaderDocumentdocs[0].page_content。

通过使用LangChain的文档加载器，您可以充分利用您的Notion数据库并与他们聊天，从而获得见解并做出更明智的决策。

小节

本节我们学习了LangChain的文档加载器，LangChain的文档加载器允许您从PDF、YouTube视频、网站和专有数据库等不同来源加载数据，使您能够构建真正理解数据并与之交互的智能应用程序。通过简化数据加载和标准化，这些加载器可以充分利用您的数据，让您提出问题，获得见解。本节的内容就到这里了，希望对同学们有所帮助。

小编是一名热爱人工智能的专栏作者，致力于分享人工智能领域的最新知识、技术和趋势。这里，你将能够了解到人工智能的最新应用和创新，探讨人工智能对未来社会的影响，以及探索人工智能背后的科学原理和技术实现。欢迎大家点赞，评论，收藏，让我们一起探索人工智能的奥秘，共同见证科技的进步！