Automated doc processing is likely one of the greatest winners of the ChatGPT revolution, as LLMs are capable of sort out a variety of topics and duties in a zero-shot setting, which means with out in-domain labeled coaching knowledge. This has made constructing AI-powered purposes to course of, parse, and robotically perceive arbitrary paperwork a lot simpler. Although naive approaches utilizing LLMs are nonetheless hindered by non-text context, comparable to figures, photos, and tables, that is what we’ll attempt to tackle on this weblog publish, with a particular deal with PDFs.
At a fundamental stage, PDFs are only a assortment of characters, photos, and features together with their precise coordinates. They don’t have any inherent “textual content” construction and weren’t constructed to be processed as textual content however solely to be seen as is. That is what makes working with them tough, as text-only approaches fail to seize all of the structure and visible parts in a lot of these paperwork, leading to a major lack of context and knowledge.
One technique to bypass this “text-only” limitation is to do heavy pre-processing of the doc by detecting tables, photos, and structure earlier than feeding them to the LLM. Tables may be parsed to Markdown or JSON, photos and figures may be represented by their captions, and the textual content may be fed as is. Nevertheless, this strategy requires customized fashions and can nonetheless end in some lack of data, so can we do higher?
Most up-to-date massive fashions at the moment are multi-modal, which means they’ll course of a number of modalities like textual content, code, and pictures. This opens the best way to a less complicated answer to our drawback the place one mannequin does all the pieces without delay. So, as an alternative of captioning photos and parsing tables, we will simply feed the web page as a picture and course of it as is. Our pipeline will be capable of load the PDF, extract every web page as a picture, cut up it into chunks (utilizing the LLM), and index every chunk. If a bit is retrieved, then the complete web page is included within the LLM context to carry out the duty. In what follows, we’ll element how this may be applied in apply.
The pipeline we’re implementing is a two-step course of. First, we section every web page into vital chunks and summarize every of them. Second, we index chunks as soon as then search the chunks every time we get a request and embrace the complete context with every retrieved chunk within the LLM context.
Step 1: Web page Segmentation and Summarization
We extract the pages as photos and move every of them to the multi-modal LLM to section them. Fashions like Gemini can perceive and course of web page structure simply:
- Tables are recognized as one chunk.
- Figures kind one other chunk.
- Textual content blocks are segmented into particular person chunks.
- …
For every aspect, the LLM generates a abstract than may be embedded and listed right into a vector database.
Step 2: Embedding and Contextual Retrieval
On this tutorial we’ll use textual content embedding just for simplicity however one enchancment can be to make use of imaginative and prescient embeddings instantly.
Every entry within the database consists of:
- The abstract of the chunk.
- The web page quantity the place it was discovered.
- A hyperlink to the picture illustration of the complete web page for added context.
This schema permits for native stage searches (on the chunk stage) whereas holding observe of the context (by linking again to the complete web page). For instance, if a search question retrieves an merchandise, the Agent can embrace the whole web page picture to supply full structure and further context to the LLM as a way to maximize response high quality.
By offering the complete picture, all of the visible cues and essential structure data (like photos, titles, bullet factors… ) and neighboring objects (tables, paragraph, …) can be found to the LLM on the time of producing a response.
We are going to implement every step as a separate, re-usable agent:
The primary agent is for parsing, chunking, and summarization. This includes the segmentation of the doc into vital chunks, adopted by the technology of summaries for every of them. This agent solely must be run as soon as per PDF to preprocess the doc.
The second agent manages indexing, search, and retrieval. This consists of inserting the embedding of chunks into the vector database for environment friendly search. Indexing is carried out as soon as per doc, whereas searches may be repeated as many occasions as wanted for various queries.
For each brokers, we use Gemini, a multimodal LLM with robust imaginative and prescient understanding talents.
Parsing and Chunking Agent
The primary agent is accountable for segmenting every web page into significant chunks and summarizing every of them, following these steps:
Step 1: Extracting PDF Pages as Photos
We use the pdf2image library. The photographs are then encoded in Base64 format to simplify including them to the LLM request.
Right here’s the implementation:
from document_ai_agents.document_utils import extract_images_from_pdf
from document_ai_agents.image_utils import pil_image_to_base64_jpeg
from pathlib import Pathclass DocumentParsingAgent:
@classmethod
def get_images(cls, state):
"""
Extract pages of a PDF as Base64-encoded JPEG photos.
"""
assert Path(state.document_path).is_file(), "File doesn't exist"
# Extract photos from PDF
photos = extract_images_from_pdf(state.document_path)
assert photos, "No photos extracted"
# Convert photos to Base64-encoded JPEG
pages_as_base64_jpeg_images = [pil_image_to_base64_jpeg(x) for x in images]
return {"pages_as_base64_jpeg_images": pages_as_base64_jpeg_images}
extract_images_from_pdf: Extracts every web page of the PDF as a PIL picture.
pil_image_to_base64_jpeg: Converts the picture right into a Base64-encoded JPEG format.
Step 2: Chunking and Summarization
Every picture is then despatched to the LLM for segmentation and summarization. We use structured outputs to make sure we get the predictions within the format we count on:
from pydantic import BaseModel, Area
from typing import Literal
import json
import google.generativeai as genai
from langchain_core.paperwork import Docclass DetectedLayoutItem(BaseModel):
"""
Schema for every detected structure aspect on a web page.
"""
element_type: Literal["Table", "Figure", "Image", "Text-block"] = Area(
...,
description="Sort of detected merchandise. Examples: Desk, Determine, Picture, Textual content-block."
)
abstract: str = Area(..., description="An in depth description of the structure merchandise.")
class LayoutElements(BaseModel):
"""
Schema for the checklist of structure parts on a web page.
"""
layout_items: checklist[DetectedLayoutItem] = []
class FindLayoutItemsInput(BaseModel):
"""
Enter schema for processing a single web page.
"""
document_path: str
base64_jpeg: str
page_number: int
class DocumentParsingAgent:
def __init__(self, model_name="gemini-1.5-flash-002"):
"""
Initialize the LLM with the suitable schema.
"""
layout_elements_schema = prepare_schema_for_gemini(LayoutElements)
self.model_name = model_name
self.mannequin = genai.GenerativeModel(
self.model_name,
generation_config={
"response_mime_type": "utility/json",
"response_schema": layout_elements_schema,
},
)
def find_layout_items(self, state: FindLayoutItemsInput):
"""
Ship a web page picture to the LLM for segmentation and summarization.
"""
messages = [
f"Find and summarize all the relevant layout elements in this PDF page in the following format: "
f"{LayoutElements.schema_json()}. "
f"Tables should have at least two columns and at least two rows. "
f"The coordinates should overlap with each layout item.",
{"mime_type": "image/jpeg", "data": state.base64_jpeg},
]
# Ship the immediate to the LLM
outcome = self.mannequin.generate_content(messages)
knowledge = json.hundreds(outcome.textual content)
# Convert the JSON output into paperwork
paperwork = [
Document(
page_content=item["summary"],
metadata={
"page_number": state.page_number,
"element_type": merchandise["element_type"],
"document_path": state.document_path,
},
)
for merchandise in knowledge["layout_items"]
]
return {"paperwork": paperwork}
The LayoutElements schema defines the construction of the output, with every structure merchandise sort (Desk, Determine, … ) and its abstract.
Step 3: Parallel Processing of Pages
Pages are processed in parallel for pace. The next technique creates a listing of duties to deal with all of the web page picture without delay for the reason that processing is io-bound:
from langgraph.sorts import Shipclass DocumentParsingAgent:
@classmethod
def continue_to_find_layout_items(cls, state):
"""
Generate duties to course of every web page in parallel.
"""
return [
Send(
"find_layout_items",
FindLayoutItemsInput(
base64_jpeg=base64_jpeg,
page_number=i,
document_path=state.document_path,
),
)
for i, base64_jpeg in enumerate(state.pages_as_base64_jpeg_images)
]
Every web page is shipped to the find_layout_items operate as an unbiased job.
Full workflow
The agent’s workflow is constructed utilizing a StateGraph, linking the picture extraction and structure detection steps right into a unified pipeline ->
from langgraph.graph import StateGraph, START, ENDclass DocumentParsingAgent:
def build_agent(self):
"""
Construct the agent workflow utilizing a state graph.
"""
builder = StateGraph(DocumentLayoutParsingState)
# Add nodes for picture extraction and structure merchandise detection
builder.add_node("get_images", self.get_images)
builder.add_node("find_layout_items", self.find_layout_items)
# Outline the move of the graph
builder.add_edge(START, "get_images")
builder.add_conditional_edges("get_images", self.continue_to_find_layout_items)
builder.add_edge("find_layout_items", END)
self.graph = builder.compile()
To run the agent on a pattern PDF we do:
if __name__ == "__main__":
_state = DocumentLayoutParsingState(
document_path="path/to/doc.pdf"
)
agent = DocumentParsingAgent()# Step 1: Extract photos from PDF
result_images = agent.get_images(_state)
_state.pages_as_base64_jpeg_images = result_images["pages_as_base64_jpeg_images"]
# Step 2: Course of the primary web page (for example)
result_layout = agent.find_layout_items(
FindLayoutItemsInput(
base64_jpeg=_state.pages_as_base64_jpeg_images[0],
page_number=0,
document_path=_state.document_path,
)
)
# Show the outcomes
for merchandise in result_layout["documents"]:
print(merchandise.page_content)
print(merchandise.metadata["element_type"])
This leads to a parsed, segmented, and summarized illustration of the PDF, which is the enter of the second agent we’ll construct subsequent.
RAG Agent
This second agent handles the indexing and retrieval half. It saves the paperwork of the earlier agent right into a vector database and makes use of the outcome for retrieval. This may be cut up into two seprate steps, indexing and retrieval.
Step 1: Indexing the Break up Doc
Utilizing the summaries generated, we vectorize them and save them in a ChromaDB database:
class DocumentRAGAgent:
def index_documents(self, state: DocumentRAGState):
"""
Index the parsed paperwork into the vector retailer.
"""
assert state.paperwork, "Paperwork ought to have at the very least one aspect"
# Verify if the doc is already listed
if self.vector_store.get(the place={"document_path": state.document_path})["ids"]:
logger.information(
"Paperwork for this file are already listed, exiting this node"
)
return # Skip indexing if already performed
# Add parsed paperwork to the vector retailer
self.vector_store.add_documents(state.paperwork)
logger.information(f"Listed {len(state.paperwork)} paperwork for {state.document_path}")
The index_documents technique embeds the chunk summaries into the vector retailer. We hold metadata such because the doc path and web page quantity for later use.
Step 2: Dealing with Questions
When a consumer asks a query, the agent searches for essentially the most related chunks within the vector retailer. It retrieves the summaries and corresponding web page photos for contextual understanding.
class DocumentRAGAgent:
def answer_question(self, state: DocumentRAGState):
"""
Retrieve related chunks and generate a response to the consumer's query.
"""
# Retrieve the top-k related paperwork based mostly on the question
relevant_documents: checklist[Document] = self.retriever.invoke(state.query)# Retrieve corresponding web page photos (keep away from duplicates)
photos = checklist(
set(
[
state.pages_as_base64_jpeg_images[doc.metadata["page_number"]]
for doc in relevant_documents
]
)
)
logger.information(f"Responding to query: {state.query}")
# Assemble the immediate: Mix photos, related summaries, and the query
messages = (
[{"mime_type": "image/jpeg", "data": base64_jpeg} for base64_jpeg in images]
+ [doc.page_content for doc in relevant_documents]
+ [
f"Answer this question using the context images and text elements only: {state.question}",
]
)
# Generate the response utilizing the LLM
response = self.mannequin.generate_content(messages)
return {"response": response.textual content, "relevant_documents": relevant_documents}
The retriever queries the vector retailer to seek out the chunks most related to the consumer’s query. We then construct the context for the LLM (Gemini), which mixes textual content chunks and pictures as a way to generate a response.
The total agent Workflow
The agent workflow has two levels, an indexing stage and a query answering stage:
class DocumentRAGAgent:
def build_agent(self):
"""
Construct the RAG agent workflow.
"""
builder = StateGraph(DocumentRAGState)
# Add nodes for indexing and answering questions
builder.add_node("index_documents", self.index_documents)
builder.add_node("answer_question", self.answer_question)
# Outline the workflow
builder.add_edge(START, "index_documents")
builder.add_edge("index_documents", "answer_question")
builder.add_edge("answer_question", END)
self.graph = builder.compile()
Instance run
if __name__ == "__main__":
from pathlib import Path# Import the primary agent to parse the doc
from document_ai_agents.document_parsing_agent import (
DocumentLayoutParsingState,
DocumentParsingAgent,
)
# Step 1: Parse the doc utilizing the primary agent
state1 = DocumentLayoutParsingState(
document_path=str(Path(__file__).mother and father[1] / "knowledge" / "docs.pdf")
)
agent1 = DocumentParsingAgent()
result1 = agent1.graph.invoke(state1)
# Step 2: Arrange the second agent for retrieval and answering
state2 = DocumentRAGState(
query="Who was acknowledged on this paper?",
document_path=str(Path(__file__).mother and father[1] / "knowledge" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
agent2 = DocumentRAGAgent()
# Index the paperwork
agent2.graph.invoke(state2)
# Reply the primary query
result2 = agent2.graph.invoke(state2)
print(result2["response"])
# Reply a second query
state3 = DocumentRAGState(
query="What's the macro common when fine-tuning on PubLayNet utilizing M-RCNN?",
document_path=str(Path(__file__).mother and father[1] / "knowledge" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
result3 = agent2.graph.invoke(state3)
print(result3["response"])
With this implementation, the pipeline is full for doc processing, retrieval, and query answering.
Let’s stroll by means of a sensible instance utilizing the doc LLM & Adaptation.pdf , a set of 39 slides containing textual content, equations, and figures (CC BY 4.0).
Step 1: Parsing and summarizing the Doc (Agent 1)
- Execution Time: Parsing the 39-page doc took 29 seconds.
- End result: Agent 1 produces an listed doc consisting of chunk summaries and base64-encoded JPEG photos of every web page.
Step 2: Questioning the Doc (Agent 2)
We ask the next query:
“Clarify LoRA, give the related equations”
End result:
Retrieved pages:
