A document analysis agent with LangGraph

Who this is for and what you will be able to do

If you want an agent that reads documents such as notes or photos of paper, uses tools and answers questions about them, this guide walks through a working design in LangGraph. By the end you should be able to define an agent state, give the model two tools, connect an assistant node and a tools node into a loop, and run the graph on a text question and an image question.

What the agent does

The example agent can process image documents, extract text with a vision language model, perform calculations, analyze content and give concise summaries, and follow specific instructions about a document^[1]. The example is written as a butler named Alfred who serves Mr. Wayne, so the prompts sound like a butler. You can change the persona to fit your own project.

The graph has two nodes. The assistant node is the model, with tools bound to it. The tools node runs whatever tool the model asked for. The flow loops between them while the model keeps calling tools^[1].

Preparation

The example runs in a notebook. Install the packages with the magic command below^[1]:

%pip install langgraph langchain_openai langchain_core

The example uses OpenAI's gpt-4o model through ChatOpenAI, so you need an OpenAI API key before the calls will succeed. The same class is used for the vision model and the text model^[1]:

vision_llm = ChatOpenAI(model="gpt-4o")

Step 1: define the state

The state holds the state of one run. Here it has two fields: the path to the input file, which is a PDF or PNG, and the message list^[1]:

class AgentState(TypedDict):
    # The document provided
    input_file: Optional[str]  # Contains file path (PDF/PNG)
    messages: Annotated[list[AnyMessage], add_messages]

The add_messages annotation is the part to notice. AnyMessage is a LangChain class for messages, and add_messages is an operator that adds the latest message instead of overwriting the list. LangGraph lets you attach operators to state fields in this way to control how updates combine^[1].

Step 2: prepare the tools

The first tool reads text from an image. It base64-encodes the file and sends it to the vision model. The prompt asks the model to return only the extracted text, with no explanations. The start of the function looks like this^[1]:

def extract_text(img_path: str) -> str:
    """
    Extract text from an image file using a multimodal model.
    
    Master Wayne often leaves notes with his training regimen or meal plans.
    This allows me to properly analyze the contents.
    """
    all_text = ""
    try:
        # Read image and encode as base64
        with open(img_path, "rb") as image_file:
            image_bytes = image_file.read()

        image_base64 = base64.b64encode(image_bytes).decode("utf-8")

If the call fails, the tool prints the error and returns an empty string, which the example comment describes as how a butler should handle errors^[1].

The second tool is a plain division function. Both tools are bound to the model with parallel tool calls turned off^[1]:

def divide(a: int, b: int) -> float:
    """Divide a and b - for Master Wayne's occasional calculations."""
    return a / b

# Equip the butler with tools
tools = [
    divide,
    extract_text
]

llm = ChatOpenAI(model="gpt-4o")
llm_with_tools = llm.bind_tools(tools, parallel_tool_calls=False)

Step 3: write the assistant node

The model needs to know what each tool does, so the assistant node writes a textual description of the tools into its system prompt. The description lists the name, arguments and return value of each tool^[1]:

def assistant(state: AgentState):
    # System message
    textual_description_of_tool="""
extract_text(img_path: str) -> str:
    Extract text from an image file using a multimodal model.

    Args:
        img_path: A local image file path (strings).

    Returns:
        A single string containing the concatenated text extracted from each image.
divide(a: int, b: int) -> float:
    Divide a and b
"""

The node then builds the system message and returns the model's reply together with the input file path. The message tells the model that it is a helpful butler named Alfred, and it names the currently loaded image^[1]:

    image=state["input_file"]
    sys_msg = SystemMessage(content=f"You are a helpful butler named Alfred that serves Mr. Wayne and Batman. You can analyse documents and run computations with provided tools:\n{textual_description_of_tool} \n You have access to some optional images. Currently the loaded image is: {image}")

    return {
        "messages": [llm_with_tools.invoke([sys_msg] + state["messages"])],
        "input_file": state["input_file"]
    }

Step 4: connect the nodes into a loop

This agent follows the ReAct pattern, often summarized as Reason-Act-Observe. The model reasons about the documents and the request, acts by calling a tool, observes the result, and repeats as needed^[1].

In LangGraph, the graph is built with StateGraph. The example registers both nodes, starts at the assistant, and routes the assistant's output with tools_condition^[1]:

# The graph
builder = StateGraph(AgentState)

# Define nodes: these do the work
builder.add_node("assistant", assistant)
builder.add_node("tools", ToolNode(tools))

# Define edges: these determine how the control flow moves
builder.add_edge(START, "assistant")
builder.add_conditional_edges(
    "assistant",
    # If the latest message requires a tool, route to tools
    # Otherwise, provide a direct response
    tools_condition,
)
builder.add_edge("tools", "assistant")
react_graph = builder.compile()

# Show the butler's thought process
display(Image(react_graph.get_graph(xray=True).draw_mermaid_png()))

tools_condition sends the flow to the tools node when the assistant's output is a tool call. Otherwise the flow goes to END and the run stops. The tools node always returns to the assistant, which is what makes the loop^[1].

Run a calculation

The first test is a plain calculation. The code below sends one message through the graph and prints each message^[1]:

messages = [HumanMessage(content="Divide 6790 by 5")]
messages = react_graph.invoke({"messages": messages, "input_file": None})

# Show the messages
for m in messages['messages']:
    m.pretty_print()

In that run, the assistant calls divide(a=6790, b=5), the tool returns 1358.0, and the assistant replies with that result^[1]. Because no file is involved, input_file is None^[1].

Run an image question

The second test asks a question about a note in an image. The graph receives the file name as input_file^[1]:

messages = [HumanMessage(content="According to the note provided by Mr. Wayne in the provided images. What's the list of items I should buy for the dinner menu?")]
messages = react_graph.invoke({"messages": messages, "input_file": "Batman_training_and_meals.png"})

In this run the assistant first calls extract_text with the image path, then answers from the text it gets back^[1]. Reading the image is a separate tool call, so you can inspect that step in the printed messages.

Limits and common problems

• Error handling is minimal in this design. A failed extract_text call returns an empty string, so the model may answer without the note's content. Check the tool output before you trust the answer^[1].
• Keep the tool descriptions accurate. The model only knows the tools through the text in its system prompt^[1].
• The example needs an OpenAI key and calls a paid model. Each run sends the image to that model.
• The graph keeps the message history through add_messages. Long conversations grow the state, so plan for trimming if you need it^[1].

Next steps

The companion notebook is on the Hugging Face Hub, so you can run the same code in Colab^[1]. For your own document butler, start with clear tools for specific document tasks, then add a state tracker and error handling^[1].

Adapted from the LangGraph document analysis lesson in Unit 2 of Hugging Face's open-source *Agents Course*^[1], released under the Apache-2.0 License.