Using Chat Models in LangChain

Who this is for and what you will be able to do

This guide is for developers who write some Python and want one codebase to work with several model providers. By the end you should be able to initialize a chat model with init_chat_model, call it with invoke, stream, and batch, bind tools to it, request structured output, limit request rates, and point the same code at an OpenAI-compatible server.

What the model layer gives you

LangChain puts chat models from many providers behind one standard interface. Each provider package implements that interface and passes model names directly to the provider's API, so a new model name works without waiting for a LangChain update, and switching providers requires few code changes^[1].

Inside an agent, the model is the reasoning engine. It decides which tools to call, interprets the results, and decides when to give a final answer^[1]. A chat model can also be called on its own for text generation, classification, or extraction^[1].

Initialize a model

The simplest route is init_chat_model^[1]. Pass parameters as keyword arguments:

model = init_chat_model(
    "claude-sonnet-4-6",
    # Kwargs passed to the model:
    temperature=0.7,
    timeout=30,
    max_tokens=1000,
    max_retries=6,  # Default; increase for unreliable networks
)

The model argument can be a plain name or a provider:model string, such as openai:o1^[1]. temperature controls randomness: higher values give more creative responses, lower values more deterministic ones^[1]. max_tokens limits the total number of tokens in the response, which effectively caps the output length^[1]. timeout is the maximum time in seconds to wait before a request is canceled^[1].

Retries on unreliable networks

max_retries defaults to 6^[1]. When a request fails because of a network timeout or a rate limit, the model retries it with exponential backoff and jitter. Network errors, rate limits (429) and server errors (5xx) are retried automatically. Client errors such as 401 (unauthorized) or 404 are not^[1].

For long-running agent tasks on unreliable networks, raise max_retries to 10 to 15 and add a checkpointer so that progress survives failures^[1].

Three ways to call a model

invoke() takes a single message or a list of messages and returns an AIMessage once the full response has been generated^[1].

stream() returns AIMessageChunk objects, each holding part of the output, so long answers can be displayed as they are generated^[1]. Adding the chunks together rebuilds the full message, using the pattern full = chunk if full is None else full + chunk^[1].

batch() sends a set of independent requests and parallelizes them on the client side. It is not the same thing as the batch APIs that providers such as OpenAI and Anthropic offer^[1]. If you want each result as soon as it finishes, use batch_as_completed(). Results may arrive out of order, so each one carries its input index^[1]. To cap the number of parallel calls, set max_concurrency in the config:

model.batch(
    list_of_inputs,
    config={
        'max_concurrency': 5,  # Limit to 5 parallel calls
    }
)

Binding tools

Tool calling is also called function calling^[1]. Turn a plain Python function into a tool with the @tool decorator:

from langchain.tools import tool
@tool
def get_weather(location: str) -> str:
    """Get the weather at a location."""
    return f"It's sunny in {location}."

bind_tools makes the tools available to the model, which can then choose to call any of them^[1]. The model only requests a call; outside an agent, your code must run the requested tool and return the result to the model in a ToolMessage, whose tool_call_id matches the ID of the call it answers^[1]. The basic loop looks like this:

# Bind (potentially multiple) tools to the model
tools = [get_weather]
tools_by_name = {tool.name: tool for tool in tools}
model_with_tools = model.bind_tools(tools)
# Step 1: Model generates tool calls
messages = [{"role": "user", "content": "What's the weather in Boston?"}]
ai_msg = model_with_tools.invoke(messages)
messages.append(ai_msg)
# Step 2: Execute tools and collect results
for tool_call in ai_msg.tool_calls:
    # Execute the tool with the generated arguments
    tool = tools_by_name[tool_call["name"]]
    tool_result = tool.invoke(tool_call)
    messages.append(tool_result)
# Step 3: Pass results back to model for final response
final_response = model_with_tools.invoke(messages)
print(final_response.text)

Most models that support tool calling enable parallel tool calls by default. OpenAI and Anthropic allow you to turn this off by passing parallel_tool_calls=False to bind_tools^[1].

Structured output

with_structured_output() constrains the response to a schema, which can be a Pydantic model, a TypedDict, or a JSON Schema. Pydantic models validate automatically; TypedDict and JSON Schema require manual validation^[1]. The example below defines the fields of a movie with Pydantic:

from pydantic import BaseModel, Field
class Movie(BaseModel):
    """A movie with details."""
    title: str = Field(description="The title of the movie")
    year: int = Field(description="The year the movie was released")
    director: str = Field(description="The director of the movie")
    rating: float = Field(description="The movie's rating out of 10")
model_with_structure = model.with_structured_output(Movie)
response = model_with_structure.invoke("Provide details about the movie Inception")

The method parameter accepts values such as 'json_schema'. With 'json_mode', the output is valid JSON, but the schema has to be described in the prompt^[1]. Passing include_raw=True also returns the raw AIMessage, which carries metadata such as token counts^[1].

Handling a refusal

Providers report a refusal as a successful response, not as an error. By default, with_structured_output raises ModelRefusalError. With include_raw=True, the error is returned in parsing_error instead^[1].

Rate limits

InMemoryRateLimiter limits the number of requests per unit of time. It is thread safe within one process and is passed to the model through the rate_limiter parameter^[1]. The following configuration allows one request every 10 seconds, with bursts of up to 10:

from langchain.rate_limiters import InMemoryRateLimiter
rate_limiter = InMemoryRateLimiter(
    requests_per_second=0.1,  # 1 request every 10s
    check_every_n_seconds=0.1,  # Check every 100ms whether allowed to make a request
    max_bucket_size=10,  # Controls the maximum burst size.
)

The limiter counts requests only. If a provider's limits also depend on request size, it cannot enforce them^[1].

Connecting to OpenAI-compatible servers

Providers such as Together AI and vLLM offer OpenAI-compatible APIs. Use model_provider="openai" with the matching base_url and api_key^[1]:

model = init_chat_model(
    model="MODEL_NAME",
    model_provider="openai",
    base_url="BASE_URL",
    api_key="YOUR_API_KEY",
)

Note that model_provider="openai" targets the official OpenAI API specification. Provider-specific fields returned through routers or proxies may not be extracted or preserved. For OpenRouter and LiteLLM, use their dedicated integrations instead^[1].

Next steps

For the full list of parameters each provider supports, check the official models documentation^[1].

Adapted from LangChain's official documentation page on models^[1], released under the MIT License.