agents/testing

33 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

GenericFakeChatModel for mocking text responses

LangChain provides GenericFakeChatModel from langchain_core.language_models.fake_chat_models for mocking text responses. It accepts an iterator of responses (AIMessage objects or strings) and returns one per invocation. It supports both regular and streaming usage. Each call to invoke() consumes the next item in the iterator.

GenericFakeChatModel example with tool calls

Example using GenericFakeChatModel to mock tool calls: from langchain_core.language_models.fake_chat_models import GenericFakeChatModel model = GenericFakeChatModel(messages=iter([ AIMessage(content="", tool_calls=[ToolCall(name="foo", args={"bar": "baz"}, id="call_1")]), "bar" ])) model.invoke("hello") # AIMessage(content='', ..., tool_calls=[{'name': 'foo', 'args': {'bar': 'baz'}, 'id': 'call_1', 'type': 'tool_call'}]) model.invoke("hello, again!") # AIMessage(content='bar', ...)

fakeModel for JavaScript testing

fakeModel is a builder-style fake chat model in JavaScript that lets you script exact responses (text, tool calls, errors) and assert what the model received. It extends BaseChatModel, so it works anywhere a real model is expected. Responses are queued with .respond() and consumed one per invoke() call in order.

fakeModel basic usage example

Example of basic fakeModel usage: import { fakeModel } from "langchain"; import { AIMessage, HumanMessage } from "@langchain/core/messages"; const model = fakeModel() .respond(new AIMessage("I can help with that.")) .respond(new AIMessage("Here's what I found.")) .respond(new AIMessage("You're welcome!")); const r1 = await model.invoke([new HumanMessage("Can you help?")]); // r1.content === "I can help with that." const r2 = await model.invoke([new HumanMessage("What did you find?")]); // r2.content === "Here's what I found." const r3 = await model.invoke([new HumanMessage("Thanks!")]); // r3.content === "You're welcome!"

fakeModel error handling when responses exhausted

When fakeModel is invoked more times than there are queued responses, it throws a descriptive error indicating which invocation has no response queued. For example, invoking twice when only one response is queued throws: "no response queued for invocation 1".

fakeModel error simulation at specific turns

Passing an Error to .respond() makes the model throw on that specific invocation. Errors can appear at any position in the queue sequence and will be thrown when that queue entry is consumed.

fakeModel error at specific turn example

Example of fakeModel error simulation at specific turns: import { fakeModel } from "langchain"; import { AIMessage, HumanMessage } from "@langchain/core/messages"; const model = fakeModel() .respond(new Error("rate limit exceeded")) // Turn 1: throws .respond(new AIMessage("Recovered!")); // Turn 2: succeeds try { await model.invoke([new HumanMessage("first")]); } catch (e) { console.log(e.message); // "rate limit exceeded" } const result = await model.invoke([new HumanMessage("retry")]); console.log(result.content); // "Recovered!"

fakeModel alwaysThrow for consistent errors

.alwaysThrow() makes every invocation throw the same error, regardless of the queue. This is useful for testing error handling and retry logic when all calls should fail.

fakeModel alwaysThrow example

Example of fakeModel .alwaysThrow(): import { fakeModel } from "langchain"; import { HumanMessage } from "@langchain/core/messages"; const model = fakeModel().alwaysThrow(new Error("service unavailable")); await model.invoke([new HumanMessage("a")]); // throws "service unavailable" await model.invoke([new HumanMessage("b")]); // throws "service unavailable"

fakeModel dynamic responses with factory functions

.respond() accepts a function that computes the response based on the input messages. The function receives the full message array and returns either a BaseMessage or an Error. Each function is a single queue entry, consumed once. To reuse the same dynamic logic for multiple turns, queue multiple respond function calls.

fakeModel factory function example

Example of fakeModel with factory functions: import { fakeModel } from "langchain"; import { AIMessage, HumanMessage } from "@langchain/core/messages"; const model = fakeModel() .respond((messages) => { const last = messages[messages.length - 1].text; return new AIMessage(`You said: ${last}`); }); const result = await model.invoke([new HumanMessage("hello")]); console.log(result.content); // "You said: hello" // Factory functions can also return errors: const model2 = fakeModel() .respond((messages) => { const content = messages[messages.length - 1].text; if (content.includes("forbidden")) { return new Error("Content policy violation"); } return new AIMessage("OK"); }); await model2.invoke([new HumanMessage("forbidden topic")]); // throws "Content policy violation"

fakeModel call recording and inspection

fakeModel records every invocation, including the messages and options passed to the model. Access call history via model.callCount (number of invocations) and model.calls (array of call objects). Each call object contains messages and options. Calls are recorded even when the model throws errors.

fakeModel call recording example

Example of fakeModel call recording: import { fakeModel } from "langchain"; import { AIMessage, HumanMessage } from "@langchain/core/messages"; const model = fakeModel() .respond(new AIMessage("first")) .respond(new AIMessage("second")); await model.invoke([new HumanMessage("question 1")]); await model.invoke([new HumanMessage("question 2")]); console.log(model.callCount); // 2 console.log(model.calls[0].messages[0].content); // "question 1" console.log(model.calls[1].messages[0].content); // "question 2" // Calls are recorded even when the model throws: const model2 = fakeModel().respond(new Error("boom")); try { await model2.invoke([new HumanMessage("will fail")]); } catch { // error handled } console.log(model2.callCount); // 1 console.log(model2.calls[0].messages[0].content); // "will fail"

Full tool-calling agent test example with vitest

Full example testing a tool-calling agent with vitest: import { describe, test, expect } from "vitest"; import { fakeModel } from "langchain"; import { AIMessage, HumanMessage, ToolMessage } from "@langchain/core/messages"; import { tool } from "@langchain/core/tools"; import { z } from "zod"; const getWeather = tool( async ({ city }) => `72°F and sunny in ${city}`, { name: "get_weather", description: "Get weather for a city", schema: z.object({ city: z.string() }), } ); async function runAgent( model: ReturnType<typeof fakeModel>, input: string ) { const messages: any[] = [new HumanMessage(input)]; const bound = model.bindTools([getWeather]); while (true) { const response = await bound.invoke(messages); messages.push(response); if (!response.tool_calls?.length) { return { messages, finalResponse: response }; } for (const tc of response.tool_calls) { const result = await getWeather.invoke(tc.args); messages.push(new ToolMessage({ content: result as string, tool_call_id: tc.id!, })); } } } describe("weather agent", () => { test("calls get_weather and returns a final answer", async () => { const model = fakeModel() .respondWithTools([ { name: "get_weather", args: { city: "SF" }, id: "call_1" }, ]) .respond(new AIMessage("It's 72°F and sunny in SF!")); const { finalResponse } = await runAgent(model, "Weather in SF?"); expect(finalResponse.content).toBe("It's 72°F and sunny in SF!"); expect(model.callCount).toBe(2); const secondCall = model.calls[1].messages; const toolMsg = secondCall.find((m: any) => m._getType() === "tool"); expect(toolMsg?.content).toContain("72°F and sunny in SF"); }); test("handles model errors gracefully", async () => { const model = fakeModel() .respond(new Error("rate limit")); await expect( runAgent(model, "Weather?") ).rejects.toThrow("rate limit"); expect(model.callCount).toBe(1); }); });

Unit testing agent logic without API calls

Unit tests exercise small, deterministic pieces of an agent in isolation. By replacing the real LLM with an in-memory fake model (fixture), you can script exact responses (text, tool calls, and errors) so tests are fast, free, and repeatable without requiring API keys.

LangSmith tracing for voice agents

To enable LangSmith tracing for voice agent applications, set environment variables: LANGSMITH_TRACING="true" and LANGSMITH_API_KEY="...". This logs traces to track multiple LLM invocations and pipeline steps. Sign up at https://smith.langchain.com to get an API key.

Evaluator function signature

An evaluator is a function that takes agent outputs and optionally reference outputs, both containing a messages field, and returns a dictionary with a key and score. The Python signature is: def evaluator(*, outputs: dict, reference_outputs: dict) -> dict. The TypeScript signature is: function evaluator({ outputs, referenceOutputs }: { outputs: Record<string, any>; referenceOutputs: Record<string, any>; }) -> { key: string; score: any }.

Trajectory match evaluator modes

The create_trajectory_match_evaluator (Python) or createTrajectoryMatchEvaluator (TypeScript) function supports four modes: strict (exact match of message structure and tool calls in same order, message content can differ), unordered (same message structure and tool calls as reference but tool calls can happen in any order), subset (agent calls only tools from reference with no extras), and superset (agent calls at least the reference tools with extras allowed).

Install agentevals package

For Python, install with: pip install -U agentevals or uv add agentevals. For JavaScript/TypeScript, install with: npm install agentevals @langchain/core. Alternatively, clone the AgentEvals repository directly from https://github.com/langchain-ai/agentevals.

Strict trajectory match evaluator

The strict mode ensures trajectories contain identical messages in the same order with the same tool calls, though it allows differences in message content. This is useful when enforcing specific sequences of operations, such as requiring a policy lookup before authorizing an action. Returns a dictionary with keys: key (typically 'trajectory_strict_match'), score (boolean), and comment (optional).

Unordered trajectory match evaluator

The unordered mode allows the same tool calls in any order. This is helpful when verifying that specific information was retrieved but the sequence does not matter, such as an agent checking both weather and events for a city with different tool calls.

Subset and superset trajectory match modes

The superset mode verifies that the agent called at least the tools in the reference trajectory, allowing additional tool calls. The subset mode ensures the agent did not call any tools beyond those in the reference. Both modes match partial trajectories.

Tool args match customization

The trajectory match evaluator can be customized using tool_args_match_mode (Python) or toolArgsMatchMode (TypeScript) property and/or tool_args_match_overrides (Python) or toolArgsMatchOverrides (TypeScript) to customize equality between tool calls. By default, only tool calls with the same arguments to the same tool are considered equal.

LLM-as-judge evaluator creation

Use create_trajectory_llm_as_judge (Python) or createTrajectoryLLMAsJudge (TypeScript) to evaluate the agent's execution path with an LLM. It does not require a reference trajectory, though one can be provided if available. The function accepts parameters: model (string like 'openai:o3-mini') and prompt (such as TRAJECTORY_ACCURACY_PROMPT or TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE).

LLM judge with reference trajectory

When using LLM-as-judge evaluator with a reference trajectory, use TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE as the prompt parameter. Call the evaluator with both outputs and reference_outputs parameters: evaluator(outputs=result['messages'], reference_outputs=reference_trajectory).

Async evaluator support in agentevals

All agentevals evaluators support Python asyncio. Async versions are available by adding 'async' after 'create_' in the function name. For example: create_async_trajectory_llm_as_judge and create_async_trajectory_match_evaluator. These are called with await.

LangSmith environment variables for evals

To run evals in LangSmith, set LANGSMITH_API_KEY to your LangSmith API key and LANGSMITH_TRACING to 'true'.

Pytest integration with LangSmith

Use @pytest.mark.langsmith decorator on test functions. Within the test, use t.log_inputs({}), t.log_outputs({'messages': result['messages']}), and t.log_reference_outputs({'messages': reference_trajectory}) to log data. Run evaluations with: pytest test_trajectory.py --langsmith-output

LangSmith evaluate function with dataset

Create a LangSmith dataset with schema: input (messages array to call agent with) and output (expected message history in agent output). Use client.evaluate(run_agent, data='your_dataset_name', evaluators=[trajectory_evaluator]) where run_agent is a function that accepts inputs and returns agent.invoke(inputs)['messages'].

Vitest/Jest integration with LangSmith

Use langsmith/vitest or langsmith/jest import. Use ls.describe() and ls.test() with parameters: inputs (message objects), referenceOutputs (expected messages), and an async test function. Within the test, use ls.logOutputs({'messages': result.messages}) and call the evaluator. Run with: vitest run test_trajectory.eval.ts or jest test_trajectory.eval.ts

LangSmith evaluate function for TypeScript

Use evaluate() function from langsmith/evaluation. Create a function runAgent(inputs: any) that returns agent.invoke(inputs).messages. Call: await evaluate(runAgent, { data: 'your_dataset_name', evaluators: [trajectoryEvaluator] }).

Evaluations measure agent trajectory execution

Evaluations assess how well an agent performs by scoring its execution trajectory, which is the sequence of messages and tool calls it produces. Unlike integration tests that verify basic correctness, evals score agent behavior against a reference or rubric, making them useful for catching regressions when prompts, tools, or models change.

Two evaluation approaches: trajectory match and LLM judge

Trajectory match uses deterministic comparison when you know expected tool calls, providing fast, cost-free checks. LLM-as-judge uses a qualitative assessment when you want to assess overall quality and reasoning without strict expectations.

Give your agent this brain