fault-tolerance

73 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Retries automatically re-run failed nodes based on exception type

A retry policy automatically re-runs a failed node attempt based on exception type and backoff settings. Pass retry_policy=RetryPolicy() to add_node() (Python) or retryPolicy to addNode() (JavaScript) to enable retries.

Timeouts cap how long a single node attempt may run

The timeout parameter on add_node() (Python) or addNode() (JavaScript) caps how long a single node attempt may run. Pass a number (seconds in Python, milliseconds in JavaScript), a timedelta (Python), or a TimeoutPolicy object for separate run and idle limits. Node timeouts only apply to async nodes; sync nodes with a timeout are rejected at compile time.

Error handlers run after node retries are exhausted

An error handler runs after a node fails and all retries are exhausted. It receives the current state and can update it or route to a different node using Command. Error handlers enable compensation flows (Saga patterns) for graceful recovery rather than aborting the entire graph.

Retry policy default exceptions - Python

In Python, by default retry_on uses default_retry_on, which retries on any exception except: ValueError, TypeError, ArithmeticError, ImportError, LookupError, NameError, SyntaxError, RuntimeError, ReferenceError, StopIteration, StopAsyncIteration, OSError. For HTTP libraries like requests and httpx, it only retries on 5xx status codes. NodeTimeoutError is retryable by default.

Retry policy default exceptions - JavaScript

In JavaScript, retries are opt-in and only occur when retryPolicy is configured. If retryPolicy omits retryOn, LangGraph uses a built-in handler that retries thrown errors except: AbortError (error.name === 'AbortError'), cancellation errors (error.message starts with 'Cancel' or 'AbortError'), GraphValueError (error.name match), aborted connections (error.code === 'ECONNABORTED'), HTTP client errors with status 400, 401, 402, 403, 404, 405, 406, 407, or 409, and OpenAI-style quota errors (error.error?.code === 'insufficient_quota'). Other HTTP statuses including 408 and 5xx are retryable unless overridden. NodeTimeoutError is retryable when a retry policy is configured.

RetryPolicy parameters - Python

Python RetryPolicy parameters: max_attempts (int, default 3) - maximum number of attempts including the first; initial_interval (float, default 0.5) - seconds before the first retry; backoff_factor (float, default 2.0) - multiplier applied to interval after each retry; max_interval (float, default 128.0) - maximum seconds between retries; jitter (bool, default True) - add random jitter to the interval; retry_on (type[Exception] | Sequence[type[Exception]] | Callable[[Exception], bool], default default_retry_on) - exceptions to retry on, or a callable returning True for retryable exceptions.

RetryPolicy parameters - JavaScript

JavaScript retryPolicy parameters: maxAttempts (number, default 3) - maximum number of attempts including the first; initialInterval (number, default 500) - milliseconds before the first retry; backoffFactor (number, default 2.0) - multiplier applied to interval after each retry; maxInterval (number, default 128000) - maximum milliseconds between retries; jitter (boolean, default true) - add random jitter to the interval; retryOn ((error: unknown) => boolean, default built-in handler when policy is set) - callable returning true for retryable exceptions; logWarning (boolean, default true) - whether to log a warning when a retry is attempted.

Inspect retry state with runtime.executionInfo.nodeAttempt - JavaScript

Use executionInfo inside a node to inspect the current attempt number. Access runtime.executionInfo?.nodeAttempt (number, 1-indexed) to get current attempt number. Also available: nodeFirstAttemptTime (number | undefined, Unix timestamp in ms of first attempt, constant across retries), threadId (string | undefined, undefined without checkpointer), runId (string | undefined, undefined when not provided in config), checkpointId (string), checkpointNs (string), taskId (string). executionInfo is available even without a retry policy; nodeAttempt defaults to 1.

Run timeout is a hard wall-clock cap on node execution

run_timeout (Python) or runTimeout (JavaScript) is a hard wall-clock cap on a single node attempt. It is never refreshed, regardless of node activity. When exceeded, LangGraph raises NodeTimeoutError, clears any writes from the failed attempt, and lets the retry policy decide whether to retry.

Idle timeout fires when node stops making observable progress

idle_timeout (Python) or idleTimeout (JavaScript) is a progress-resetting cap that fires only when the node stops making observable progress for the specified duration. Unlike run_timeout, the clock resets whenever the node produces a progress signal. Progress signals include state writes, stream output, child-task scheduling, runtime stream-writer calls, and LangChain callback events.

TimeoutPolicy parameters - Python

Python TimeoutPolicy has two fields: run_timeout (float | None) - hard wall-clock cap in seconds on a single attempt, never refreshed; idle_timeout (float | None) - progress-resetting cap in seconds, fires when node stops making progress. Can set both together; whichever fires first cancels the attempt.

Idle timeout progress signals under refresh_on='auto' - Python

Under default refresh_on='auto', the idle clock resets on: state writes via CONFIG_KEY_SEND, stream output (yielded async stream chunks), child-task scheduling, runtime stream-writer calls, any LangChain callback event from the node or its descendants (LLM tokens, tool calls, chain start/end, etc.).

Idle timeout progress signals under refreshOn='auto' - JavaScript

Under default refreshOn: 'auto', the idle clock resets on: state writes through the graph write path, custom stream output via runtime.writer, child-task scheduling, any LangChain callback event from the node or its descendants (LLM tokens, tool calls, chain start/end, etc.).

Heartbeat mode for strict idle definition

Set refresh_on='heartbeat' (Python) or refreshOn: 'heartbeat' (JavaScript) to narrow the refresh source to explicit runtime.heartbeat() calls only. This is useful when you want a strict idle definition that isn't reset by chatty subordinates. Call runtime.heartbeat() to manually reset the idle clock during long-running work that doesn't naturally emit progress signals. runtime.heartbeat() is a no-op outside an idle-timed attempt, so you can call it unconditionally.

NodeTimeoutError structure and attributes - JavaScript

When a timeout fires, LangGraph raises NodeTimeoutError with attributes: node (string) - name of the timed-out node; elapsed (number) - milliseconds elapsed before timeout fired; kind ('idle' | 'run') - which timeout fired; timeout (number) - the value in milliseconds of the timeout that fired; idleTimeout (number | undefined) - configured idle timeout in milliseconds, if any; runTimeout (number | undefined) - configured run timeout in milliseconds, if any. Use isNodeTimeoutError(error) to narrow caught errors in TypeScript.

NodeTimeoutError is retryable by default

NodeTimeoutError is retryable by default in both Python and JavaScript. Combining timeout with a retry policy works out of the box—the timeout clock resets on each new attempt, and writes from a timed-out attempt are cleared before the next retry.

Dynamic timeouts with Send override static timeouts

When using Send to dispatch nodes dynamically (for example, in map-reduce patterns), you can pass a timeout directly on the Send to override the target node's static timeout for that specific push. If the timeout is omitted on the Send, the target node's timeout (set at add_node/addNode time) applies. This lets you set a default timeout on the node and tighten it for individual calls.

Fault tolerance composition order: attempt, timeout, retry, error handler

When a node attempt raises any exception (including NodeTimeoutError from a timeout), the retry policy decides whether to retry. Only after retries are exhausted does the error handler run. This is a fixed composition order.

Error handler receives NodeError with node name and exception

Error handlers receive failure context through a typed error: NodeError parameter (injected by type annotation). NodeError is a dataclass/class with two fields: node (str/string) - name of the failed node; error (BaseException/Error) - the exception raised by the failed node. The error: NodeError parameter is opt-in; handlers that don't need failure context can use simpler signatures like (state) or (state, runtime).

Error handlers can route with Command for Saga patterns

Error handlers can return a Command to update state and route to a specific node, enabling Saga/compensation patterns. This allows recovery flows where after a node fails and retries are exhausted, the handler can compensate by updating state and routing to a different node instead of aborting the graph.

Failure provenance is checkpointed for error handlers

Failure provenance is checkpointed. If the graph is interrupted or the process crashes after a node fails but before the handler completes, the handler sees the same NodeError context when the graph resumes from its checkpoint.

interrupt() does not route to error handler

interrupt() raised inside a node is not routed to the error handler. Interrupts use the GraphBubbleUp mechanism to pause graph execution for human-in-the-loop workflows, bypassing both retry policies and error handlers. The graph pauses as usual.

Subgraph exceptions surface to parent node error handler

If a node wraps a subgraph and the subgraph raises an unhandled exception, that exception surfaces to the parent node. If the parent node has an error handler, the handler fires with the subgraph's exception in error.error.

setNodeDefaults configures fault tolerance for all nodes - JavaScript

Use setNodeDefaults() on StateGraph to configure retryPolicy, errorHandler, timeout, and cachePolicy once for all nodes instead of repeating them on every addNode() call. Both stepA and stepB now share the same retry policy, error handler, and timeout without duplication.

Per-node values override set_node_defaults defaults

Per-node values passed directly to add_node()/addNode() always override the defaults set by set_node_defaults()/setNodeDefaults(). Defaults are resolved at compile() time, so you can call set_node_defaults()/setNodeDefaults() before or after add_node()/addNode() in any order.

Default error handler for marking external processes failed

The error_handler/errorHandler default is particularly valuable when every graph run maps to an external process (for example a background job row) and any unhandled node failure should mark that process as failed, without repeating error_handler/errorHandler on every add_node/addNode. Per-node handlers still take precedence when a step needs its own logic.

Graph defaults applicability matrix for error-handler nodes

Not all defaults apply to all node types. Error-handler nodes (those registered via add_node/addNode with error_handler/errorHandler) are excluded from certain defaults: retry_policy/retryPolicy applies to both regular and error-handler nodes (handlers should be retried on transient failures); timeout applies to both (stuck handlers should be cancelled); error_handler/errorHandler applies to regular nodes only (handlers must never catch themselves); cache_policy/cachePolicy applies to regular nodes only (caching handler results is unsafe).

Graph defaults are not inherited by subgraphs

Defaults set on a parent graph are not inherited by subgraphs. Each graph maintains its own defaults.

Functional API timeout and retry_policy parameters - Python

The same timeout and retry_policy parameters are available on @task and @entrypoint in the functional API. The behavior is identical to add_node: NodeTimeoutError is raised on timeout, buffered writes are cleared, and the retry policy decides whether to retry.

Functional API timeout and retry parameters - JavaScript

The timeout option is available on task and entrypoint in the JavaScript/TypeScript functional API; task also accepts a retry option (not retryPolicy). The behavior matches addNode: NodeTimeoutError is raised on timeout, buffered writes are cleared, and the retry policy decides whether to retry. Error handlers are not available on task/entrypoint; use StateGraph.addNode(..., { errorHandler }) instead.

Graceful shutdown with RunControl and requestDrain - JavaScript

Cooperative shutdown lets you stop an in-flight graph run after the current superstep completes and save a resumable checkpoint. Create a RunControl and pass it as control to invoke or stream. Call requestDrain() from any context to signal that the run should stop. Catch GraphDrained exception when thrown. Resume a drained run with invoke(null, config) using the same thread_id.

Drain semantics and behaviors - JavaScript

Drain is cooperative and operates between supersteps, never preempting work already running. Behaviors: node mid-execution runs to completion, drain takes effect on next superstep; node with retry policy currently retrying - retry loop runs to exhaustion or success, drain takes effect after; graph finishes naturally on same tick as drain - returns normally, inspect control.drainRequested to distinguish from normal run; more supersteps remain - raises GraphDrained(reason), checkpoint is saved and resumable; subgraph requests drain - GraphDrained bubbles up through parent and stops it at its own next superstep boundary.

Read drain state inside a node - JavaScript

Access drain state through the runtime parameter to adjust node behavior before the superstep boundary is reached. Check runtime.control?.drainRequested (boolean) and runtime.control.drainReason (string) to skip expensive work and return a minimal result if drain is requested.

SIGTERM hook pattern for graceful shutdown - JavaScript

Recommended pattern for handling process shutdown: create RunControl, register SIGTERM event listener that calls control.requestDrain('sigterm'), wrap invoke call in try-catch to catch GraphDrained, resume on next startup with the same config. Note: requestDrain() does not cancel in-flight async work. For a hard upper bound, pair drain with a graceful timeout and an AbortSignal.

Limitations: timeouts are async-only - Python

Sync nodes with a timeout are rejected at compile time in Python. To wrap blocking I/O, use asyncio.to_thread inside an async node.

Limitations: error handlers are StateGraph-only - JavaScript

Error handlers are available on StateGraph.addNode only, not the base Graph class. Error handlers are not available on task/entrypoint in the JavaScript/TypeScript SDK.

One error handler per node maximum

Each node can have at most one error_handler/errorHandler.

Handler failures bubble up without recovery

If the error handler itself raises an exception, that exception propagates as if the node had no handler.

Retry policy custom logic with default_retry_on - Python

Pass a callable or exception type to retry_on. Import default_retry_on to extend the default behavior. Example: define custom_retry_on(exc) that checks for MyCustomError and returns False, otherwise calls default_retry_on(exc), then pass it to RetryPolicy(max_attempts=3, retry_on=custom_retry_on).

Retry policy custom logic - JavaScript

Pass a callable to retryOn. Unlike Python, there is no exported defaultRetryOn helper—implement your own predicate that checks error instanceof MyCustomError and returns false, otherwise returns true for other errors.

Retry policy - Python code example

Example: from langgraph.types import RetryPolicy; builder.add_node('call_api', call_api, retry_policy=RetryPolicy(max_attempts=3))

Retry policy - JavaScript code example

Example: const graph = new StateGraph(State).addNode('callApi', callApi, { retryPolicy: { maxAttempts: 3 } }).compile();

Timeout simple cap - Python code examples

Examples: builder.add_node('call_model', call_model, timeout=60); builder.add_node('call_model', call_model, timeout=timedelta(minutes=2));

Timeout TimeoutPolicy - Python code example

Example: builder.add_node('call_model', call_model, timeout=TimeoutPolicy(run_timeout=120, idle_timeout=30))

Timeout simple cap - JavaScript code example

Example: new StateGraph(State).addNode('callModel', callModel, { timeout: 60_000 });

Timeout TimeoutPolicy - JavaScript code example

Example: new StateGraph(State).addNode('callModel', callModel, { timeout: { runTimeout: 120_000, idleTimeout: 30_000 } });

Runtime heartbeat for manual idle clock reset - JavaScript code example

Example of long-running node with heartbeat: const longRunningNode = async (state: typeof State.State, runtime: Runtime<typeof State>) => { for (const batch of fetchBatches()) { process(batch); runtime.heartbeat?.(); // Reset idle clock } return { result: 'done' }; };

Error handler with Command routing - JavaScript code example

Example of error handler that compensates: const paymentErrorHandler = (state: typeof State.State, error: NodeError) => new Command({ update: { status: `compensated: ${error.error.message}` }, goto: 'finalize', });

set_node_defaults graph-wide configuration - Python code example

Example: graph = (StateGraph(State) .set_node_defaults( retry_policy=RetryPolicy(max_attempts=3), error_handler=default_error_handler, timeout=TimeoutPolicy(run_timeout=30), ) .add_node('step_a', step_a) .add_node('step_b', step_b) .add_edge(START, 'step_a') .compile())

setNodeDefaults graph-wide configuration - JavaScript code example

Example: const graph = new StateGraph(State) .setNodeDefaults({ retryPolicy: { maxAttempts: 3 }, errorHandler: defaultErrorHandler, timeout: { runTimeout: 30_000 }, cachePolicy: { ttl: 60 }, }) .addNode('stepA', stepA) .addNode('stepB', stepB) .addEdge(START, 'stepA') .compile();

Functional API timeout and retry_policy - Python code example

Example: @task( timeout=TimeoutPolicy(idle_timeout=30), retry_policy=RetryPolicy(max_attempts=3), ) async def call_api(url: str) -> str: response = await fetch(url) return response.text

Functional API timeout and retry - JavaScript code example

Example: const callApi = task( { name: 'callApi', timeout: { idleTimeout: 30_000 }, retry: { maxAttempts: 3 }, }, async (url: string) => { const response = await fetch(url); return response.text(); } );

RunControl and graceful shutdown - Python code example

Example: from langgraph.runtime import RunControl from langgraph.errors import GraphDrained control = RunControl() try: result = graph.invoke(inputs, config, control=control) except GraphDrained as e: print(f'Drained: {e.reason}')

RunControl and graceful shutdown - JavaScript code example

Example: import { RunControl, GraphDrained } from '@langchain/langgraph'; const control = new RunControl(); try { const result = await graph.invoke(inputs, { ...config, control }); } catch (e) { if (e instanceof GraphDrained) { console.log(`Drained: ${e.reason}`); } }

Inspect and act on node attempt number - Python code example

Example: from langgraph.runtime import Runtime def my_node(state: State, runtime: Runtime) -> State: if runtime.execution_info.node_attempt > 1: return {'result': call_fallback_api()} return {'result': call_primary_api()}

Inspect and act on node attempt number - JavaScript code example

Example: const myNode = async (state: typeof State.State, runtime: Runtime<typeof State>) => { if ((runtime.executionInfo?.nodeAttempt ?? 1) > 1) { return { result: await callFallbackApi() }; } return { result: await callPrimaryApi() }; };

Dynamic timeouts with Send - Python code example

Example: from langgraph.types import Send, TimeoutPolicy def fan_out(state: OverallState): return [ Send('process_item', {'item': item}, timeout=TimeoutPolicy(idle_timeout=15)) for item in state['items'] ]

Dynamic timeouts with Send - JavaScript code example

Example: import { Send } from '@langchain/langgraph'; const fanOut = (state: typeof State.State) => state.items.map( (item) => new Send('processItem', { item }, { timeout: { idleTimeout: 15_000 } }) );

Inspect drain state in node - JavaScript code example

Example: import { type Runtime } from '@langchain/langgraph'; const myNode = async (state: typeof State.State, runtime: Runtime<typeof State>) => { if (runtime.control?.drainRequested) { return { status: 'skipped', reason: runtime.control.drainReason }; } return { status: await doWork() }; };

Error handler with RunnableConfig access - Python

Error handlers can accept RunnableConfig as an optional third argument if you need access to config values such as thread_id. Example signature: def mark_process_failed(state: State, error: NodeError, config: RunnableConfig) -> State: thread_id = config['configurable'].get('thread_id'); return {'status': f'failed on thread {thread_id}: {error.error}'}

Give your agent this brain