observability

62 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

LangSmith for RAG observability

LangSmith traces RAG applications and logs traces for each query. After signing up, set LANGSMITH_TRACING=true and provide LANGSMITH_API_KEY to enable trace logging for inspection of retrieval, tool calls, and model responses.

LangSmith Engine for production agents

LangSmith Engine is recommended for production agents. It monitors traces, detects issues, and proposes fixes.

Detecting in-flight threads: LangSmith tracing monitoring

Use LangSmith tracing to monitor which nodes are being entered and exited in production. This is the most reliable signal that a node or state field is no longer reachable in any active code path.

LangSmith observability prerequisites and setup

To use LangSmith for observability, you need a LangSmith account (free signup available at smith.langchain.com) and a LangSmith API key. Enable tracing by setting environment variables: LANGSMITH_TRACING=true and LANGSMITH_API_KEY=<your-api-key>. By default, traces are logged to the project named 'default'.

What traces represent in LangSmith

Traces are a series of steps that an application takes to go from input to output. Each individual step in a trace is represented by a run. LangSmith can be used to visualize these execution steps, debug locally running applications, evaluate application performance, and monitor applications.

Selective tracing with tracing_context in Python

You can selectively trace specific invocations using LangSmith's tracing_context context manager. When enabled=True is set within the context, that invocation will be traced. Invocations outside the context will not be traced if LANGSMITH_TRACING is not set. Example: with ls.tracing_context(enabled=True): agent.invoke(...) will be traced.

Selective tracing with LangChainTracer in JavaScript

In JavaScript, you can selectively trace specific invocations by instantiating a LangChainTracer and passing it in the callbacks array: const tracer = new LangChainTracer(); await agent.invoke({...}, { callbacks: [tracer] }). Invocations without the tracer callback will not be traced if LANGSMITH_TRACING is not set.

Set custom project name statically with LANGSMITH_PROJECT

You can set a custom project name for your entire application by setting the LANGSMITH_PROJECT environment variable: export LANGSMITH_PROJECT=my-agent-project.

Set custom project name dynamically in Python

You can set the project name programmatically for specific operations using ls.tracing_context(project_name="email-agent-test", enabled=True): response = agent.invoke({"messages": [...]}).

Set custom project name dynamically in JavaScript

In JavaScript, you can set the project name programmatically by passing projectName to LangChainTracer: const tracer = new LangChainTracer({ projectName: "email-agent-test" }); await agent.invoke({...}, { callbacks: [tracer] }).

Add metadata and tags to traces in Python

You can annotate traces with custom metadata and tags by passing them in the config parameter: response = agent.invoke({...}, config={"tags": ["production", "email-assistant", "v1.0"], "metadata": {"user_id": "user_123", "session_id": "session_456", "environment": "production"}}).

Add metadata and tags to traces with tracing_context in Python

The tracing_context context manager also accepts tags and metadata parameters: with ls.tracing_context(project_name="email-agent-test", enabled=True, tags=["production", "email-assistant", "v1.0"], metadata={"user_id": "user_123", "session_id": "session_456", "environment": "production"}): response = agent.invoke({...}).

Add metadata and tags to traces in JavaScript

In JavaScript, pass tags and metadata in the config parameter: await agent.invoke({...}, { config: { tags: ["production", "email-assistant", "v1.0"], metadata: { userId: "user123", sessionId: "session456", environment: "production" } } }).

Prevent logging sensitive data with anonymizers in Python

You can mask sensitive data in traces using LangSmith anonymizers. Create an anonymizer with pattern rules: anonymizer = create_anonymizer([{ "pattern": r"\b\d{3}-?\d{2}-?\d{4}\b", "replace": "<ssn>" }]). Pass it to the Client: tracer_client = Client(anonymizer=anonymizer). Then apply the tracer to the graph: graph = StateGraph(...).compile().with_config({'callbacks': [tracer]}).

Prevent logging sensitive data with anonymizers in JavaScript

In JavaScript, create an anonymizer: const anonymizer = createAnonymizer([{ pattern: /\b\d{3}-?\d{2}-?\d{4}\b/, replace: "<ssn>" }]). Pass it to the Client: const langsmithClient = new Client({ anonymizer }). Then create the tracer: const tracer = new LangChainTracer({ client: langsmithClient }). Apply it to the graph: graph.withConfig({ callbacks: [tracer] }).

LangSmith integration with LangGraph

LangSmith is a platform for tracing, evaluation, prompts, and deployment across frameworks. It integrates with LangGraph to provide observability and deployment capabilities. LangSmith Engine detects issues in LangGraph agent traces and proposes fixes. LangSmith Fleet is a no-code agent builder for templates, integrations, and routine automation.

LangSmith Studio purpose and capabilities

LangSmith Studio is a free visual interface for developing and testing LangChain agents locally. It connects to a locally running agent to show each step the agent takes, including prompts sent to the model, tool calls and their results, and final output. Users can test different inputs, inspect intermediate states, and iterate on agent behavior without additional code or deployment. Studio captures exceptions with surrounding state to help understand what went wrong.

Studio prerequisites and setup requirements

Before setting up Studio, you need: a LangSmith account (free signup available at smith.langchain.com), a LangSmith API key (created following the Create an API key guide), and optionally the ability to disable tracing by setting LANGSMITH_TRACING=false in the .env file. With tracing disabled, no data leaves the local server.

Studio development server hot-reloading capability

The development server supports hot-reloading, allowing changes to prompts or tool signatures in code to be immediately reflected in Studio without restarting. Users can re-run conversation threads from any step to test changes without starting over.

Studio traces and observability in LangSmith

When using Studio, the full execution trace including prompts, tool arguments, return values, and token/latency metrics can be inspected in LangSmith. Studio captures exceptions with the surrounding state to help understand what happened during agent execution.

Observability benefits of separate nodes

Smaller nodes provide more inspection points. You can see exactly what classification the LLM made, what search results were found, what draft was generated, before human review. Easier to debug why agent took certain path and where failures occurred.

Distributed tracing propagates trace context via HTTP headers

When calling a deployed Agent Server from another service, distributed tracing propagates trace context so the entire request appears as a single unified trace in LangSmith. The client infers trace context from the current run and sends it as HTTP headers. The server reads these headers and adds them to the run's config and metadata as configurable values.

RemoteGraph distributed tracing example

from langgraph.graph import StateGraph from langgraph.pregel.remote import RemoteGraph remote_graph = RemoteGraph( "agent", url="<DEPLOYMENT_URL>", distributed_tracing=True, # Enable trace propagation ) def subgraph_node(query: str): # Trace context is automatically propagated return remote_graph.invoke({ "messages": [{"role": "user", "content": query}] })['messages'][-1]['content'] graph = ( StateGraph(str) .add_node(subgraph_node) .add_edge("__start__", "subgraph_node") .compile() ) result = graph.invoke("What's the weather in SF?")

Distributed tracing headers: langsmith-trace and baggage

The headers used for distributed tracing are: langsmith-trace (contains the trace's dotted order) and baggage (specifies the LangSmith project and other optional tags and metadata).

Server must read trace headers from configurable field

To accept distributed trace context, a graph must read the trace headers from the config and set the tracing context. The headers are passed through the configurable field as langsmith-trace and langsmith-project.

Security warning: only propagate distributed tracing headers from trusted services

Distributed tracing headers (langsmith-trace, baggage) are consumed as trusted tracing context. Only configure your server to apply inbound trace context for deployments called by trusted, internal services. If your Agent Server receives requests directly from untrusted third parties or the public internet, do not propagate these headers into the tracing context: strip them at your gateway or proxy instead. Trusting baggage from an external caller lets them influence how your runs are recorded.

Server-side distributed tracing setup with contextlib and langsmith tracing_context

To set up distributed tracing on the server, create a context manager that reads langsmith-trace and langsmith-project from the config's configurable field, along with optional langsmith-metadata and langsmith-tags, then pass them to ls.tracing_context(parent=parent_trace, project_name=parent_project, metadata=metadata, tags=tags).

RemoteGraph distributed_tracing parameter automatically propagates trace headers

When initializing RemoteGraph with distributed_tracing=True, it automatically propagates trace headers on all requests without requiring manual header management.

SDK distributed tracing with run_tree.to_headers()

When using the LangGraph SDK directly, propagate trace headers manually by calling run_tree.to_headers() within a langsmith.trace context and passing the headers to client.runs.stream() as the headers parameter.

SDK distributed tracing example

from langgraph_sdk import get_client import langsmith as ls client = get_client(url="<DEPLOYMENT_URL>") with ls.trace("call_remote_agent", inputs={"query": query}) as rt: headers = rt.to_headers() async for chunk in client.runs.stream( thread_id=None, assistant_id="agent", input={"messages": [{"role": "user", "content": query}]}, stream_mode="values", headers=headers, # Pass trace headers ): pass return chunk

Server-side distributed tracing setup example with contextlib

import contextlib import langsmith as ls from langgraph.graph import StateGraph, MessagesState builder = StateGraph(MessagesState) # ... add nodes and edges ... my_graph = builder.compile() @contextlib.contextmanager async def graph(config): configurable = config.get("configurable", {}) parent_trace = configurable.get("langsmith-trace") parent_project = configurable.get("langsmith-project") metadata = configurable.get("langsmith-metadata") tags = configurable.get("langsmith-tags") with ls.tracing_context(parent=parent_trace, project_name=parent_project, metadata=metadata, tags=tags): yield my_graph

Feedback data model design patterns

Feedback keys can be designed in multiple ways depending on the use case. Separate boolean-style keys (user_liked, user_disliked) group feedback by type, while a single numeric score key (user_score) groups all preference signals under one feedback key. For example, user_score with score=1 can represent user_liked, and user_score with score=-1 can represent user_disliked. More complex rubrics with multiple feedback keys can also be used. The feedback data model is flexible and should be designed for the specific application's needs.

Productionized feedback collection workflow

A productionized feedback collection system follows these steps: (1) Create the run from backend or frontend, (2) Capture the feedback object and store the returned pre-signed URLs, (3) Render feedback controls such as thumbs up/down buttons and feedback forms in the frontend, (4) On feedback submission, POST or GET a feedback URL based on the user's feedback intent, (5) Optionally disable the feedback controls after submission and show confirmation to the user.

Feedback event stream format from Agent Server

The streaming response emits a feedback event with the format: event: feedback, followed by data containing a JSON object where each key matches a value passed in feedback_keys, and each value is a pre-signed URL. For example: {"user_liked":"https://api.smith.langchain.com/api/v1/feedback/tokens/ef19fedf-dcac-4cbb-a59c-00661efd6425", "user_disliked": "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5"}

Submitting feedback to LangSmith via pre-signed URL

After receiving a feedback URL from the Agent Server response, submit feedback by making a POST or GET request to that URL. The POST request body can include score, value, comment, correction, and metadata fields. The GET request supports score, value, comment, and correction as query parameters but does not support metadata. After submission, LangSmith records the feedback on the trace using the corresponding feedback key.

Python SDK example: streaming runs with feedback_keys

from langgraph_sdk import get_client client = get_client(url="<DEPLOYMENT_URL>", api_key="<API_KEY>") thread = await client.threads.create() thread_id = thread["thread_id"] feedback_urls = {} async for event in client.runs.stream( thread_id, "agent", input={ "messages": [ {"role": "user", "content": "Tell me a joke about databases."} ] }, stream_mode="updates", feedback_keys=["user_liked", "user_disliked"], ): if event.event == "feedback": feedback_urls = event.data print("Feedback URLs:", feedback_urls) elif event.event == "updates": print(event.data) This example demonstrates creating a run with feedback_keys and extracting the feedback URLs from the streamed response.

JavaScript SDK example: streaming runs with feedbackKeys

import { Client } from "@langchain/langgraph-sdk"; const client = new Client({ apiUrl: "<DEPLOYMENT_URL>", apiKey: "<API_KEY>" }); const thread = await client.threads.create(); const threadId = thread.thread_id; let feedbackUrls = {}; const streamResponse = client.runs.stream(threadId, "agent", { input: { messages: [{ role: "user", content: "Tell me a joke about databases." }], }, streamMode: "updates", feedbackKeys: ["user_liked", "user_disliked"], }); for await (const event of streamResponse) { if (event.event === "feedback") { feedbackUrls = event.data; console.log("Feedback URLs:", feedbackUrls); } else if (event.event === "updates") { console.log(event.data); } } This example demonstrates creating a run with feedbackKeys in the JavaScript SDK and extracting the feedback URLs from the streamed response.

cURL example: streaming Agent Server run with feedback_keys

curl --request POST \ --url "<DEPLOYMENT_URL>/threads/<THREAD_ID>/runs/stream" \ --header "Content-Type: application/json" \ --header "x-api-key: <API_KEY>" \ --data '{ "assistant_id": "agent", "input": { "messages": [ { "role": "user", "content": "Tell me a joke about databases." } ] }, "stream_mode": "updates", "feedback_keys": ["user_liked", "user_disliked"] }' This example shows how to create a streaming run with feedback_keys using the Agent Server API directly.

feedback_keys parameter in Agent Server runs

When creating a run with the Agent Server API, include the feedback_keys field in the request body as an array of strings, such as ["user_liked", "user_disliked"]. The response will contain a feedback object with pre-signed URLs for each key.

POST request body for submitting feedback

When submitting feedback via POST to a pre-signed feedback URL, the request body can include the following fields: score (numeric), value (numeric), comment (string), correction (object), and metadata (object). All fields are optional.

GET request parameters for submitting feedback

When submitting feedback via GET to a pre-signed feedback URL, the following query parameters are supported: score, value, comment, and correction. The metadata field is not supported with GET requests.

Prometheus metrics export breaking changes in v0.11.0rc8

Agent Server metrics are now emitted through OpenTelemetry/Prometheus client on the dedicated Prometheus scrape endpoint (LSD_PROM_METRICS_PORT, default 9464). Potentially breaking changes: lg_api_http_requests_latency_seconds is now lg_api_http_requests_latency reporting milliseconds; pool request counters now use _total suffix; lg_api_pending_runs_wait_time_* gauges removed in favor of lg_api_run_queue_wait_time_1st_attempt histogram.

langsmith_session_name field tracks LangSmith tracing project

The langsmith_session_name field is added to each run and represents the LangSmith tracing project name when tracing is enabled. Support is exposed via /info endpoint so Studio can detect API versions that support this field.

A2A messageId mapped to LangChain message IDs

A2A messageId is now mapped to LangChain message IDs for proper message tracking across protocols.

OTLP latency histogram bucket configuration fixed

Fixed OTLP latency histogram bucket configuration so latency metrics use legacy second-scale buckets converted to milliseconds, restoring accurate p95/p99 for long HTTP polls, queue waits, and run execution.

Enable Prometheus metrics with environment variable

To expose OpenTelemetry metrics on a dedicated Prometheus scrape endpoint at port LSD_PROM_METRICS_PORT (default 9464), set LSD_PROM_METRICS_ENABLED=true. This exposes metrics for run lifecycle, latency, stream, and worker gauges. Datadog OTLP push continues to work alongside Prometheus when both are configured.

Required fields for subagent run type

For subagent run types, the following fields are required: ls_subagent_id (tier: always, stable identifier for the subagent) and ls_subagent_type (tier: always, type or role of the subagent like 'researcher').

Coding agent metadata contract schema

The coding agent metadata contract is the authoritative schema that standardizes what trace metadata coding agents must emit when sending runs to LangSmith. It defines which fields are required on every run, which fields are expected when the runtime can supply them, and which fields apply only to specific run types. Coding agent integrations use this schema to ensure their traces are consistently structured, queryable, and compatible with LangSmith's observability and filtering features.

Supported coding agent integrations for LangSmith

The following integrations implement the coding agent metadata contract: Claude Code (ls_integration: claude-code), OpenAI Codex (openai-codex), Deep Agents (deepagents-code), Cursor (cursor), Pi (pi), Opencode (opencode), and GitHub Copilot (copilot).

Global identity block fields for coding agent runs

Every run type must include the following identity fields in its metadata: ls_agent_type (one of 'root', 'subagent', 'middleware', or 'compaction'), ls_agent_purpose (high-level purpose like 'coding'), ls_integration (identifier of the integration emitting the run), ls_agent_runtime (human-readable runtime name like 'Claude Code 1.0.28'), thread_id (stable identifier for the conversation thread), and ls_trace_schema_version (currently 'coding-agent-v1').

Availability tiers for metadata fields

Fields in the coding agent metadata schema are marked with one of three availability tiers: always (must be present on every run), where_known (required whenever the runtime can expose the value, omit only when the runtime genuinely cannot provide the information), and contextual (optional metadata, omit when not applicable).

Coding agent run types

The coding agent metadata schema distinguishes five run types: root (the top-level run representing a full agent turn or session), llm (a language model call within a turn), tool (a tool invocation within a turn), subagent (a nested or delegated agent run), and interrupted (a run that was interrupted before completion).

Required fields on all coding agent run types

In addition to the global identity block fields, the following fields are required on all run types: ls_agent_version (tier: where_known, version string for the agent runtime like '1.0.28'), git_branch (tier: where_known, active Git branch in the repository being edited), git_commit_sha (tier: where_known, full SHA of the current Git commit), git_repo_url (tier: where_known, remote URL of the repository), and working_directory (tier: where_known, absolute path of the working directory).

Required fields for llm run type

For llm run types, the following fields are required: ls_model_name (tier: where_known, model identifier like 'claude-opus-4-5') and ls_provider (tier: where_known, model provider like 'anthropic').

Required fields for tool run type

For tool run types, the following field is required: ls_tool_name (tier: always, name of the tool invoked like 'bash' or 'computer').

Interrupted run type fields

Interrupted runs carry the same fields as root runs. The run type itself signals the abnormal termination state; no additional required fields are added.

LangSmith Engine for production agents

LangSmith Engine is used to detect recurring failures in production agent traces, diagnose root causes, and resolve them.

draw_mermaid_png parameters

The draw_mermaid_png() method in Python accepts the following parameters: curve_style (CurveStyle.LINEAR), node_colors (NodeStyles with first, last, and default color hex codes), wrap_label_n_words (integer for label wrapping), output_file_path (file path or None), draw_method (MermaidDrawMethod.PYPPETEER or other), background_color (color string), and padding (integer pixels).

Render graph as PNG using graphviz

To render a graph as PNG using graphviz, install graphviz via pip and call app.get_graph().draw_png(). If graphviz is not installed, an ImportError will be raised with instructions to install pygraphviz dependencies.

Get graph as Mermaid syntax

You can convert a compiled graph into Mermaid syntax by calling app.get_graph().draw_mermaid() in Python or await app.getGraphAsync().drawMermaid() in TypeScript. This returns Mermaid flowchart code that can be rendered as a visualization.

Give your agent this brain