Agent Server Redis message size ceiling
In v0.2.44, a ceiling for Redis message size was introduced with an option to skip messages larger than 128 MB for improved performance.
LangChain & LangGraph · all subjects
deployment & infrastructure
73 notes in this subject, read out of this brain and free to use. This is page 1 of 2.
Agent Server graceful shutdown on SIGINT
In v0.2.57, middleware was introduced to gracefully shut down the server after completing in-flight requests upon receiving a SIGINT signal.
Agent Server SIGTERM signal handler
In v0.2.56, a handler for SIGTERM signals was added to improve application stability.
Agent Server core capabilities overview
The Agent Server provides HTTP and gRPC APIs for managing assistants, threads, runs, and crons. It supports streaming, human-in-the-loop interrupts, persistence with checkpointing, custom encryption, webhooks, and both Python and Node.js runtime execution. Key features include thread management with TTL, run scheduling via crons, state persistence with PostgreSQL/Redis backends, A2A (agent-to-agent) protocol support, and Store API for key-value operations.
Distributed runtime graceful shutdown
The distributed runtime supports graceful shutdown handoff allowing in-flight runs to transfer to the next pod without consuming a retry attempt. This improves resilience during pod restarts and scaling operations.
TypeScript and JavaScript runtime support
The Agent Server supports Node.js 24 (latest LTS) with mixed schema support for StateGraph and type bag patterns for GraphNode and ConditionalEdgeRouter utilities. JavaScript runtime includes support for SDK structlog, proper async/await handling, and gRPC client support for streaming operations.
Agent Server interrupt ID exposure in thread state retrieval
As of v0.2.97, the interrupt ID is exposed when retrieving the thread state to improve API transparency.
Agent Server context added to langgraph nodes
In v0.2.98, context was added to langgraph nodes to improve log filtering and trace visibility.
Agent Server TTL feature for resumable streams
In v0.2.52, a time-to-live (TTL) feature was implemented for resumable streams. The default time-to-live for resumable streams was later reduced to 2 minutes in v0.2.83.
Agent Server webhook timeout retries
In v0.2.78, timeout retries were added to webhook calls to improve reliability.
Agent Server webhook global disable configuration
In v0.2.79, a configuration flag was introduced to disable webhooks globally across all routes.
Agent Server on_disconnect field support for runs/wait
In v0.2.85, support was added for the on_disconnect field to the runs/wait endpoint, with disconnect logs included for better debugging.
Distributed tracing security warning
Distributed-tracing headers (langsmith-trace, baggage) are consumed as trusted tracing context. Only configure your server to apply inbound trace context for deployments called by trusted, internal services. If your Agent Server receives requests directly from untrusted third parties or the public internet, do not propagate these headers into the tracing context: strip them at your gateway or proxy instead. Trusting baggage from an external caller lets them influence how your runs are recorded.
langgraph.json configuration for distributed tracing
Export the graph function that handles distributed tracing context in langgraph.json with the following structure: {"graphs": {"agent": "./src/agent.py:graph"}} where the path points to the module and the context manager function that accepts the config parameter.
Agent Server feedback POST submission example
Example showing how to submit feedback to a pre-signed URL using POST: ```bash curl --request POST \ --url "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5" \ --header "Content-Type: application/json" \ --data '{ "score": 1, "value": 0, "comment": "I didn't like this joke because it didn't make me laugh.", "correction": {}, "metadata": {} }' ``` This example submits feedback via POST with score, value, comment, correction, and metadata fields.
Agent Server feedback_keys request parameter
When creating a run with the Agent Server streaming API, include the `feedback_keys` field in the request body. For example, set `feedback_keys` to ["user_liked", "user_disliked"] to enable feedback collection for those specific feedback types.
Agent Server feedback response structure
The response from creating a run with feedback_keys contains a `feedback` object with pre-signed URLs for each key. Each key maps to a unique pre-signed URL that clients can use to submit feedback. Example: {"user_liked": "https://api.smith.langchain.com/api/v1/feedback/tokens/ef19fedf-dcac-4cbb-a59c-00661efd6425", "user_disliked": "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5"}
Agent Server feedback submission endpoint
Submit feedback by POST or GET request to the pre-signed URL returned in the feedback event. POST requests accept a JSON body with fields: score (numeric), value (numeric), comment (string), correction (object), and metadata (object). GET requests support score, value, comment, and correction as query parameters, but not metadata.
Python SDK streaming run with feedback_keys example
Example showing how to create a run and handle feedback URLs in Python SDK: ```python from langgraph_sdk import get_client client = get_client(url="<DEPLOYMENT_URL>", api_key="<API_KEY>") thread = await client.threads.create() thread_id = thread["thread_id"] feedback_urls = {} async for event in client.runs.stream( thread_id, "agent", input={ "messages": [ {"role": "user", "content": "Tell me a joke about databases."} ] }, stream_mode="updates", feedback_keys=["user_liked", "user_disliked"], ): if event.event == "feedback": # Example: {"user_liked": ".../feedback/tokens/<id>", "user_disliked": "..."} feedback_urls = event.data print("Feedback URLs:", feedback_urls) elif event.event == "updates": print(event.data) ``` This example shows how to stream a run with feedback collection enabled and capture the returned feedback URLs.
JavaScript SDK streaming run with feedback_keys example
Example showing how to create a run and handle feedback URLs in JavaScript SDK: ```javascript import { Client } from "@langchain/langgraph-sdk"; const client = new Client({ apiUrl: "<DEPLOYMENT_URL>", apiKey: "<API_KEY>" }); const thread = await client.threads.create(); const threadId = thread.thread_id; let feedbackUrls = {}; const streamResponse = client.runs.stream(threadId, "agent", { input: { messages: [{ role: "user", content: "Tell me a joke about databases." }], }, streamMode: "updates", feedbackKeys: ["user_liked", "user_disliked"], }); for await (const event of streamResponse) { if (event.event === "feedback") { // Example: { user_liked: ".../feedback/tokens/<id>", user_disliked: "..." } feedbackUrls = event.data; console.log("Feedback URLs:", feedbackUrls); } else if (event.event === "updates") { console.log(event.data); } } ``` This example shows how to stream a run with feedback collection enabled and capture the returned feedback URLs in JavaScript.
cURL streaming run with feedback_keys example
Example showing how to create a streaming run with feedback collection using cURL: ```bash curl --request POST \ --url "<DEPLOYMENT_URL>/threads/<THREAD_ID>/runs/stream" \ --header "Content-Type: application/json" \ --header "x-api-key: <API_KEY>" \ --data '{ "assistant_id": "agent", "input": { "messages": [ { "role": "user", "content": "Tell me a joke about databases." } ] }, "stream_mode": "updates", "feedback_keys": ["user_liked", "user_disliked"] }' ``` This example shows how to call the streaming run endpoint with feedback_keys parameter.
Agent Server feedback GET submission example
Example showing how to submit feedback to a pre-signed URL using GET: ```bash curl --request GET \ --url "https://api.smith.langchain.com/api/v1/feedback/tokens/e952734e-c0a0-417b-a04d-fc2209691ed5?score=1&value=0&comment=I%20didn%27t%20like%20this%20joke%20because%20it%20didn%27t%20make%20me%20laugh.&correction=%7B%7D" ``` GET requests support score, value, comment, and correction as query parameters, but metadata is not supported with GET.
Agent Server feedback data model optimization
Feedback keys can be modeled flexibly based on use case. Multiple feedback types can be grouped under a single key with different score values. For example, use `key="user_score"` with `score=1` for user_liked and `score=-1` for user_disliked. This groups all user preference signals under one feedback key for simpler analysis. The feedback data model should be designed for the specific application needs.
Agent Server feedback workflow for production UI
A productionized feedback implementation follows these steps: (1) Create the run from backend or frontend, (2) Capture the feedback object and store the returned URLs, (3) Render feedback controls such as thumbs up/down buttons and feedback forms, (4) On feedback submission, POST or GET a feedback URL based on user feedback intent, (5) Optionally disable feedback controls after submission and show confirmation to the user.
Agent Server primitives
The Agent Server runtime works with three core primitives: assistants for configuration, threads for state, and runs for workloads.
Agent Server core capabilities
Agent Server provides capabilities including: developing applications with structured app configuration and dependencies for Python, JavaScript, and monorepos; runtime support for assistants, threads, runs, and cron jobs; streaming to users, pausing for human review, and handling concurrent input; connecting via MCP and A2A; authentication and resource-level access control; and server customization including caching, custom stores and checkpointers, lifespan hooks, middleware, custom routes, encryption, and configurable headers and logs.
Agent Server deployment frameworks
Agent Server can deploy other agent frameworks such as Strands and CrewAI by wrapping existing agents with the Functional API.
Request concurrency vs run concurrency in Agent Server
Two independent kinds of concurrency determine how the Agent Server scales. Request concurrency is how many API requests (creating runs, reading thread state, streaming results) the deployment serves at once. API servers handle requests asynchronously, and request concurrency scales horizontally with the number of API server replicas. Run concurrency is how many runs execute at once. A single queue worker executes up to N_JOBS_PER_WORKER runs concurrently (default 10). Run concurrency is capped at the number of queue workers multiplied by N_JOBS_PER_WORKER. Creating a run is a fast write request: the API server persists a pending run and returns immediately, without waiting for the run to execute.
N_JOBS_PER_WORKER tuning based on assistant characteristics
N_JOBS_PER_WORKER controls the maximum number of runs a single queue worker can execute at a time and defaults to 10. For CPU-bounded assistants, the default value of 10 is likely sufficient; lower N_JOBS_PER_WORKER if excessive CPU usage on queue workers or delays in run execution are observed. For memory-bounded assistants or when queue workers approach memory limits, lower N_JOBS_PER_WORKER to reduce concurrent runs per worker. For IO-bounded assistants, increase N_JOBS_PER_WORKER to handle more concurrent runs per worker. There is no upper limit to N_JOBS_PER_WORKER, but setting it too high in environments with bursty traffic can lead to uneven worker utilization, increased run execution times, and high memory usage on queue workers.
Avoid synchronous blocking operations in Agent Server
Avoid synchronous blocking operations in code and prefer asynchronous operations. Long synchronous operations can block the main event loop, causing longer request and run execution times and potential timeouts. Instead of using synchronous code like time.sleep(1), use asynchronous code like await asyncio.sleep(1). If an assistant requires synchronous blocking operations, run those in asyncio.to_thread() or equivalent.
Enable queue workers to offload API server
By default, the API server manages the queue and does not use queue workers. Enable queue workers by setting queue.enabled to true. This offloads queue management from the API server to dedicated queue workers, reducing load on the API server and allowing it to focus on handling requests.
Calculate required queue workers for throughput
To calculate required capacity: available_jobs = number_of_queue_workers * N_JOBS_PER_WORKER. Throughput per second = available_jobs / average_run_execution_time_seconds. Therefore, minimum number of queue workers = throughput_per_second * average_run_execution_time_seconds / N_JOBS_PER_WORKER. This formula sizes run-execution capacity (queue workers), separate from request-serving capacity (API server replicas).
Use filtering and pagination to reduce read load
Agent Server provides a search API for each resource type. These APIs implement pagination by default and offer many filtering options. Use filtering to reduce the number of resources returned per request and improve performance.
Use /join endpoint instead of polling runs
Avoid polling the state of a run by using the /join API endpoint. This method returns the final state of the run once the run is complete. If real-time monitoring of run output is needed, use the /stream API endpoint instead, which streams the run output including the final state of the run.
Agent Server write load components
Write load is primarily driven by: creation of new runs, creation of new checkpoints during run execution, writing to long term memory, creation of new threads, creation of new assistants, and deletion of runs, checkpoints, threads, assistants and cron jobs. Components primarily responsible for handling write load are: API server (handles initial request and persistence to database), Queue worker (handles execution of runs), Redis (handles storage of ephemeral data about on-going runs), and Postgres (handles storage of all data including run, thread, assistant, cron job, checkpointing and long term memory).
Agent Server read load components
Read load is primarily driven by: getting results of a run, getting state of a thread, searching for runs, threads, cron jobs and assistants, and retrieving checkpoints and long term memory. Components primarily responsible for handling read load are: API server (handles request and direct retrieval from database), Postgres (handles storage of all data including run, thread, assistant, cron job, checkpointing and long term memory), and Redis (handles storage of ephemeral data about on-going runs, including streaming messages from queue workers to api servers).
Example Agent Server configuration - high reads, high writes
For 500 read requests per second and 500 write requests per second: API servers: 15 replicas (1 CPU, 2Gi requests, 2 CPU and 4Gi limits per server); Queue workers: 10 replicas (1 CPU, 2Gi requests, 2 CPU and 4Gi limits per worker); N_JOBS_PER_WORKER: 50; Redis resources: 2 Gi (requests and limits); Postgres: 8 CPU and 32 Gi memory requests, 16 CPU and 64 Gi memory limits.
Example Agent Server autoscaling configuration
For deployments experiencing bursty traffic, enable autoscaling to scale the number of API servers and queue workers. Example configuration for high reads and high writes: API servers with autoscaling enabled, minReplicas 15, maxReplicas 25; Queue workers with autoscaling enabled, minReplicas 10, maxReplicas 20.
Configure autoscaling for bursty workloads
Autoscaling is disabled by default but should be configured for bursty workloads. Using throughput calculations (throughput_per_second * average_run_execution_time_seconds / N_JOBS_PER_WORKER), determine the maximum number of queue workers to allow the autoscaler to scale to based on maximum expected throughput. Similarly, determine the maximum number of API servers for bursty read workloads.
Agent Server overview and capabilities
LangSmith Deployment's Agent Server offers an API for creating and managing agent-based applications. It is built on the concept of assistants, which are agents configured for specific tasks, and includes built-in persistence and a task queue. The Agent Server can be used to create and manage assistants, threads, runs, and cron jobs.
Graph deployment as compiled graph vs factory function
There are two ways to register a graph for deployment: (1) Export an already-compiled CompiledGraph instance, which the server loads once at container startup and reuses for every run with no compilation overhead per request (recommended). (2) Export an agent factory function that the server invokes each time it needs the graph, use this only when you need per-run graph customization such as choosing different models or tools based on the assistant config. Compiled graphs are recommended unless you specifically need per-run customization, since factory functions add overhead on every invocation.
Task queue role in Agent Server
When a client creates a run, the API server enqueues it and a queue worker picks it up for execution. Workers can be signaled to cancel a run in progress and publish output events that open /stream connections forward to the client in real time. Redis handles the signaling, cancellation, and streaming pub/sub between API servers and queue workers. Redis stores only ephemeral data—no user or run data persists in Redis. Run data itself is always read from and written to PostgreSQL.
Three Agent Server runtime deployment modes
Agent Server supports three runtime configurations: (1) Single host - the API server manages the task queue directly with no separate queue workers, this is the default for self-hosted deployments and is suitable for development and low-traffic use cases. (2) Split API and queue - dedicated queue workers handle run execution on separate hosts from the API server, enable this for self-hosted deployments by setting 'queue.enabled: true' in configuration, each tier scales independently with API servers scaling on request volume and queue workers scaling on pending run count. (3) Distributed runtime - the API and queue processes run separately, with one process for orchestration and one process for execution, use this for large-scale deployments with high concurrency requirements.
Container architecture in Agent Server deployments
A typical Agent Server deployment consists of two kinds of long-running containers, both built from the same Docker image: (1) API servers handle client requests (creating runs, reading thread state, streaming results) but do not execute agent code themselves. (2) Queue workers are the execution engine, they listen to the durable task queue, execute your graph code, and write checkpoints. Containers are stateless but persistent. At least 1 queue worker must listen to the task queue at any time to ensure no runs are orphaned. The containers can serve many runs over their lifetime. API servers and queue workers are separate container pools and scale independently.
Run execution lifecycle in Agent Server
When you invoke a run in Agent Server, the request flows through these steps: (1) A client sends a request to an API server, which creates a pending run in the durable task queue. (2) A queue worker picks up the run, acquires a lease on it, loads the appropriate graph, and begins execution. The queue enforces that at most 1 run can be executed for a given thread at one time. (3) As the graph executes, the worker writes checkpoints to the persistence layer (frequency depends on the durability mode) and broadcasts streaming events over the configured pubsub provider. (4) If the client opened a /stream connection, the API server subscribes to the pubsub channel and forwards events to the client via server-sent events in real time. (5) When execution completes, the worker updates the run status and releases its slot for the next run.
Concurrent run execution in Agent Server workers
Each worker executes up to N_JOBS_PER_WORKER runs concurrently (default: 10), so a single worker container serves many runs in parallel. This bounds concurrent run execution, not the number of API requests the deployment can serve. API servers handle requests independently and scale separately, so request-serving capacity is not capped by N_JOBS_PER_WORKER.
Graphs do not have to be written with LangGraph
When deploying with Agent Server, graphs do not have to be written with LangGraph. You can deploy agents built with other frameworks such as Strands, Claude Agent SDK, Google ADK, and more using the LangGraph Functional API or the deployments-wrap-sdk package.
Database management for Agent Server persistence
LangSmith cloud manages the database for you. If you are deploying on your own infrastructure, you will need to set up the database yourself.
Supported integrations and ls_integration values
The following integrations implement the coding agent metadata contract: Claude Code uses 'claude-code', OpenAI Codex uses 'openai-codex', Deep Agents uses 'deepagents-code', Cursor uses 'cursor', Pi uses 'pi', Opencode uses 'opencode', and GitHub Copilot uses 'copilot'.
langgraph deploy command deploys apps that export a graph from langgraph.json
The `langgraph deploy` command deploys any application that exports a graph from a `langgraph.json` config file. This works the same way regardless of which framework was used to author the agent, including LangGraph, Claude Agent SDK, Strands, CrewAI, AutoGen, or Google ADK.
Prerequisites for langgraph deploy command
To use the `langgraph deploy` command, you need: a LangSmith account on the Plus plan or above with an API key; optionally Docker installed with Docker daemon running for local builds (not required if using remote builds); optionally on Apple Silicon (M1/M2/M3), Docker Buildx for cross-compiling to linux/amd64; and the LangGraph CLI installed via `uv tool install langgraph-cli`.
LangGraph CLI installation command
Install the LangGraph CLI with the command: `uv tool install langgraph-cli`
Create deployable app with langgraph new template
Create a new deployable app using: `langgraph new path/to/your/app --template new-langgraph-project-python`. Running `langgraph new` without the `--template` flag shows an interactive menu of available templates.
Set LangSmith API key for deployment
Add your LangSmith API key to a `.env` file in your project root as `LANGSMITH_API_KEY=lsv2_...`. The `langgraph deploy` command reads this automatically. Alternatively, pass it inline: `LANGSMITH_API_KEY=lsv2_... langgraph deploy`.
langgraph deploy creates Serverless deployment by default
Running `langgraph deploy` creates a Serverless deployment named after your project directory by default. Use the `--name` flag to override the name or `--deployment-type dedicated` to change the deployment type.
Update existing deployment by re-running langgraph deploy
To update an existing deployment after making code changes, re-run `langgraph deploy`. It finds the existing deployment by name and updates it in place.
langgraph deploy subcommands for managing deployments
Use `langgraph deploy list` to see all deployments, `langgraph deploy logs` to tail runtime logs, and `langgraph deploy delete <ID>` to remove a deployment.
Deploy from Studio using langgraph dev
To deploy from Studio, run `langgraph dev` to start the local development server, which automatically opens Studio. Then click the `deploy` button in Studio.
Studio capabilities for testing deployed agents
Studio is an interactive agent IDE connected directly to your deployment. Use it to send messages, inspect intermediate state at each node, edit state mid-run, and replay from any prior checkpoint without writing code.
Access deployed application through Studio
After deployment is ready, go to LangSmith at https://smith.langchain.com, select Deployments in the left sidebar, select your deployment, and click Studio in the top right corner to open the interactive IDE.