OR and AND in PostgreSQL auth filters
Support for $or and $and in PostgreSQL auth filters was added to enable complex logic in authentication checks.
LangChain & LangGraph · all subjects
configuration and testing
47 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
Exact match for noauth path in auth middleware
Security was enhanced by requiring an exact match for the noauth path in authentication middleware.
middleware_order setting for auth priority
A new middleware_order setting was introduced to apply authentication middleware before custom middleware, allowing finer control over protected route configurations.
OpenAI provider configuration
The openai provider uses the OPENAI_API_KEY credential. An optional OPENAI_BASE_URL can be set for OpenAI-compatible gateways that expose the Responses API.
OpenAI ChatGPT provider configuration
The openai-chatgpt provider uses ChatGPT OAuth tokens for sign-in. Usage draws on Plus/Pro/Team Codex allowance.
GitHub Copilot provider configuration
The copilot provider requires either a GitHub CLI session or the COPILOT_API_KEY credential. An optional COPILOT_BASE_URL can be set. For CI environments, use an OAuth token, not a classic Personal Access Token.
OpenRouter provider configuration
The openrouter provider uses the OPENROUTER_API_KEY credential. An optional OPENWIKI_OPENROUTER_PROVIDER_ONLY allowlist can restrict routing when OpenRouter serves a model through multiple upstream providers.
Anthropic provider configuration
The anthropic provider uses the ANTHROPIC_API_KEY credential. An optional ANTHROPIC_BASE_URL can be set.
Google Gemini provider configuration
The gemini provider uses the GEMINI_API_KEY credential from Google AI Studio.
AWS Bedrock provider configuration
The bedrock provider requires AWS credentials and region. Use BEDROCK_AWS_ACCESS_KEY_ID, BEDROCK_AWS_SECRET_ACCESS_KEY, and BEDROCK_AWS_REGION, or rely on the AWS SDK default credential provider chain. Paste the Bedrock model ID directly; some newer models require a cross-region inference profile ID (for example us.anthropic.claude-sonnet-5).
Baseten provider configuration
The baseten provider uses the BASETEN_API_KEY credential. An optional BASETEN_BASE_URL can be set.
Fireworks provider configuration
The fireworks provider uses the FIREWORKS_API_KEY credential. An optional FIREWORKS_BASE_URL can be set.
Nebius provider configuration
The nebius provider uses the NEBIUS_API_KEY credential from Nebius Token Factory.
NVIDIA provider configuration
The nvidia provider uses the NVIDIA_API_KEY credential. An optional NVIDIA_BASE_URL can be set.
OpenAI-compatible provider configuration
The openai-compatible provider is used for gateways or local servers that expose OpenAI-compatible chat completions. It requires OPENAI_COMPATIBLE_API_KEY, OPENAI_COMPATIBLE_BASE_URL, and a custom model ID in OPENWIKI_MODEL_ID. Examples include Ollama (http://localhost:11434/v1) and LM Studio (http://localhost:1234/v1).
OpenWiki credentials storage location
Credentials and defaults are stored in ~/.openwiki/.env. Process environment values take priority over file values.
Setting active provider and model in OpenWiki
Set the active provider and model with OPENWIKI_PROVIDER and OPENWIKI_MODEL_ID environment variables, for example: OPENWIKI_PROVIDER=openai and OPENWIKI_MODEL_ID=gpt-5.6-terra.
OpenWiki provider retry attempts
Override retries after the first provider request with OPENWIKI_PROVIDER_RETRY_ATTEMPTS, which must be a positive integer. If unset, OpenWiki defaults to 3 retries.
OpenWiki Bedrock max tokens override
Override Bedrock's default 16000-token output ceiling with OPENWIKI_BEDROCK_MAX_TOKENS when a model supports a lower ceiling.
OpenWiki reasoning effort in interactive chat
In interactive chat, use /effort to choose an available value or /effort default to restore the provider default.
OpenRouter max tokens handling
By default OpenRouter sends no max_tokens, so credit pre-checks budget for the model's full advertised output ceiling and low balances can fail with 402 errors. OPENWIKI_OPENROUTER_MAX_TOKENS takes precedence over OPENWIKI_MAX_OUTPUT_TOKENS on OpenRouter runs. Using a limit means truncation is possible on long generations instead of 402 failures.
OpenWiki reasoning effort setting
OPENWIKI_REASONING_EFFORT is an optional global setting for models that advertise reasoning support. Leave it unset to preserve the provider default. Invalid provider, model, or effort combinations fail before a request is sent.
OpenAI reasoning effort support
The openai and openai-chatgpt providers support reasoning effort for models gpt-5.6-terra, gpt-5.6-luna, and gpt-5.6-sol. Supported values are none, low, medium, high, xhigh, and max. These map to Responses API reasoning.effort.
NVIDIA reasoning effort support
The nvidia provider supports reasoning effort for model nvidia/nemotron-3-super-120b-a12b. Supported values are none, low, and high. These map to Chat Completions reasoning_effort.
GitHub Copilot setup process
To use GitHub Copilot, select it during openwiki --init. If an active GitHub CLI session exists, OpenWiki can reuse it; otherwise run gh auth login from the credential prompt. Then choose a model such as gpt-5.5.
GitHub Copilot token storage
OpenWiki leaves the GitHub CLI token in the GitHub CLI credential store and does not copy that token into ~/.openwiki/.env. For CI or headless environments without a GitHub CLI session, set COPILOT_API_KEY to a GitHub OAuth token, not a Personal Access Token (classic or fine-grained).
GitHub Copilot provider example configuration
To use GitHub Copilot, set OPENWIKI_PROVIDER=copilot and OPENWIKI_MODEL_ID=gpt-5.5.
OpenAI ChatGPT provider authentication
The openai-chatgpt provider calls OpenAI's Codex backend using the ChatGPT subscription instead of a metered API key. The wizard opens the OpenAI auth page in the browser and prints the URL for headless use. OpenWiki stores managed OAuth tokens in ~/.openwiki/.env and refreshes the access token automatically. Treat the refresh token like a password.
OpenAI ChatGPT provider setup command
To set up OpenAI ChatGPT provider, run: OPENWIKI_PROVIDER=openai-chatgpt openwiki code --init
Gemini Enterprise provider setup
The gemini-enterprise provider uses Google Application Default Credentials. No API key is required. Set OPENWIKI_PROVIDER=gemini-enterprise, GOOGLE_CLOUD_PROJECT to the GCP project, and GOOGLE_CLOUD_LOCATION (defaults to global). Partner and open-weight models are region-specific, so set GOOGLE_CLOUD_LOCATION explicitly when using them.
AWS Bedrock stream idle timeout configuration
For Bedrock stream idle timeout, set OPENWIKI_STREAM_IDLE_TIMEOUT in milliseconds (an integer from 0 to 2147483647). Set 0 to disable the watchdog. If unset, OpenWiki preserves the @langchain/aws provider default.
AWS Bedrock provider example configuration
To use AWS Bedrock, set: OPENWIKI_PROVIDER=bedrock, BEDROCK_AWS_ACCESS_KEY_ID, BEDROCK_AWS_SECRET_ACCESS_KEY, BEDROCK_AWS_REGION=us-east-1, and OPENWIKI_MODEL_ID=anthropic.claude-sonnet-5
OpenAI-compatible provider streaming configuration
Some gateways accept only streaming requests where the model returns output in chunks over an open connection. When OpenWiki hits one of those gateways with a non-streaming request, the gateway may reject the call or return HTTP 200 with empty content. Enable streaming with OPENWIKI_OPENAI_COMPATIBLE_STREAMING=true when the gateway requires it. Streaming stays off by default because this provider can point at arbitrary third-party endpoints where streaming is not guaranteed to work through proxies and load balancers. Enabling it also makes the client report estimated rather than server-reported token counts.
OpenAI-compatible provider Responses API option
To opt the openai-compatible provider into the Responses API instead of chat completions, set: OPENWIKI_OPENAI_COMPATIBLE_USE_RESPONSES_API=true
OpenAI-compatible provider example configuration
To use the openai-compatible provider with a gateway, set: OPENWIKI_PROVIDER=openai-compatible, OPENAI_COMPATIBLE_API_KEY=your-gateway-key, OPENAI_COMPATIBLE_BASE_URL=https://your-gateway.example.com/v1, and OPENWIKI_MODEL_ID=your-gateway-model-name
OpenAI-compatible provider local examples
Local examples such as Ollama (http://localhost:11434/v1) and LM Studio (http://localhost:1234/v1) use the openai-compatible provider. OpenWiki still requires OPENAI_COMPATIBLE_API_KEY even when the local server ignores the key value.
OpenRouter provider pinning example
When OpenRouter serves a model through multiple upstream providers, restrict routing with: OPENWIKI_PROVIDER=openrouter, OPENROUTER_API_KEY=your-key, OPENWIKI_OPENROUTER_PROVIDER_ONLY=Novita
Three main types of agent evaluations
Agent evaluation can focus on three distinct approaches: (1) Final response evaluation - evaluates the agent's final output given a prompt and optional tools; (2) Trajectory evaluation - evaluates whether the agent took the expected sequence of steps or tool calls to arrive at the final answer; (3) Single step evaluation - evaluates any specific agent step in isolation, such as whether it selects the appropriate first tool for a given step.
Final response evaluator with LLM-as-judge
A final response evaluator compares an agent's output to reference outputs using an LLM-as-judge. The evaluator receives inputs (the prompt), outputs (the agent's response), and reference_outputs (ground truth response). The judge LLM grades based on: (1) factual accuracy relative to ground truth; (2) absence of conflicting statements; (3) tolerance for additional correct information beyond the ground truth. The evaluator returns a boolean indicating correctness.
Trajectory evaluator implementation
A trajectory evaluator checks if an agent's actual sequence of steps matches an expected reference trajectory. It uses subsequence matching: if the expected trajectory has more steps than the actual trajectory, it returns False. Otherwise, it iterates through both trajectories and counts matching steps in order, returning the percentage of expected steps that were completed (i/len(expected_trajectory)). This gives partial credit when an agent takes some correct steps even if it fails to reach the right answer.
Mock mode testing with RunnableConfig
To avoid side effects during testing, graph nodes can inspect RunnableConfig to check for a configurable variable. Use config.get('configurable', {}).get('env', 'prod') to read configuration. When testing, pass config={'env': 'test'} to graph.ainvoke() or graph.astream(). Nodes can then execute mock logic (e.g., simulating database operations without actually modifying data) when this flag is set.
Accessing individual graph nodes for isolated testing
Individual nodes within a compiled graph can be accessed and invoked directly via graph.nodes[node_name]. This allows single-step evaluation of specific components without running the entire graph. For example, await graph.nodes['intent_classifier'].ainvoke(inputs) runs only the intent classifier node and returns its output.
LangSmith evaluation workflow with client.aevaluate
LangSmith provides client.aevaluate() for running evaluations against a dataset. It takes: a target function (async callable returning outputs dict), a dataset_name (string), evaluators (list of evaluator functions), experiment_prefix (string for tracking runs), num_repetitions (int), and max_concurrency (int). The function returns experiment_results with a .to_pandas() method to view results as a DataFrame. Evaluators receive inputs, outputs, and reference_outputs dicts and return a boolean or float score.
LangSmith dataset creation and management
Create a dataset with client.create_dataset(dataset_name=string), which returns a dataset object with an id. Add examples using client.create_examples(dataset_id=dataset.id, examples=list). Each example is a dict with 'inputs' and 'outputs' keys containing the test data. Use client.has_dataset(dataset_name=string) to check if a dataset already exists before creating it to avoid duplicates.
LLM-as-judge evaluator implementation
An LLM-as-judge evaluator is an async function that takes inputs, outputs, and reference_outputs dicts, constructs a prompt comparing ground truth and actual responses, invokes a judge LLM with structured output, and returns a boolean score. The judge LLM uses detailed grading criteria in system instructions and returns a Grade TypedDict with 'reasoning' (explanation) and 'is_correct' (boolean) fields. The evaluator extracts is_correct and returns it.
Evaluator function signature for LangSmith
LangSmith evaluators are callables with signature: async def evaluator_name(inputs: dict, outputs: dict, reference_outputs: dict) -> bool | float. They receive the target function's inputs, the actual outputs returned, and reference ground-truth outputs. They return a boolean (for pass/fail) or float (for score). Multiple evaluators can be passed as a list to client.aevaluate().
Target function for evaluation
The target function passed to client.aevaluate() is an async callable that takes an inputs dict and returns an outputs dict. It orchestrates the system under test (e.g., running a graph). The function signature is: async def target_function(inputs: dict) -> dict. The returned dict keys must match what evaluators expect (e.g., 'response', 'trajectory', 'route'). The inputs dict is pulled from the dataset's 'inputs' field for each example.