agents/prompting

43 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

System prompt to constrain agent behavior

Example system prompt that restricts a model to only answer from its knowledge base: You are a helpful assistant. Check your knowledge base before answering any questions. Only respond to questions using information from tool calls. if no relevant information is found in the tool calls, respond, "Sorry, I don't know."

prepareStep callback for context compaction

prepareStep is a callback that runs before each model step in agents and core functions. It receives the current messages array that will be sent to the model. When prepareStep returns a new messages array, the SDK uses it for the current step and as the base for following steps, allowing you to mutate message state between agent steps.

pruneMessages built-in helper function

pruneMessages is a built-in helper that removes selected messages and message parts from an agent's context. It accepts options for reasoning, toolCalls, and emptyMessages. Example options: reasoning can be 'all' to remove all reasoning messages; toolCalls accepts 'before-last-3-messages' to keep only the last 3 tool calls; emptyMessages accepts 'remove' to delete empty messages.

Token-based compaction trigger pattern

A common pattern for deciding when to compact agent context is to estimate tokens by converting messages to JSON and dividing by 4. Set a threshold like COMPACT_AFTER_TOKENS = 100_000 and trigger pruneMessages when estimateTokens(messages) exceeds this threshold. For tighter accounting, use a real tokenizer or provider usage data.

Agent context compaction with prepareStep

To compact an agent's context: define a compaction trigger (like token count), then in the prepareStep callback check if compaction is needed. If so, return an object with a new messages array generated by pruneMessages or custom logic. The compacted messages persist into later steps while new assistant and tool response messages continue to be appended.

initialMessages and responseMessages in prepareStep

prepareStep receives initialMessages (the original input messages) and responseMessages (the model responses so far) in addition to messages. Use these when you want to rebuild the message state from scratch each step instead of building on previous prepareStep overrides. Example: return {messages: [...initialMessages, ...responseMessages.slice(-10)]} to keep only the last 10 responses.

prepareStep works with ToolLoopAgent, generateText, and streamText

The prepareStep callback pattern for message compaction is consistent across ToolLoopAgent, generateText, and streamText. All three accept a prepareStep option with the same interface: receives messages, initialMessages, responseMessages, and stepNumber; returns an object with a new messages array to override the current message state.

Agent context compaction with generateText example

This example shows how to use prepareStep with generateText to compact context: ```ts import { generateText, isStepCount, pruneMessages } from 'ai'; const result = await generateText({ model: __MODEL__, prompt: 'Read the project documents and summarize the migration plan.', tools: { readDocument, }, stopWhen: isStepCount(10), prepareStep: ({ messages }) => { if (estimateTokens(messages) > COMPACT_AFTER_TOKENS) { return { messages: pruneMessages({ messages, reasoning: 'all', toolCalls: 'before-last-3-messages', emptyMessages: 'remove', }), }; } }, }); ```

Agent context compaction with ToolLoopAgent example

This example shows how to use prepareStep with ToolLoopAgent to compact context when token count exceeds a threshold: ```ts import { ToolLoopAgent, isStepCount, pruneMessages, tool, type ModelMessage, } from 'ai'; import { z } from 'zod'; const COMPACT_AFTER_TOKENS = 100_000; const estimateTokens = (messages: ModelMessage[]) => { return JSON.stringify(messages).length / 4; }; const readDocument = tool({ description: 'Read a document by name', inputSchema: z.object({ name: z.string(), }), execute: async ({ name }) => { return { name, text: await loadLargeDocument(name), }; }, }); const agent = new ToolLoopAgent({ model: __MODEL__, tools: { readDocument, }, stopWhen: isStepCount(10), prepareStep: ({ messages }) => { if (estimateTokens(messages) > COMPACT_AFTER_TOKENS) { return { messages: pruneMessages({ messages, reasoning: 'all', toolCalls: 'before-last-3-messages', emptyMessages: 'remove', }), }; } }, }); const result = await agent.generate({ prompt: 'Read the project documents and summarize the migration plan.', }); ```

OpenAI Responses API capabilities

The OpenAI Responses API offers persistent chat history, web search tool, file search tool, and computer use tool for building agents that can interact with and operate computers.

Call GPT-4o with Responses API using generateText

To use OpenAI's Responses API with the AI SDK, import generateText from 'ai' and openai from '@ai-sdk/openai', then call openai.responses('gpt-4o') as the model parameter.

Persist chat history with previousResponseId

To continue a conversation from a previous response, pass providerOptions: { openai: { previousResponseId: result1.providerMetadata?.openai.responseId } } to the next generateText call.

Continue conversation using Conversation ID

Create a conversation via OpenAI's Conversation API, then continue it by passing providerOptions: { openai: { conversation: 'conv_123' } } to generateText, where 'conv_123' is the Conversation ID.

Migrate from Completions API to Responses API

To migrate, change openai('gpt-4o') to openai.responses('gpt-4o'). Provider-specific options remain in the providerOptions object but the model creation syntax changes.

Generate structured data with Responses API using Output.object

Use generateText with output: Output.object({ schema: z.object({...}) }) to generate type-safe structured JSON data that conforms to a Zod schema with the Responses API.

Enable multi-step agentic behavior with stopWhen

The stopWhen parameter with isStepCount(5) transforms a single LLM call into an agent that can autonomously call tools, analyze results, and make additional tool calls as needed to complete complex tasks.

Prompt engineering best practices for Claude 4

Claude 4 models respond well to clear, explicit instructions. Best practices include: (1) Provide explicit instructions by clearly stating what you want the model to do including specific steps or formats for the response; (2) Include context and motivation by explaining why a task is being performed to help the model understand underlying goals; (3) Avoid negative examples by only demonstrating desired behavior rather than what to avoid.

Enable extended thinking with Claude 3.7 Sonnet

Claude 3.7 Sonnet supports extended thinking for complex reasoning. Enable it using the providerOptions with the anthropic key, setting thinking to { type: 'enabled', budgetTokens: <number> }. The response will include reasoningText containing the model's reasoning steps and reasoning containing reasoning details including redacted reasoning.

Extended thinking example with AI SDK

Example of enabling extended thinking with a 12000 token budget: const { text, reasoningText, reasoning } = await generateText({ model: anthropic('claude-3-7-sonnet-20250219'), prompt: 'How many people will live in the world in 2040?', providerOptions: { anthropic: { thinking: { type: 'enabled', budgetTokens: 12000 } } } });

Reasoning effort control for GPT-5

Use the reasoningEffort parameter in providerOptions.openai to calibrate model autonomy. Set it to 'high' to increase autonomous exploration.

Verbosity control in GPT-5

Use the textVerbosity parameter in providerOptions.openai to control response length. Set to 'low' for terse, minimal responses or 'high' for comprehensive, detailed responses.

Reasoning summaries in GPT-5

Enable reasoning summaries by setting providerOptions.openai.reasoningSummary to 'auto' for condensed summaries or 'detailed' for comprehensive reasoning. Stream reasoning and text separately by checking part.type for 'reasoning' or 'text-delta'.

Prompt engineering core principles for GPT-5

Be precise and unambiguous, avoiding contradictory or ambiguous instructions. Use structured prompts with XML-like tags to organize different sections. Write prompts as you would explain to a skilled colleague while maintaining clarity.

Optimization workflow for GPT-5 prompts

Start with a clear, simple prompt. Test and identify areas of ambiguity or confusion. Iteratively refine by removing contradictions. Consider using OpenAI's Prompt Optimizer tool for complex prompts. Document successful patterns for reuse.

Prompt engineering best practices for o1 models

o1 models perform best with straightforward prompts. Keep prompts simple and direct without extensive guidance. Avoid chain-of-thought prompts since these models perform reasoning internally; prompting them to 'think step by step' or 'explain your reasoning' is unnecessary. Use delimiters like triple quotation marks, XML tags, or section titles to clearly indicate distinct parts of input. When using retrieval-augmented generation (RAG), include only the most relevant information to prevent the model from overcomplicating its response.

DeepSeek R1 prompt engineering best practices

DeepSeek R1 models perform optimally with: (1) structured format using <think> tags for reasoning and <answer> tags for final results, (2) zero-shot prompts avoiding few-shot prompting as it can degrade performance, (3) specified output expectations defining desired formats such as markdown or XML-like tags for clarity.

Prevent model retry after tool denial with system prompt

When a tool execution is denied by the user, add a system prompt instruction like 'When a tool execution is not approved by the user, do not retry it. Inform the user that the action was not performed.' to prevent the model from repeatedly attempting the same tool call.

Use prepareCall to expose usage on agent context

Implement prepareCall callback that receives options and settings, then returns { ...settings, context: { lastInputTokens: options.lastInputTokens } }. This makes the callOptionsSchema data available on the context object that prepareStep can access.

LanguageModelUsage type for tracking tokens

Import LanguageModelUsage from 'ai' package. This type contains inputTokens property and is included in finish-step parts and accessible via message metadata. Use it to define metadata types and track consumption across agent lifecycle.

Define AgentUIMessage with metadata type

InferAgentUIMessage accepts a second generic parameter for metadata type. Define a metadata type like type AgentMetadata = { usage: LanguageModelUsage }, then pass it as the second generic: InferAgentUIMessage<typeof agent, AgentMetadata>. This provides type-safe access to m.metadata.usage on the client.

Track token usage with messageMetadata callback

Pass a messageMetadata callback to toUIMessageStream that receives a part object. Check if part.type === 'finish-step' to access usage data (part.usage), then return an object with the usage metadata. This attaches usage information to each message.

Pass previous message usage back to agent with callOptionsSchema

Define callOptionsSchema on ToolLoopAgent with z.object({ lastInputTokens: z.number() }). In the route handler, extract the last input token count from previous assistant messages (messages.filter(m => m.role === 'assistant').at(-1)?.metadata?.usage?.inputTokens ?? 0) and pass it in options: { lastInputTokens }. This makes cross-request usage accessible to the agent.

Using stopWhen with isStepCount in agent generation

The generateText call can use the stopWhen parameter with isStepCount(5) to limit the agent to a maximum of 5 tool-calling steps, preventing infinite loops in tool usage.

Manual agent loop pattern overview

A manual agent loop pattern allows you to manage the agentic flow yourself rather than using prepareStep and stopWhen, giving you full control over tool execution, message history management, and loop termination conditions. This is useful when you need to implement custom logic between tool calls, handle tool execution errors in specific ways, add custom logging, integrate with external systems, or have complete control over conversation history.

Manual agent loop with streamText example

Here is a complete manual agent loop implementation using Node.js with the AI SDK: ```ts import { ModelMessage, streamText, tool } from 'ai'; import 'dotenv/config'; import z from 'zod'; const getWeather = async ({ location }: { location: string }) => { return `The weather in ${location} is ${Math.floor(Math.random() * 100)} degrees.`; }; const messages: ModelMessage[] = [ { role: 'user', content: 'Get the weather in New York and San Francisco', }, ]; async function main() { while (true) { const result = streamText({ model: 'openai/gpt-4o', messages, tools: { getWeather: tool({ description: 'Get the current weather in a given location', inputSchema: z.object({ location: z.string(), }), }), }, }); for await (const chunk of result.stream) { if (chunk.type === 'text-delta') { process.stdout.write(chunk.text); } if (chunk.type === 'tool-call') { console.log('\nCalling tool:', chunk.toolName); } } const responseMessages = await result.responseMessages; messages.push(...responseMessages); const finishReason = await result.finishReason; if (finishReason === 'tool-calls') { const toolCalls = await result.toolCalls; for (const toolCall of toolCalls) { if (toolCall.toolName === 'getWeather') { const toolOutput = await getWeather(toolCall.input); messages.push({ role: 'tool', content: [ { toolName: toolCall.toolName, toolCallId: toolCall.toolCallId, type: 'tool-result', output: { type: 'text', value: toolOutput }, }, ], }); } } } else { console.log('\n\nFinal message history:'); console.dir(messages, { depth: null }); break; } } } main().catch(console.error); ``` This example demonstrates building a complete manual agent loop where tool execution is handled explicitly within the loop.

Message management in manual agent loop

In a manual agent loop, maintain a messages array that tracks the entire conversation history. After each model response, add the generated messages to this history using `messages.push(...responseMessages)` where responseMessages is obtained from `await result.responseMessages`. This ensures all model responses are recorded in the conversation history.

Loop termination in manual agent loop

The loop continues while the finish reason equals 'tool-calls'. When the finish reason is anything else (typically 'stop'), exit the loop with a break statement. You can customize termination logic to implement your own conditions, such as maximum iterations or time limits.

Maximum iterations control in manual agent loop

Implement a maximum iterations limit in a manual agent loop by tracking an iterations counter and checking it against a MAX_ITERATIONS constant in the while condition. Increment the counter at the start of each loop iteration.

SQL query generation system prompt structure

The system prompt for SQL generation should include: (1) schema description with table and column definitions, (2) rules for handling queries like using ILIKE for case-insensitive string matching, (3) edge case handling like comma-separated fields and whitespace, (4) exact list of available categories to avoid mismatches, (5) data transformation rules like interpreting '10b' as '10.0' billion, (6) rules ensuring output is chart-friendly with at least two columns. The prompt identifies that select_investors is comma-separated, some fields may be null, and quantitative data should always be returned for chart plotting.

Database schema context for SQL generation prompt

The Unicorns table schema used in this guide has columns: id (SERIAL PRIMARY KEY), company (VARCHAR(255) NOT NULL UNIQUE), valuation (DECIMAL(10, 2) NOT NULL), date_joined (DATE), country (VARCHAR(255) NOT NULL), city (VARCHAR(255) NOT NULL), industry (VARCHAR(255) NOT NULL), and select_investors (TEXT NOT NULL). The schema should be included in the system prompt so the model knows what data is available.

Prompt engineering for SQL query context provision

When generating SQL queries from natural language, the prompt should structure context as: (1) Database schema definition showing table names and columns, (2) Explicit rules for string matching (e.g., use ILIKE with LOWER for case-insensitive search), (3) Handling of special field formats (e.g., comma-separated lists in select_investors), (4) Enumeration of valid categories to prevent invalid values, (5) Data semantics (e.g., valuation is in billions), (6) Output requirements (e.g., always return at least two columns for charting), (7) Specific field selection rules (e.g., select identifying columns like company name when asking about specific companies).

System instructions guide agent behavior

System instructions define the agent's role, expertise, behavior, personality, and constraints. They set context for all interactions and guide how the agent responds to queries and uses tools. Use clear, detailed instructions with specific guidelines, rules, and tool usage patterns.

System messages best practice and security

Use the top-level instructions property instead of a system message for system instructions. AI SDK functions reject system messages in the prompt or messages field by default unless allowSystemInMessages is set to true. Opting in to allow system messages in the messages field can create a prompt injection risk if users can inject system messages.

Give your agent this brain