prompts

36 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

providerOptions at function call level

Functions like `streamText` or `generateText` accept a `providerOptions` property for passing provider-specific metadata. Use this level when granular control is not needed. Example: `providerOptions: { openai: { reasoningEffort: 'low' } }`

providerOptions at message level

Provider options can be passed to individual message objects for granular control. Add `providerOptions` to the message object itself. Example: `instructions: { role: 'system', content: '...', providerOptions: { anthropic: { cacheControl: { type: 'ephemeral' } } } }`

providerOptions at message part level

Certain provider-specific options require configuration at the message part level. Add `providerOptions` to individual content parts within a message, such as text or file parts.

Text prompts - basic usage with generateText

Text prompts are strings used for simple generation use cases. Set them using the `prompt` property in AI SDK functions like `streamText` or `generateText`. The prompt can be a simple string or use template literals to inject variables dynamically.

System prompts with instructions property

System prompts are initial instructions that guide and constrain model behavior. Set them using the `instructions` property in functions like `generateText`. System prompts work with both `prompt` and `messages` properties. System messages in `prompt` or `messages` are rejected by default.

allowSystemInMessages option security risk

Setting `allowSystemInMessages: true` allows system messages in message histories but creates a prompt injection risk where users can override system prompts. In most cases, only trusted server-side code should set system instructions via the `instructions` property.

Message prompts structure

A message prompt is an array of user, assistant, and tool messages. Set using the `messages` property. Each message has a `role` and `content` property. Content can be text (for user and assistant messages) or an array of relevant parts (data) for that message type. Message prompts are great for chat interfaces and complex multi-modal prompts.

UIMessage objects do not support provider options

AI SDK UI hooks like `useChat` return arrays of `UIMessage` objects, which do not support provider options. Use the `convertToModelMessages` function to convert `UIMessage` objects to `ModelMessage` objects before applying or appending message(s) with `providerOptions`.

Text content in user messages

Text content is the most common message content type. It is a string passed to the model. If sending only text, the `content` property can be a string, but it can also be an array of content parts that includes text and other types.

Image formats supported in user messages

User messages can include image parts in multiple formats: base64-encoded (string or data URL), binary (ArrayBuffer, Uint8Array, or Buffer), or URL (http(s) URL string or URL object). Image parts use `type: 'file'` with `mediaType: 'image'` and `data` property.

File parts provider support limitations

Only a few providers and models support file parts: Google Generative AI, Google Vertex AI, OpenAI (for wav/mp3 audio with gpt-4o-audio-preview and for pdf), and Anthropic.

File parts in user messages

User messages can include file parts in formats: base64-encoded (string or data URL), binary (ArrayBuffer, Uint8Array, or Buffer), or URL (http(s) URL string or URL object). File parts require specifying the MIME type via `mediaType` property. Optional `filename` property is not used by all providers.

experimental_download custom download function

Custom download functions can be passed via the `experimental_download` property to implement throttling, retries, authentication, caching, and more. The function receives an array of objects with `url` (URL object) and `isUrlSupportedByModel` (boolean) and returns a Promise of an array with `data` (Uint8Array) and `mediaType` (string or undefined). The default implementation automatically downloads files in parallel when not supported by the model.

experimental_download is experimental

The `experimental_download` option is experimental and may change in future releases.

Assistant messages content types

Assistant messages have a role of 'assistant' and are typically previous responses from the assistant. They can contain text, reasoning, tool call parts, and file parts (for model-generated files on supported models).

Tool call parts in assistant messages

Assistant messages can contain tool call parts with structure: `type: 'tool-call'`, `toolCallId`, `toolName`, and `input`. A single assistant message can call multiple tools (parallel calling).

Tool messages with tool results

For models supporting tool calls, tool messages contain tool result parts with structure: `type: 'tool-result'`, `toolCallId` (must match tool call id), `toolName`, and `output`. A single tool message can contain multiple tool results (parallel calling).

Tool result output types

Tool result `output` can be `{ type: 'json', value: {...} }` for simple outputs or `{ type: 'content', value: [...] }` for multi-part tool results containing text and images.

System messages in message array

System messages are messages with `role: 'system'` sent before user messages to guide assistant behavior. Alternatively, use the `instructions` property instead of including system messages in the messages array.

Model-generated file content in assistant messages

Assistant messages can include file parts for model-generated files using `type: 'file'` with `mediaType` and `data`. Only a few models support this, and only for file types they can generate.

Binary image example with Buffer

```ts const result = await generateText({ model, messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe the image in detail.' }, { type: 'file', mediaType: 'image', data: fs.readFileSync('./data/comic-cat.png'), }, ], }, ], }); ``` This example shows how to send a binary image from a Buffer in user message content.

Base64-encoded image example

```ts const result = await generateText({ model: __MODEL__, messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe the image in detail.' }, { type: 'file', mediaType: 'image', data: fs.readFileSync('./data/comic-cat.png').toString('base64'), }, ], }, ], }); ``` This example shows how to send a base64-encoded image string in user message content.

Image URL example

```ts const result = await generateText({ model: __MODEL__, messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe the image in detail.' }, { type: 'file', mediaType: 'image', data: 'https://github.com/vercel/ai/blob/main/examples/ai-functions/data/comic-cat.png?raw=true', }, ], }, ], }); ``` This example shows how to send an image via HTTPS URL in user message content.

PDF file example with Buffer

```ts import { google } from '@ai-sdk/google'; import { generateText } from 'ai'; const result = await generateText({ model: google('gemini-2.5-flash'), messages: [ { role: 'user', content: [ { type: 'text', text: 'What is the file about?' }, { type: 'file', mediaType: 'application/pdf', data: fs.readFileSync('./data/example.pdf'), filename: 'example.pdf', }, ], }, ], }); ``` This example shows how to send a PDF file from a Buffer in user message content.

Audio file example with mp3

```ts import { openai } from '@ai-sdk/openai'; import { generateText } from 'ai'; const result = await generateText({ model: openai('gpt-4o-audio-preview'), messages: [ { role: 'user', content: [ { type: 'text', text: 'What is the audio saying?' }, { type: 'file', mediaType: 'audio/mpeg', data: fs.readFileSync('./data/galileo.mp3'), }, ], }, ], }); ``` This example shows how to send an mp3 audio file from a Buffer in user message content with OpenAI's gpt-4o-audio-preview model.

Tool result with multimodal content example

```ts const result = await generateText({ model: __MODEL__, messages: [ // ... { role: 'tool', content: [ { type: 'tool-result', toolCallId: '12345', toolName: 'get-nutrition-data', output: { type: 'content', value: [ { type: 'text', text: 'Here is the nutrition data for the cheese:', }, { type: 'file-data', data: fs .readFileSync('./data/roquefort-nutrition-data.png') .toString('base64'), mediaType: 'image/png', }, ], }, }, ], }, ], }); ``` This example shows how to create a multi-part tool result containing both text and an image for models that support multi-part tool results.

Text prompt with template literal example

```ts const result = await generateText({ model: __MODEL__, prompt: `I am planning a trip to ${destination} for ${lengthOfStay} days. ` + `Please suggest the best tourist activities for me to do.`, }); ``` This example shows how to use template literals to inject dynamic variables into a text prompt.

System prompt with instructions example

```ts const result = await generateText({ model: __MODEL__, instructions: `You help planning travel itineraries. ` + `Respond to the users' request with a list ` + `of the best stops to make in their destination.`, prompt: `I am planning a trip to ${destination} for ${lengthOfStay} days. ` + `Please suggest the best tourist activities for me to do.`, }); ``` This example shows how to combine system instructions with a text prompt.

LLMs learn from massive text corpuses

Large Language Models learn by training on massive text corpuses, which means they are better suited to some use cases than others. For example, a model trained on GitHub data would understand the probabilities of sequences in source code particularly well. However, generated sequences can sometimes be random and not grounded in reality.

What is a prompt

Prompts are the starting points for LLMs. They are the inputs that trigger the model to generate text. The scope of prompt engineering involves crafting prompts and understanding related concepts such as hidden prompts, tokens, token limits, and the potential for prompt hacking, which includes phenomena like jailbreaks and leaks.

Why prompt engineering is needed

Prompt engineering plays a pivotal role in shaping the responses of LLMs. It allows tweaking the model to respond more effectively to a broader range of queries. This includes techniques like semantic search, command grammars, and the ReActive model architecture. Performance, context window, and cost of LLMs vary between models and providers, creating trade-offs between cost and performance.

Temperature controls model confidence and randomness

Temperature is a value from 0 to 1 that governs the model's confidence level in making predictions. A lower temperature (closer to 0) implies lesser risk, leading to more precise and deterministic completions. A higher temperature (closer to 1) yields a broader range of completions with more variation.

Temperature 0 produces deterministic results

When temperature is set to 0, the same prompt yields the same or nearly the same completions each time, because the model makes predictions with highest confidence and no randomness.

Using examples in prompts improves output

Incorporating examples in your prompt can aid in conveying patterns or subtleties to the model. Providing examples of expected output for certain inputs prompts the model to generate the kind of results aimed for, which can lead to better quality completions.

Prompt specificity influences completions

Making instructions more specific influences the model's completions. For example, adding a single descriptive term like 'organic' to a prompt changes the output. Crafting your prompt is the means by which you instruct or program the model.

2.x prompt helpers removed

Prompt helpers for constructing message prompts are no longer needed with the AI SDK provider architecture and have been removed.

Give your agent this brain