multi-modal/overview

18 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Multi-modal agent definition and capabilities

A multi-modal agent is an AI agent capable of understanding and generating responses in multiple formats. In the context of the AI SDK, this typically refers to the ability to process both images and PDFs - two common document types that modern language models can process natively.

Use generateText for Gemini 2.5 Flash Image generation, not generateImage

Gemini 2.5 Flash Image is a multimodal language model, so you must use the generateText or streamText functions to generate images, not the generateImage function. The model determines which modality to respond in based on your prompt and configuration.

Image generation with Gemini 2.5 Flash using generateText

This example shows how to generate images with Gemini 2.5 Flash Image. Use generateText with model 'google/gemini-2.5-flash-image' and a text prompt. Generated images are returned in result.files array as Uint8Array data. Detailed prompts yield better results. The example creates a picture of a nano banana dish in a fancy restaurant with a Gemini theme.

Image editing with Gemini 2.5 Flash using generateText

To edit existing images, use generateText with model 'google/gemini-2.5-flash-image' and a message array containing text instructions and a file object. The file object has type 'file', data (URL or DataContent), and mediaType. The model can add elements, modify styles, or transform images while maintaining core characteristics. Example: add a wizard hat to a cat image.

Generated images returned in result.files array as Uint8Array

When generating images with Gemini 2.5 Flash Image, the generated images are returned in the result.files array. Each file object contains a Uint8Array of the image data that can be written to disk using fs.promises.writeFile.

GPT-5 key features

GPT-5 offers verbosity control for tailored response lengths, integrated web search capabilities, reasoning summaries for transparency, and native support for text, images, audio, and PDFs.

Multi-Modal Agent guide

The AI SDK provides a guide on how to build a multi-modal agent that can process images and PDFs.

Chat with PDFs server implementation using Claude

Create a route handler at app/api/chat/route.ts that uses Anthropic's Claude model to process messages and PDFs. Import convertToModelMessages, createUIMessageStreamResponse, streamText, toUIMessageStream, and UIMessage type from 'ai'. In the POST handler, extract messages from the request body, call streamText with model 'anthropic/claude-sonnet-4' and the converted messages, then return the response using createUIMessageStreamResponse wrapped around toUIMessageStream.

Chat with PDFs client implementation using useChat hook

Create a client component using the useChat hook from '@ai-sdk/react' with DefaultChatTransport configured to post to '/api/chat'. Implement a convertFilesToDataURLs async function that accepts a FileList and returns an array of objects with type 'file', filename, mediaType, and url (as data URL) by reading each file with FileReader.readAsDataURL(). In the form submission, convert selected files to data URLs and send them as file parts alongside text parts in the message using sendMessage({ role: 'user', parts: [{ type: 'text', text: input }, ...fileParts] }).

generateText with image from URL

To include an image from a URL in a generateText call, use a message content array with a text field and a file field. The file field has type 'file', mediaType 'image', and data as a new URL object pointing to the image URL. The example uses openai/gpt-4.1 model with maxOutputTokens 512.

generateText with image from file buffer

To include an image from a local file in a generateText call, read the file with fs.readFileSync using base64 encoding, then pass it in a message content array. The file field has type 'file', mediaType 'image', and data as the base64-encoded string. The example uses openai/gpt-4.1 model with maxOutputTokens 512.

Image content field format for generateText

When passing images to generateText, use a file content object with three properties: type set to 'file', mediaType set to 'image', and data which can be either a URL object (for remote images) or a base64-encoded string (for local files).

Multimodal message content structure for images

A message with image content uses a content array containing multiple items: a text item with type: 'text' and a text property, and an image item with type: 'file', mediaType: 'image', and a data property containing the binary image buffer.

Image input formats: URL-based images

Images can be included in streamText messages by using a file-type content object with mediaType set to 'image' and data pointing to a URL. The URL is passed as a new URL() object instance.

Image input formats: File buffer (base64-encoded)

Images can be included in streamText messages by using a file-type content object with mediaType set to 'image' and data containing base64-encoded file content. The file is read using fs.readFileSync with encoding set to 'base64'.

Call tools with image prompt using generateText

To call tools with image prompts in the AI SDK, use generateText with messages containing both text and image content. The content array can include text objects with type 'text' and file objects with type 'file' and mediaType 'image'. Define tools using the tool function from the 'ai' package with a description, inputSchema defined with zod, and an execute function that handles the tool logic.

Image file format in tool call messages

When including images in messages for tool calls, use a file content object with type 'file', mediaType set to 'image', and data set to a URL object. The URL should point to the image resource (e.g., new URL('https://...')). This format allows the model to receive and process the image when determining which tool to call.

Example: log food item tool with image input

const result = await generateText({ model: 'openai/gpt-4.1', messages: [ { role: 'user', content: [ { type: 'text', text: 'can you log this meal for me?' }, { type: 'file', mediaType: 'image', data: new URL( 'https://upload.wikimedia.org/wikipedia/commons/thumb/e/e4/Cheeseburger_%2817237580619%29.jpg/640px-Cheeseburger_%2817237580619%29.jpg', ), }, ], }, ], tools: { logFood: tool({ description: 'Log a food item', inputSchema: z.object({ name: z.string(), calories: z.number(), }), execute({ name, calories }) { storeInDatabase({ name, calories }); // your implementation here }, }), }, }); This example shows how to use generateText with a vision-capable model to process an image of food and call the logFood tool to extract and store food information.

Give your agent this brain