multi-modal/file-upload

13 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Convert files to data URLs helper function

Create a helper function convertFilesToDataURLs that takes a FileList parameter and returns a Promise of an array of file parts. Each file part has type 'file', mediaType (the file's MIME type), and url (the data URL from FileReader.readAsDataURL()). Use Promise.all to handle multiple files concurrently, with each file read asynchronously using FileReader.

File to data URL conversion code example

```tsx async function convertFilesToDataURLs(files: FileList) { return Promise.all( Array.from(files).map( file => new Promise<{ type: 'file'; mediaType: string; url: string; }>((resolve, reject) => { const reader = new FileReader(); reader.onload = () => { resolve({ type: 'file', mediaType: file.type, url: reader.result as string, }); }; reader.onerror = reject; reader.readAsDataURL(file); }), ), ); } ``` This helper converts FileList objects to an array of file parts with data URLs for sending to the model.

File upload and image/PDF rendering in chat UI

Add state for files and a ref to the file input field. In the message rendering loop, check part.type: for text parts render as span, for file parts with mediaType starting with 'image/' render as Next.js Image component with 500x500 dimensions, for file parts with mediaType 'application/pdf' render as iframe with 500x600 dimensions. In the form onSubmit, convert files to data URLs and include them in the message parts array alongside the text.

Full chat UI with file upload code example

```tsx 'use client'; import { useChat } from '@ai-sdk/react'; import { DefaultChatTransport } from 'ai'; import { useRef, useState } from 'react'; import Image from 'next/image'; async function convertFilesToDataURLs(files: FileList) { return Promise.all( Array.from(files).map( file => new Promise<{ type: 'file'; mediaType: string; url: string; }>((resolve, reject) => { const reader = new FileReader(); reader.onload = () => { resolve({ type: 'file', mediaType: file.type, url: reader.result as string, }); }; reader.onerror = reject; reader.readAsDataURL(file); }), ), ); } export default function Chat() { const [input, setInput] = useState(''); const [files, setFiles] = useState<FileList | undefined>(undefined); const fileInputRef = useRef<HTMLInputElement>(null); const { messages, sendMessage } = useChat({ transport: new DefaultChatTransport({ api: '/api/chat', }), }); return ( <div className="flex flex-col w-full max-w-md py-24 mx-auto stretch"> {messages.map(m => ( <div key={m.id} className="whitespace-pre-wrap"> {m.role === 'user' ? 'User: ' : 'AI: '} {m.parts.map((part, index) => { if (part.type === 'text') { return <span key={`${m.id}-text-${index}`}>{part.text}</span>; } if (part.type === 'file' && part.mediaType?.startsWith('image/')) { return ( <Image key={`${m.id}-image-${index}`} src={part.url} width={500} height={500} alt={`attachment-${index}`} /> ); } if (part.type === 'file' && part.mediaType === 'application/pdf') { return ( <iframe key={`${m.id}-pdf-${index}`} src={part.url} width={500} height={600} title={`pdf-${index}`} /> ); } return null; })} </div> ))} <form className="fixed bottom-0 w-full max-w-md p-2 mb-8 border border-gray-300 rounded shadow-xl space-y-2" onSubmit={async event => { event.preventDefault(); const fileParts = files && files.length > 0 ? await convertFilesToDataURLs(files) : []; sendMessage({ role: 'user', parts: [{ type: 'text', text: input }, ...fileParts], }); setInput(''); setFiles(undefined); if (fileInputRef.current) { fileInputRef.current.value = ''; } }} > <input type="file" accept="image/*,application/pdf" className="" onChange={event => { if (event.target.files) { setFiles(event.target.files); } }} multiple ref={fileInputRef} /> <input className="w-full p-2" value={input} placeholder="Say something..." onChange={e => setInput(e.target.value)} /> </form> </div> ); } ``` Complete multi-modal chat UI with file upload that displays text, images, and PDFs in the conversation.

Image editing file parameter format for generateText

When editing images, pass a file object in the content array with type 'file', data property (can be URL or DataContent as string, Uint8Array, ArrayBuffer, or Buffer), and mediaType property specifying the image type (e.g., 'image/jpeg').

Image generation base64 production consideration

Sending base64 image data in chat messages can cause generation to fail because it significantly increases the payload sent to the model. Instead, save generated images to blob storage (e.g., AWS S3, Vercel Blob, Azure Storage) and return a URL in the tool result.

File to data URL conversion function for PDF uploads

Implement convertFilesToDataURLs function that takes a FileList parameter and returns a Promise of an array containing objects with properties: type (string value 'file'), filename (from file.name), mediaType (from file.type), and url (data URL string from FileReader.readAsDataURL result). Use Promise.all to handle multiple files concurrently, creating a FileReader for each file and resolving with the structured object when onload fires.

Message structure for PDF uploads

When sending messages with PDF attachments, structure the message with role 'user' and parts array containing both text parts (type 'text' with text property) and file parts (type 'file' with filename, mediaType, and url properties as data URLs).

PDF file upload form submission with Next.js

Create a client-side form that accepts PDF files with type='file' accept='application/pdf'. Submit the form via fetch to POST /api/analyze with the FormData object containing the file under the 'pdf' field name. The response is text that contains JSON which can be parsed and displayed.

Convert PDF File to base64 data URL for AI SDK

To send a PDF file through the AI SDK, convert the File object to a base64 data URL: await file.arrayBuffer() to get ArrayBuffer, convert to Uint8Array, map each byte to a character string, join into binary string, encode with btoa() to base64, then create data URL as 'data:application/pdf;base64,[base64Data]'.

Send PDF file in message content with generateText

In the messages array, add a message with content array containing both text and file objects. The file object has type: 'file', data: the data URL string, and mediaType: 'application/pdf'. This allows the LLM to process the PDF content alongside text prompts.

Reading image files for API requests

Image data is passed to the AI SDK by reading the file from disk using fs.readFileSync() and passing the buffer directly to the data property of the file content object.

File prompt structure for streaming text

The file content in a streaming message uses an object with type 'file', data as a Buffer (e.g., from fs.readFileSync()), and mediaType as a string indicating the file type. This allows sending file content directly to the model for analysis within the same message as text prompts.

Give your agent this brain