openai/capabilities

32 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

OpenAI models support matrix

OpenAI models and their capabilities: gpt-5.6, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-5.5, gpt-5.4-mini, gpt-5.4-nano, gpt-5.2-pro, gpt-5.2, gpt-5.1, gpt-5.1-codex, gpt-5, gpt-5-mini, gpt-4.1, gpt-4.1-mini, gpt-4o, and gpt-4o-mini all support Image Input, Object Generation, Tool Usage, and Tool Streaming.

OpenAI reasoning models usage metadata

OpenAI reasoning models provide usage.outputTokenDetails.reasoningTokens in the response to access the number of reasoning tokens generated by the model.

OpenAI chat models support tool calling

OpenAI chat models support tool calls. This is available through the `.chat()` factory method for models like gpt-4 and gpt-5.

OpenAI image models support multiple input image formats

OpenAI's vision models can accept image content via the message content array using the 'file' type with mediaType 'image'. Input images can be provided as Buffer, ArrayBuffer, Uint8Array, or base64-encoded strings. For gpt-image-* models, each image should be a png, webp, or jpg file less than 50MB.

OpenAI responses API context compaction feature

OpenAI Responses API supports server-side context compaction to automatically compress conversation context when token usage crosses a configured threshold. Configuration requires type: 'compaction' and compactThreshold (token count at which compaction is triggered). The compaction item is opaque and encrypted. When enabled, set store: false for ZDR-friendly operation.

OpenAI responses API compaction in streaming

When using streamText with OpenAI Responses API compaction, you can detect compaction by checking part.providerMetadata?.openai?.type === 'compaction' on 'text-start' and 'text-end' events.

OpenAI text batch API support

OpenAI provider supports the Batch API for text generation. Use experimental_startTextBatch, experimental_getBatchResults, and experimental_getBatchStatus functions. The batch is serializable and can be persisted to check status or retrieve results later.

OpenAI image editing with gpt-image models

OpenAI's gpt-image-* models support powerful image editing capabilities. Pass input images via prompt.images to transform, combine, or edit existing images. Supports inpainting with mask (transparent areas = edit regions), background removal by setting background to 'transparent', and multi-image combining with support for up to 16 input images.

OpenAI image generation metadata response

OpenAI image generation models return metadata via providerMetadata.openai.images array with image-specific metadata: - revisedPrompt (string): The revised prompt used for generation - created (number): Unix timestamp in seconds - size (string): One of 1024x1024, 1024x1536, or 1536x1024 - quality (string): One of low, medium, or high - background (string): Either transparent or opaque - outputFormat (string): One of png, webp, or jpeg

OpenAI PDF support in chat API

OpenAI Chat API supports reading PDF files passed as part of message content using type: 'file' with mediaType: 'application/pdf'. Can pass PDF via: data field with file buffer, file-id from OpenAI Files API, or URL of a PDF. Filename is optional.

OpenAI audio input support with gpt-4o-audio-preview

The gpt-4o-audio-preview model accepts audio files passed as type: 'file' with mediaType: 'audio/mpeg'. This model is in preview and requires at least some audio inputs; it will not work with non-audio data.

OpenAI responses API compaction example code

Example of enabling compaction with OpenAI Responses API: ```ts const result = await generateText({ model: openai.responses('gpt-5.2'), messages: conversationHistory, providerOptions: { openai: { store: false, contextManagement: [{ type: 'compaction', compactThreshold: 50000 }], } satisfies OpenAILanguageModelResponsesOptions, }, }); ```

OpenAI reasoning model example with reasoningEffort

Example of using OpenAI reasoning models with reasoningEffort: ```ts const { text, usage } = await generateText({ model: openai.chat('gpt-5'), prompt: 'Invent a new holiday and describe its traditions.', providerOptions: { openai: { reasoningEffort: 'low', } satisfies OpenAILanguageModelChatOptions, }, }); console.log(text); console.log('Reasoning tokens:', usage.outputTokenDetails.reasoningTokens); ```

OpenAI strict structured outputs example

Example of disabling strict structured outputs: ```ts const result = await generateText({ model: openai.chat('gpt-4o-2024-08-06'), providerOptions: { openai: { strictJsonSchema: false, } satisfies OpenAILanguageModelChatOptions, }, output: Output.object({ schema: z.object({ name: z.string(), ingredients: z.array( z.object({ name: z.string(), amount: z.string(), }), ), steps: z.array(z.string()), }), schemaName: 'recipe', schemaDescription: 'A recipe for lasagna.', }), prompt: 'Generate a lasagna recipe.', }); ```

OpenAI logprobs example

Example of accessing logprobs information: ```ts const result = await generateText({ model: openai.chat('gpt-5'), prompt: 'Write a vegetarian lasagna recipe for 4 people.', providerOptions: { openai: { logprobs: true, } satisfies OpenAILanguageModelChatOptions, }, }); const openaiMetadata = (await result.providerMetadata)?.openai; const logprobs = openaiMetadata?.logprobs; ```

OpenAI image input support example

Example of passing image files to OpenAI chat models: ```ts const result = await generateText({ model: openai.chat('gpt-5'), messages: [ { role: 'user', content: [ { type: 'text', text: 'Please describe the image.', }, { type: 'file', mediaType: 'image', data: readFileSync('./data/image.png'), }, ], }, ], }); ``` Images can also be passed as URLs.

OpenAI PDF support example

Example of passing PDF files to OpenAI chat models: ```ts const result = await generateText({ model: openai.chat('gpt-5'), messages: [ { role: 'user', content: [ { type: 'text', text: 'What is an embedding model?', }, { type: 'file', data: readFileSync('./data/ai.pdf'), mediaType: 'application/pdf', filename: 'ai.pdf', }, ], }, ], }); ``` Can also pass file-id from OpenAI Files API or URL.

OpenAI predicted outputs example

Example of using predicted outputs: ```ts const result = streamText({ model: openai.chat('gpt-5'), messages: [ { role: 'user', content: 'Replace the Username property with an Email property.', }, { role: 'user', content: existingCode, }, ], providerOptions: { openai: { prediction: { type: 'content', content: existingCode, }, } satisfies OpenAILanguageModelChatOptions, }, }); const openaiMetadata = (await result.providerMetadata)?.openai; const acceptedPredictionTokens = openaiMetadata?.acceptedPredictionTokens; const rejectedPredictionTokens = openaiMetadata?.rejectedPredictionTokens; ```

OpenAI audio input example

Example of passing audio to gpt-4o-audio-preview: ```ts const result = await generateText({ model: openai.chat('gpt-4o-audio-preview'), messages: [ { role: 'user', content: [ { type: 'text', text: 'What is the audio saying?' }, { type: 'file', mediaType: 'audio/mpeg', data: readFileSync('./data/galileo.mp3'), }, ], }, ], }); ```

OpenAI transcription example with timestamps

Example of transcription with word-level timestamps: ```ts const result = await transcribe({ model: openai.transcription('whisper-1'), audio: new Uint8Array([1, 2, 3, 4]), providerOptions: { openai: { timestampGranularities: ['word'], } satisfies OpenAITranscriptionModelOptions, }, }); console.log(result.segments); ```

OpenAI speech generation example

Example of generating speech: ```ts const result = await generateSpeech({ model: openai.speech('tts-1'), text: 'Hello, world!', voice: 'alloy', }); ``` Available voices: alloy, ash, coral, echo, fable, onyx, nova, sage, shimmer.

OpenAI image editing with gpt-image models example

Example of transforming an existing image: ```ts const imageBuffer = readFileSync('./input-image.png'); const { images } = await generateImage({ model: openai.image('gpt-image-2'), prompt: { text: 'Turn the cat into a dog but retain the style of the original image', images: [imageBuffer], }, }); ```

OpenAI image inpainting with mask example

Example of inpainting with mask: ```ts const image = readFileSync('./input-image.png'); const mask = readFileSync('./mask.png'); const { images } = await generateImage({ model: openai.image('gpt-image-2'), prompt: { text: 'A sunlit indoor lounge area with a pool containing a flamingo', images: [image], mask: mask, }, }); ``` Transparent areas in the mask indicate where image should be edited.

OpenAI background removal example

Example of removing image background: ```ts const imageBuffer = readFileSync('./input-image.png'); const { images } = await generateImage({ model: openai.image('gpt-image-1.5'), prompt: { text: 'do not change anything', images: [imageBuffer], }, providerOptions: { openai: { background: 'transparent', outputFormat: 'png', } satisfies OpenAIImageModelEditOptions, }, }); ```

OpenAI multi-image combining example

Example of combining multiple images: ```ts const cat = readFileSync('./cat.png'); const dog = readFileSync('./dog.png'); const owl = readFileSync('./owl.png'); const bear = readFileSync('./bear.png'); const { images } = await generateImage({ model: openai.image('gpt-image-2'), prompt: { text: 'Combine these animals into a group photo, retaining the original style', images: [cat, dog, owl, bear], }, }); ``` gpt-image-* models support up to 16 input images.

OpenAI text batch API example

Example of using Batch API for text generation: ```ts const model = openai('gpt-4.1-nano'); const batch = await startTextBatch({ model, requests: [ { id: 'france', prompt: 'What is the capital of France?' }, { id: 'germany', prompt: 'What is the capital of Germany?' }, ], }); const { status } = await getBatchStatus({ model, batch }); if (status !== 'pending') { for await (const result of getBatchResults({ model, batch })) { console.log(result); } } ```

OpenAI response source document handling

OpenAI Responses API returns source document annotations with providerMetadata normalized to camelCase. Three annotation types exist: file_citation (from file_search with fileId, index), container_file_citation (from code_interpreter with containerId, fileId), and file_path (with fileId, index). Filename is available via part.filename.

OpenAI responses API annotation example

Example of handling source document annotations: ```ts for (const part of result.content) { if (part.type === 'source') { if (part.sourceType === 'document') { const providerMetadata = part.providerMetadata as | OpenaiResponsesSourceDocumentProviderMetadata | undefined; if (!providerMetadata) continue; const annotation = providerMetadata.openai; switch (annotation.type) { case 'file_citation': // file_citation has: type, fileId, index break; case 'container_file_citation': // container_file_citation has: type, containerId, fileId break; case 'file_path': // file_path has: type, fileId, index break; } } } } ```

OpenAI realtime API support

OpenAI supports Realtime API via openai.experimental_realtime(modelId) factory method. Realtime is an experimental feature. Sessions run in browser and require short-lived token created via openai.experimental_realtime.getToken() on server.

OpenAI batch APIs are experimental

Text batch APIs are experimental and may change in future releases.

OpenAI realtime is experimental

Realtime is an experimental feature.

OpenAI speech translation is experimental

Speech translation is an experimental feature.

Give your agent this brain