llama-cpp/capabilities

8 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

llama.cpp limitations

The llama.cpp provider has the following limitations: macOS only (Windows and Linux are not supported), no tool/function calling support, and no image input support (only text prompts).

llama.cpp capabilities

The llama.cpp provider supports: native C++ bindings with N-API, GPU acceleration with automatic Metal support on macOS, streaming and non-streaming text generation via generateText and streamText, structured output with JSON schema validation using Output, embeddings via embed and embedMany, automatic or configurable chat template formatting, and GGUF model format support.

llama.cpp generation parameters

The llama.cpp provider supports standard AI SDK generation parameters: maxTokens, temperature, topP, topK, and stopSequences.

llama.cpp text generation example

import { generateText } from 'ai'; import { llamaCpp } from 'ai-sdk-llama-cpp'; const model = llamaCpp({ modelPath: './models/llama-3.2-1b-instruct.Q4_K_M.gguf', }); try { const { text } = await generateText({ model, prompt: 'Explain quantum computing in simple terms.', }); console.log(text); } finally { await model.dispose(); }

llama.cpp streaming example

import { streamText } from 'ai'; import { llamaCpp } from 'ai-sdk-llama-cpp'; const model = llamaCpp({ modelPath: './models/llama-3.2-1b-instruct.Q4_K_M.gguf', }); try { const result = streamText({ model, prompt: 'Write a haiku about programming.', }); for await (const chunk of result.textStream) { process.stdout.write(chunk); } } finally { await model.dispose(); }

llama.cpp structured output example

import { generateText, Output } from 'ai'; import { z } from 'zod'; import { llamaCpp } from 'ai-sdk-llama-cpp'; const model = llamaCpp({ modelPath: './models/your-model.gguf', }); try { const { output: recipe } = await generateText({ model, output: Output.object({ schema: z.object({ name: z.string(), ingredients: z.array( z.object({ name: z.string(), amount: z.string(), }), ), steps: z.array(z.string()), }), }), prompt: 'Generate a recipe for chocolate chip cookies.', }); console.log(recipe); } finally { await model.dispose(); }

llama.cpp embedding model example

import { embed, embedMany } from 'ai'; import { llamaCpp } from 'ai-sdk-llama-cpp'; const model = llamaCpp.embedding({ modelPath: './models/nomic-embed-text-v1.5.Q4_K_M.gguf', }); try { const { embedding } = await embed({ model, value: 'Hello, world!', }); const { embeddings } = await embedMany({ model, values: ['Hello, world!', 'Goodbye, world!'], }); } finally { model.dispose(); }

llama.cpp structured output uses GBNF

The structured output feature in llama.cpp uses GBNF grammar constraints to ensure the model generates valid JSON that conforms to your schema.

Give your agent this brain