ai-tools/embeddings

68 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

What embeddings are and their primary use cases

Embeddings capture the relatedness of text, images, video, or other types of information. The primary use cases are search (measuring similarity between a search term and body of text), recommendations (measuring similarity between products), classifications (categorizing text), and clustering (identifying trends).

How embeddings represent semantic similarity

Embeddings compress discrete information like words and symbols into distributed continuous-valued data in the form of vectors. When plotted in multidimensional space, phrases or pieces of content with similar meanings sit close together, while unrelated content sits far apart. This geometric property allows computers to understand semantic relationships even between phrases with no common vocabulary.

Typical dimensionality of embedding models

Most embedding models output many dimensions to effectively capture the complexities of human language. For example, the open source gte-small model outputs 384 dimensions. In practice, embeddings have far more dimensions than the simplified 2-dimensional examples used for explanation.

Using embeddings for semantic search workflow

The typical workflow for semantic search with embeddings is: (1) Pre-process your knowledge base and generate embeddings for each page, (2) Store the embeddings to be referenced later, (3) Build a search interface that prompts users for input, (4) Take the user's input, generate a one-time embedding, then perform a similarity search against the pre-processed embeddings using vector math operations like cosine distance, (5) Return the most similar pages to the user.

Why humans understand language and machines need training

Humans use words and symbols to communicate, but words in isolation are mostly meaningless. Understanding requires drawing from shared knowledge and experience. Similarly, neural network models must be trained on millions of examples to understand what each word, phrase, sentence, or paragraph could mean in different contexts.

OpenAI text-embedding model for Retrieval Plugin

The ChatGPT Retrieval Plugin uses OpenAI's text-embedding-ada-002 model to convert document chunks and queries into embeddings. An OpenAI API key is required and should be exported as OPENAI_API_KEY. The API key can be obtained from the User Settings - API keys page on platform.openai.com.

Image search setup with Poetry and Supabase CLI

To set up an image search project, install Poetry with `pip install poetry`, initialize a new Poetry project with `poetry new image-search`, install the Supabase CLI, and initialize Supabase in the project root with `supabase init`. Then start the local Supabase stack with `supabase start` to get the local DB URL.

Image search project dependencies for CLIP model

The image search example requires three main dependencies: `vecs` (Supabase Vector Python Client), `sentence-transformers` (framework for sentence, text and image embeddings, used with OpenAI CLIP model), and `matplotlib` (for displaying image results). Add them with `poetry add vecs sentence-transformers matplotlib`.

CLIP model imports and database connection setup

Import these modules: `from PIL import Image`, `from sentence_transformers import SentenceTransformer`, `import vecs`, `from matplotlib import pyplot as plt`, and `from matplotlib import image as mpimg`. Set the database connection to: `DB_CONNECTION = "postgresql://postgres:postgres@localhost:54322/postgres"`.

Creating and seeding image embeddings with vecs

Create a vector store client with `vx = vecs.create_client(DB_CONNECTION)`. Get or create a collection with `vx.get_or_create_collection(name="image_vectors", dimension=512)`. Load the CLIP model with `model = SentenceTransformer('clip-ViT-B-32')`. Encode images with `model.encode(Image.open('./path/to/image.jpg'))` which returns an embedding. Upsert records with `images.upsert(records=[(id, embedding_vector, metadata_dict), ...])`. Create an index for fast search with `images.create_index()`.

Querying image vectors with text embedding

To search images from a text query, encode the text string with `text_emb = model.encode(query_string)`. Query the collection with `images.query(data=text_emb, limit=1, filters={"type": {"$eq": "jpg"}})` where `data` is required, `limit` specifies the number of records to return, and `filters` applies metadata filtering using the `$eq` operator.

CLIP model uses 512-dimensional embeddings

The 'clip-ViT-B-32' model produces embeddings with 512 dimensions, as specified when creating the collection with `vx.get_or_create_collection(name="image_vectors", dimension=512)`.

Inspecting embeddings in local Supabase dashboard

After seeding embeddings, visit the local Supabase dashboard at `localhost:54323/project/default/editor`, select the `vecs` schema, and view the collection table (e.g., `image_vectors`) to inspect the generated embeddings.

OpenAI CLIP model capabilities

The OpenAI CLIP model was trained on (image, text) pairs and can perform Text-to-Image search, Image-to-Text search, Image-to-Image search, and Text-to-Text search. It can be fine-tuned on custom image and text data using the regular SentenceTransformers training code.

Video search with Mixpeek and Supabase Vector example

This guide demonstrates implementing video search using the Mixpeek Embed API for video processing and embedding, combined with Supabase Vector for storing and querying embeddings. It supports Text-to-Video, Video-to-Text, Video-to-Video, and Text-to-Text search.

Python dependencies for video search implementation

The following Python packages are required: supabase (Supabase Python Client), mixpeek (Mixpeek Python Client for embedding generation), and poetry for dependency management.

Environment variables for Mixpeek and Supabase integration

Three environment variables are required: SUPABASE_URL, SUPABASE_API_KEY, and MIXPEEK_API_KEY. These are imported into the Python script using os.getenv().

Video embeddings table schema

The video_chunks table structure includes: id (text), start_time (float8), end_time (float8), embedding (extensions.vector(768)), and metadata (jsonb). The embedding field uses a 768-dimensional vector type.

Mixpeek video processing parameters

The mixpeek.tools.video.process() method accepts: video_source (URL), chunk_interval (in seconds, example uses 1), and resolution (array format like [720, 1280]).

Mixpeek embed API usage for video and text

The mixpeek.embed.video() method generates embeddings with parameters: model_id (string, example uses 'vuse-generic-v1'), input (base64 string or text string), and input_type (either 'base64' for video chunks or 'text' for text queries).

Vector index creation for video chunks

Create an ivfflat index on the embedding column using cosine similarity for fast search performance: CREATE INDEX ON video_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)

Querying video embeddings with RPC

Use supabase.rpc('match_video_chunks', {...}).execute() to query embeddings. The RPC call takes parameters: query_embedding (the vector to search with), match_threshold (similarity threshold like 0.8), and match_count (number of results to return, example uses 5).

Setting up local Supabase for video search development

Install the Supabase CLI, run 'supabase init' in your project root, then 'supabase start' to start the local Supabase stack. The local database dashboard is accessible at localhost:54323.

Complete video search Python implementation

From supabase import create_client, Client from mixpeek import Mixpeek import os SUPABASE_URL = os.getenv("SUPABASE_URL") SUPABASE_KEY = os.getenv("SUPABASE_API_KEY") MIXPEEK_API_KEY = os.getenv("MIXPEEK_API_KEY") def seed(): supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY) mixpeek = Mixpeek(MIXPEEK_API_KEY) supabase.table("video_chunks").create({ "id": "text", "start_time": "float8", "end_time": "float8", "embedding": "extensions.vector(768)", "metadata": "jsonb" }) video_url = "https://example.com/your_video.mp4" processed_chunks = mixpeek.tools.video.process( video_source=video_url, chunk_interval=1, resolution=[720, 1280] ) for chunk in processed_chunks: print(f"Processing video chunk: {chunk['start_time']}") embed_response = mixpeek.embed.video( model_id="vuse-generic-v1", input=chunk['base64_chunk'], input_type="base64" ) supabase.table("video_chunks").insert({ "id": f"chunk_{chunk['start_time']}", "start_time": chunk["start_time"], "end_time": chunk["end_time"], "embedding": embed_response['embedding'], "metadata": {"video_url": video_url} }).execute() print("Video processed and embeddings inserted") supabase.query("CREATE INDEX ON video_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)").execute() print("Created index") def search(): supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY) mixpeek = Mixpeek(MIXPEEK_API_KEY) query_string = "a car chase scene" text_emb = mixpeek.embed.video( model_id="vuse-generic-v1", input=query_string, input_type="text" ) results = supabase.rpc( 'match_video_chunks', { 'query_embedding': text_emb['embedding'], 'match_threshold': 0.8, 'match_count': 5 } ).execute() if results.data: for result in results.data: print(f"Matched chunk from {result['start_time']} to {result['end_time']} seconds") print(f"Video URL: {result['metadata']['video_url']}") print(f"Similarity: {result['similarity']}") print("---") else: print("No matching video chunks found")

Vector search example: Next.js with OpenAI and Supabase

This example builds a ChatGPT-style doc search using Next.js, OpenAI embeddings, and Supabase with pgvector. The flow converts markdown into embeddings using OpenAI, stores embeddings in Postgres using pgvector, and deploys a function for answering user questions. The full working example is available on GitHub at https://github.com/supabase-community/nextjs-openai-doc-search

Enable pgvector extension in Supabase

To enable vector functionality, run this SQL in a migration file: `create extension if not exists vector with schema public;`

Vector similarity search database function: match_page_sections

Create a database function for similarity search: ```sql create or replace function match_page_sections( embedding extensions.vector(1536), match_threshold float, match_count int, min_content_length int ) returns table ( id bigint, page_id bigint, slug text, heading text, content text, similarity float ) language plpgsql as $$ #variable_conflict use_variable begin return query select nods_page_section.id, nods_page_section.page_id, nods_page_section.slug, nods_page_section.heading, nods_page_section.content, (nods_page_section.embedding <#> embedding) * -1 as similarity from nods_page_section where length(nods_page_section.content) >= min_content_length and (nods_page_section.embedding <#> embedding) * -1 > match_threshold order by nods_page_section.embedding <#> embedding limit match_count; end; $$; ``` The function takes a query embedding, threshold, count limit, and minimum content length, returning matching page sections ordered by similarity. It uses the dot product operator (<#>) which is faster for normalized OpenAI embeddings.

Create embedding for query using OpenAI API

To turn a user question into an embedding for similarity search: ```ts const embeddingResponse = await fetch('https://api.openai.com/v1/embeddings', { method: 'POST', headers: { Authorization: `Bearer ${openAiKey}`, 'Content-Type': 'application/json', }, body: JSON.stringify({ model: 'text-embedding-ada-002', input: sanitizedQuery.replaceAll('\n', ' '), }), }) if (embeddingResponse.status !== 200) { throw new ApplicationError('Failed to create embedding for question', embeddingResponse) } const { data: [{ embedding }], } = await embeddingResponse.json() ```

Perform vector similarity search via RPC

Call the match_page_sections database function using RPC to find relevant sections: ```ts const { error: matchError, data: pageSections } = await supabaseClient.rpc( 'match_page_sections', { embedding, match_threshold: 0.78, match_count: 10, min_content_length: 50, } ) ```

Text completion request to OpenAI API with context

After finding relevant content via similarity search, build a prompt and send a completion request to OpenAI: ```ts const prompt = `You are a very enthusiastic Supabase representative who loves to help people! Given the following sections from the Supabase documentation, answer the question using only that information, outputted in markdown format. If you are unsure and the answer is not explicitly written in the documentation, say "Sorry, I don't know how to help with that." Context sections: ${contextText} Question: """ ${sanitizedQuery} """ Answer as markdown (including related code snippets if available):` const completionOptions = { model: 'gpt-3.5-turbo-instruct', prompt, max_tokens: 512, temperature: 0, stream: true, } const response = await fetch('https://api.openai.com/v1/completions', { method: 'POST', headers: { Authorization: `Bearer ${openAiKey}`, 'Content-Type': 'application/json', }, body: JSON.stringify(completionOptions), }) if (!response.ok) { const error = await response.json() throw new ApplicationError('Failed to generate completion', error) } return new Response(response.body, { headers: { 'Content-Type': 'text/event-stream', }, }) ```

Process OpenAI streaming completion response on frontend

Stream and display the OpenAI completion response on the frontend: ```ts const handleConfirm = React.useCallback( async (query: string) => { setAnswer(undefined) setQuestion(query) setSearch('') dispatchPromptData({ index: promptIndex, answer: undefined, query }) setHasError(false) setIsLoading(true) const eventSource = new SSE(`api/vector-search`, { headers: { apikey: process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY ?? '', Authorization: `Bearer ${process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY}`, 'Content-Type': 'application/json', }, payload: JSON.stringify({ query }), }) function handleError<T>(err: T) { setIsLoading(false) setHasError(true) console.error(err) } eventSource.addEventListener('error', handleError) eventSource.addEventListener('message', (e: any) => { try { setIsLoading(false) if (e.data === '[DONE]') { setPromptIndex((x) => x + 1) return } const completionResponse = JSON.parse(e.data) const text = completionResponse.choices[0].text setAnswer((answer) => { const currentAnswer = answer ?? '' dispatchPromptData({ index: promptIndex, answer: currentAnswer + text, }) return (answer ?? '') + text }) } catch (err) { handleError(err) } }) eventSource.stream() setIsLoading(true) }, [promptIndex, promptData] ) ```

Build script for generating embeddings at build time

Configure package.json scripts to generate embeddings during the build process: ```json "scripts": { "dev": "next dev", "build": "pnpm run embeddings && next build", "start": "next start", "embeddings": "tsx lib/generate-embeddings.ts" } ```

Environment variables for vector search with OpenAI

Required environment variables for the vector search setup: - NEXT_PUBLIC_SUPABASE_URL - NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY - SUPABASE_SECRET_KEY (obtain via `supabase status`) - OPENAI_API_KEY (obtain from https://platform.openai.com/account/api-keys) Do not commit the .env file to source control.

Local Supabase workflow for vector search

Workflow for local development with Supabase: 1) Install latest Supabase CLI, 2) Run `supabase init` in project root, 3) Create migration with `supabase migration new init`, 4) Add SQL to enable pgvector and create schema, 5) Run `supabase start` to apply migrations locally, 6) When ready, run `supabase link --project-ref=your-project-ref` to link to hosted project, 7) Run `supabase db push` to push local changes to hosted database.

Supabase Vecs client for data science and ephemeral workloads

For data science or ephemeral workloads, the Supabase Vecs client is recommended. It requires only a connection string and handles setting up the database to store and query vectors with associated metadata.

pgvector for production Python applications with migrations

For production Python applications with version controlled migrations, register the vector type with your ORM using pgvector. pgvector provides bindings for Django, SQLAlchemy, SQLModel, psycopg, asyncpg, and Peewee.

Supabase Vecs repository location

The Supabase Vecs client is hosted at https://supabase.github.io/vecs/

hybrid_search Postgres function signature

The hybrid_search function accepts parameters: query_text (text, required) - the user's query text, query_embedding (extensions.vector(512), required) - vector representation of user's query produced by embedding model matching the embedding size on documents table, match_count (int, required) - number of records returned, full_text_weight (float, optional, default 1) - weight for full-text search in final score, semantic_weight (float, optional, default 1) - weight for semantic search in final score, rrf_k (int, optional, default 50) - smoothing constant k added to reciprocal rank. Returns setof documents.

hybrid_search function weights

In the hybrid_search function, full_text_weight and semantic_weight parameters control how much each search method contributes to the final score. Both default to 1, meaning equal contribution. A full_text_weight of 2 and semantic_weight of 1 gives full-text search twice as much weight as semantic search in the final ranking.

hybrid_search Postgres function implementation

The hybrid_search function uses Common Table Expressions (CTEs) to perform full-text search and semantic search queries separately, then combines results using Reciprocal Ranked Fusion (RRF). The full_text CTE selects records where fts @@ websearch_to_tsquery(query_text), ranks by ts_rank_cd, limits to least(match_count, 30) * 2. The semantic CTE selects all records, ranks by embedding <#> query_embedding distance, limits to least(match_count, 30) * 2. A full outer join merges the CTEs on id, then joins to documents table. Final ordering uses RRF scoring: coalesce(1.0 / (rrf_k + full_text.rank_ix), 0.0) * full_text_weight + coalesce(1.0 / (rrf_k + semantic.rank_ix), 0.0) * semantic_weight, descending, with final limit of least(match_count, 30).

hybrid_search SQL example

To run hybrid search in SQL: select * from hybrid_search('Italian recipes with tomato sauce', '[...]'::extensions.vector(512), 10); Where the first parameter is the user query, the second is the embedding vector generated from the user query, and the third is the number of records to return.

tsquery operators for hybrid search

The hybrid_search function uses websearch_to_tsquery(query_text) for full-text search. This operator converts user-friendly search queries into proper tsquery format that can be used with the @@ operator for full-text search on tsvector columns.

Vector distance operator for semantic search

The hybrid_search function uses the inner product (<#>) operator for vector similarity ranking: embedding <#> query_embedding. The index must match the operator used; HNSW index should be created with vector_ip_ops for inner product, or with cosine_ops if using cosine distance (<=>), or other appropriate operators.

Hybrid search documents table schema

The documents table for hybrid search in Postgres contains: (1) id: bigint primary key generated always as identity - auto-generated unique ID for the record used to match records during RRF, (2) content: text - the actual text being searched over, (3) fts: tsvector generated always as (to_tsvector('english', content)) stored - auto-generated column using text from content for full text search by keyword, (4) embedding: extensions.vector(512) - vector column storing vector representation from embedding model for semantic search by meaning. Adjust embedding dimensions to match your embedding model output.

Hybrid search indexes for Postgres

Create two indexes for hybrid search: (1) create index on documents using gin(fts); - GIN (generalized inverted) index designed for composite values like tsvector for full-text search, (2) create index on documents using hnsw (embedding vector_ip_ops); - HNSW index for high-performing approximate nearest neighbor search for semantic vector search. Use vector_ip_ops operator because the query uses inner product (<#>) operator; if using cosine distance (<=>), update the index operator accordingly.

When to use structured vs unstructured embeddings metadata

Use structured metadata when fields are known in advance or query patterns are predictable, such as in production Supabase applications. Use unstructured metadata when fields are unknown, user-defined, or when working with data interactively such as in exploratory research or data science work. Both approaches are valid and the choice depends on the specific use-case.

Python vecs client library creates unstructured metadata tables

The Python vecs client library automatically creates tables with unstructured metadata in jsonb format when calling get_or_create_collection. It uses the pattern docs.upsert(vectors=[(id, embedding_array, {metadata_dict})]) to insert embeddings. When moving to production, the auto-generated SQL DDL should be added to database migrations to maintain a single source of truth for the database schema.

Structured metadata with embeddings using dedicated SQL columns

Structured metadata associates embeddings with data stored in dedicated SQL columns. Each metadata field gets its own column, allowing filtering, constraints, indexing, and full SQL operations. This approach is best when metadata fields are known in advance or query patterns are predictable, and fits naturally with traditional Supabase applications managed via database migrations.

Unstructured metadata with embeddings using jsonb column

Unstructured metadata stores all metadata in a flexible jsonb column without specifying expected fields. This approach does not require predefined schemas and is recommended for ephemeral/interactive workloads, user-defined metadata fields, and rapid prototyping. The tradeoff is less flexible querying/filtering capabilities and pushes metadata integrity enforcement onto application code rather than the database.

Hybrid metadata approach combining structured and unstructured

Hybrid metadata uses both dedicated SQL columns for known fields and a jsonb column for unknown/flexible fields in the same table. Known fields should use dedicated columns for best query performance and throughput, while unknown fields are stored in the jsonb column. This approach works when you have a combination of known and unknown metadata fields.

Structured embeddings table schema example

Create a docs table with vector embeddings and structured metadata using dedicated columns. The table has id (uuid primary key), embedding (vector type with dimension), content (text), and url (text) columns. Insert values with the embedding stored as an array like array[0.1, 0.2, 0.3].

Unstructured embeddings table schema example

Create a docs table with vector embeddings and unstructured metadata in a jsonb column. The table has id (uuid primary key), embedding (vector type with dimension), and meta (jsonb) columns. Insert values with the embedding as an array and metadata as JSON like '{"content": "Hello world", "url": "/hello-world"}'.

Hybrid embeddings table schema example

Create a docs table combining structured and unstructured metadata. The table has id (uuid primary key), embedding (vector type with dimension), content (text), url (string), and meta (jsonb) columns. This allows known fields in dedicated columns and unknown fields in the jsonb column for flexibility.

Vector store extension type for embeddings

Embeddings in Supabase are stored using the vector extension type with a specified dimension, specified as extensions.vector(dimension) where dimension is an integer like 3 or 1536. The vector values are inserted as arrays like array[0.1, 0.2, 0.3].

Face similarity search workflow steps

The face similarity search workflow consists of: (1) launching a Postgres database with pgvector, (2) launching a notebook connecting to the database, (3) loading the ashraq/tmdb-people-image celebrity dataset, (4) using the face_recognition model to create embeddings for every celebrity photo, (5) searching for similar faces in the dataset.

Use connection pooling string for Vecs with Google Colab

When using Supabase Vecs with Google Colab, you must use the connection pooling string (domain ending in *.pooler.supabase.com) instead of the direct connection string, because Colab does not support IPv6.

Example: Vecs client initialization

import vecs DB_CONNECTION = "postgresql://<user>:<pa••••••d>@<host>:<port>/<db_name>" # create vector store client vx = vecs.create_client(DB_CONNECTION)

Face similarity search uses pgvector and Supabase Vecs

The face similarity search example uses Postgres with pgvector extension to store embeddings and Supabase Vecs Python client to perform vector operations. The example identifies celebrities who look most similar to a given person by creating embeddings from face photos.

Vecs Python client connection string format

The Vecs Python client is created with vecs.create_client(DB_CONNECTION) where DB_CONNECTION is a PostgreSQL connection string in the format postgresql://<user>:<pa••••••d>@<host>:<port>/<db_name>. The connection string must start with postgresql:// (not postgres://) for SQLAlchemy compatibility.

Roboflow Inference CLIP embedding dimension

CLIP embeddings from Roboflow Inference have 512 dimensions. When creating a vecs collection for storing CLIP embeddings, use dimension=512.

Give your agent this brain