storage/vector-storage

14 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Vector insertion with metadata using putVectors

Insert vectors into an index using the putVectors method. Each vector requires a key, float32 embedding data array, and optional metadata object. The JavaScript example shows calling index.putVectors() with an array of vector objects, each containing key, data.float32 array, and metadata fields like title and source.

Vector storage via S3 Vector Wrapper SQL

Vectors can be inserted using SQL via the S3 Vector Wrapper. Insert into s3_vectors.<index_name> table with key, data (as embd type), and metadata (as jsonb). Example: INSERT INTO s3_vectors.documents_openai (key, data, metadata) VALUES ('doc-1', '[0.1, 0.2, 0.3]'::embd, '{"title": "...", "source": "..."}'::jsonb).

Python vector storage with put method

The Python SDK stores vectors using the put() method on an index. Pass a list of dictionaries with key, data.float32 array, and metadata fields. Example: index.put([{'key': 'doc-1', 'data': {'float32': [0.1, 0.2, 0.3]}, 'metadata': {'title': '...', 'source': '...'}}]).

Vector batching limit

Vectors must be stored in batches with a maximum of 500 vectors per request. For large datasets, split vectors into batches of 500 and process sequentially.

Vector update by key replacement

To update a vector, call putVectors or put() again with the same key but new embedding data and metadata. The system replaces the existing vector. In SQL, delete the old vector and insert a new one with the same key.

Vector deletion via deleteVectors

Delete vectors using the deleteVectors method in JavaScript by passing an array of keys. Example: index.deleteVectors({keys: ['doc-1', 'doc-2']}).

Vector deletion in Python

Delete vectors in Python using the delete method on an index, passing a list of keys. Example: index.delete(['doc-1', 'doc-2']).

Vector deletion via SQL

Delete vectors in SQL using a delete statement on the s3_vectors table. Example: delete from s3_vectors.documents_openai where key in ('doc-1', 'doc-2').

Metadata best practices for vectors

Keep metadata lightweight to reduce query response size. Use consistent data types for the same field across vectors. Mark fields used for filtering to improve query performance. Avoid nested objects; flat structures are easier to filter. Include contextual information like product_id, category, price, tags, and descriptions for filtering and enrichment.

Vector batch processing from JSONL file

Read embeddings from a JSONL file where each line contains key, embedding array, and metadata fields. Parse each line, extract the embedding, and group into batches of 500. Send each batch to putVectors or put() with error handling. Example shows processing embeddings.jsonl file in JavaScript and Python.

Vector storage from OpenAI embeddings API

Generate embeddings using OpenAI's text-embedding-3-small model via the OpenAI API, then store them in Supabase vector storage. Prepare vectors by mapping documents to their embeddings and adding metadata like title and source. Store in batches of max 500 vectors per request. Example shows generating embeddings for multiple documents and batching the storage.

Batch operations performance for vectors

Use batch operations instead of looping individual vector insertions. Sending all vectors in a single putVectors or put() call is more efficient than multiple requests with one vector each. For large datasets, batch into groups of 500.

Lean metadata reduces response size

Keep metadata fields small. Avoid storing full document text, large nested objects, or detailed analysis in metadata. Store only essential filtering fields like doc_id, category, and a brief summary to keep query responses fast.

Vector storage feature status

Vector storage with Supabase is currently in alpha. Expect rapid changes, limited features, and possible breaking updates.

Give your agent this brain