BigQuery replication limitations
BigQuery Pipelines replication has these limitations: row size limited to 10 MB per row due to BigQuery Storage Write API constraints; source tables must have a primary key with at most 16 columns, and BigQuery does not enforce key uniqueness; BigQuery CDC supports at most 2,000 top-level columns; updates and deletes require a supported primary-key identity or REPLICA IDENTITY FULL; source schema and table names cannot start or end with underscore or contain double quote or semicolon; arrays cannot contain NULL elements, and numeric values with more than 38 fractional digits and exact JSON integers outside BigQuery's supported range cannot be replicated; while CDC is active, BigQuery does not support mutating DML (UPDATE, DELETE, MERGE), wildcard table queries, or search indexes on the destination table; do not delete or modify tables or views created by Pipelines, as this can stop replication and require a billable initial sync; schema changes are limited to supported changes only.
BigQuery REPLICA IDENTITY support table
BigQuery replication support by REPLICA IDENTITY setting: REPLICA IDENTITY DEFAULT with primary key—Supported (recommended for most tables; BigQuery uses replicated source primary key for upserts and deletes). REPLICA IDENTITY FULL—Supported (recommended for tables with large text, jsonb, bytea, or TOAST values, especially when rows update). REPLICA IDENTITY USING INDEX—Limited (supported only when the unique index contains exactly the source primary-key columns; alternative unique-key identity not supported). REPLICA IDENTITY NOTHING—Insert-only (inserts replicate but updates and deletes lack row identity for BigQuery to apply safely). REPLICA IDENTITY DEFAULT without primary key—Not supported (BigQuery requires a source primary key).
BigQuery service account IAM permissions required
A GCP service account for BigQuery replication requires the following permissions: bigquery.datasets.get, bigquery.jobs.create, bigquery.tables.create, bigquery.tables.delete, bigquery.tables.get, bigquery.tables.getData, bigquery.tables.list, bigquery.tables.update, bigquery.tables.updateData. These can be granted via BigQuery Data Editor role on the dataset and BigQuery Job User role on the project.
BigQuery destination configuration form fields
When configuring BigQuery as a Supabase Pipelines destination, fill in: Destination name (identifier), Publication (the publication to replicate from), Region (fixed to AWS eu-central-1/Frankfurt, not changeable), Project ID (BigQuery project identifier from GCP Console), Dataset ID (dataset name without project ID—if shown as project-id.dataset-id in GCP Console, enter only the part after the dot), and Service Account Key (JSON format).
BigQuery partial update row issue and REPLICA IDENTITY FULL
Postgres does not always send a complete old row through logical replication and can mark unchanged toasted values as 'unchanged toast' instead of resending the value. BigQuery CDC upserts require a complete new row because omitted columns are not preserved in the destination. If a BigQuery pipeline fails with an error about a partial update row, set REPLICA IDENTITY FULL on the affected source table and restart the pipeline. Changing replica identity only affects new WAL records, so a retained update written before the change may still need to be skipped by recreating the pipeline or restarting the table's initial sync.
BigQuery destination setup prerequisites
Before configuring BigQuery as a Supabase Pipelines destination, you must: 1) Have a Google Cloud Platform account with BigQuery API and BigQuery Storage API enabled, 2) Create a BigQuery dataset in your GCP project, 3) Create a GCP service account key with specific IAM permissions. Supabase Pipelines runs in AWS eu-central-1 (Frankfurt), so choose the closest BigQuery location to reduce network latency. Dataset location cannot be changed after creation.
BigQuery source table primary key requirement
BigQuery replication requires each source table to have a primary key, and the publication must include all primary-key columns. Pipelines declares these columns as the BigQuery destination primary key so BigQuery CDC can apply UPSERT and DELETE operations. BigQuery primary keys are NOT ENFORCED. BigQuery CDC supports composite primary keys with up to 16 columns. The source primary key must remain unique and non-null because BigQuery uses it to match CDC rows.
Check table replica identity SQL query
To check a table's current replica identity, run: select n.nspname as schema_name, c.relname as table_name, c.relreplident as replica_identity from pg_class as c join pg_namespace as n on n.oid = c.relnamespace where n.nspname = 'public' and c.relname = 'your_table'; The replica_identity value is 'd' for default, 'f' for full, 'i' for index, and 'n' for nothing.
BigQuery column defaults handling
Column defaults are handled independently from whether the column allows NULL. BigQuery does not support ADD COLUMN DEFAULT on an existing table, so Pipelines first adds the nullable column and then applies supported default metadata with a separate statement. This is destination metadata for future BigQuery writes that omit the column; pipeline writes already contain the value evaluated by Postgres, and the metadata does not populate existing destination rows. Unsupported defaults, including defaults on destination primary-key columns, are skipped with a warning instead of failing replication.
BigQuery Pipelines advanced settings
BigQuery Pipelines offers these advanced settings: Connection pool size (default 4 connections) controls the number of BigQuery Storage Write API connections for destination writes—more connections increase throughput but consume more resources. Maximum staleness (default Freshest results) sets acceptable staleness in whole minutes of table data while BigQuery applies CDC UPSERT and DELETE changes; leave unset for freshest data, or set a number like 15 to allow data up to 15 minutes stale. This value applies when Pipelines creates or recreates a table; changing it does not alter existing tables.
BigQuery schema changes after table creation constraints
After a BigQuery table exists, BigQuery requires every newly added scalar, top-level column to be NULLABLE. If Postgres adds a NOT NULL column, Pipelines adds it as NULLABLE in BigQuery and logs a warning. For Postgres DROP NOT NULL, Pipelines relaxes an existing BigQuery column from REQUIRED to NULLABLE. For Postgres SET NOT NULL, BigQuery cannot change an existing NULLABLE column to REQUIRED, so Pipelines leaves it nullable and logs a warning. Postgres remains the source of truth and rejects NULL values before they reach the destination.
Set REPLICA IDENTITY FULL SQL command
To set full replica identity on a table: alter table public.your_table replica identity full; REPLICA IDENTITY FULL increases WAL volume because Postgres logs the full old row for updates and deletes. Use it on tables where update correctness is more important than the extra replication overhead.
BigQuery schema creation NULL handling
When the initial sync creates a BigQuery table, Pipelines preserves whether each scalar, non-array source column allows NULL: Postgres NOT NULL columns become REQUIRED in BigQuery, and nullable columns become NULLABLE. BigQuery represents Postgres arrays as REPEATED fields instead of using REQUIRED or NULLABLE mode.
BigQuery replication how it works
Once configured, replication to BigQuery: 1) Captures INSERT, UPDATE, DELETE, and TRUNCATE operations included by the Postgres publication, 2) Optimizes delivery automatically, 3) Creates destination tables from the replicated source schema using BigQuery-compatible names and types, 4) Streams data to BigQuery. Pipelines keeps a current-state table for each replicated source table and may replace its destination data during truncate or new initial sync. It does not provide a history of every row version.
Configuring replication in Supabase Dashboard
You can view publications in the Dashboard but all steps to configure replication must be done using the SQL Editor or a CLI tool of your choice. Replication cannot be fully configured through the Dashboard.
XMIN replication method and when to use it
Xmin is a different form of replication from logical replication and should only be used if logical replication is not available for your database (i.e. older versions of Postgres). Xmin performs replication by checking the xmin system column and determining if that row has already been synchronized. It does not capture deletion of data and is not recommended, particularly for larger databases.
Configure database settings for replication using Supabase CLI
Using the Supabase CLI, you can configure database settings to optimize them for your replication needs. These values can vary depending on your database size and activity.
Important WAL and replication slot configuration options
Key configuration options for database replication are: max_wal_size (maximum size the WAL can grow between automatic WAL checkpoints), max_slot_wal_keep_size (maximum size of WAL files that replication slots are allowed to retain), wal_keep_size (minimum number of past WAL files to keep for standby servers), and max_wal_senders (maximum number of concurrent connections from standby servers or streaming backup clients). These settings help ensure replication slots don't run out of space and replicas can reconnect without requiring a full re-sync.
IPv6 support for replication tools
If the replication tool in use does not support IPv6, you can enable the IPv4 add-on for your Supabase project.
Direct connection string required for logical replication
Always use the direct connection string for logical replication. Connections through a pooler, such as Supavisor, will not work.
Replication slot states in pg_replication_slots
A replication slot can be in one of three states: active (the slot is receiving data), inactive (the slot is not receiving data), or lost (the slot is lost and not receiving data). Check with query: select slot_name, active, state from pg_replication_slots;
pg_stat_replication_send_lag indicates publisher or network issues
pg_stat_replication_send_lag measures the lag in sending WAL files from the source database. A high send lag means either the publisher is not being asked to send new WAL files or there are network issues.
Query replication status on primary with pg_stat_replication
The pg_stat_replication table on the primary shows the status of any replicas connected to the primary database. Example query: select pid, application_name, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, sync_state from pg_stat_replication;
pg_stat_replication_replay_lag definition
pg_stat_replication_replay_lag measures the lag to replay WAL files from the source database on the target database. This lag is throttled by disk speed or high activity on the target.
pg_stat_subscription null PID indicates inactive subscription
In pg_stat_subscription on the subscriber, if the PID is null, then the subscription is not active.
pg_stat_subscription_stats error_count field
The pg_stat_subscription_stats table has an error_count field that shows if there were issues applying or syncing data. If error_count is not zero, check the logs to determine the cause.
Three ways to monitor replication lag in Supabase
Replication lag can be monitored through three methods: Dashboard (view replication lag in Reports), Database (using pg_stat_subscription, pg_stat_subscription_stats, and pg_replication_slots tables), and Metrics (using the prometheus endpoint with replication_slots_max_lag_bytes, pg_stat_replication_replay_lag, and pg_stat_replication_send_lag).
replication_slots_max_lag_bytes is the most important replication metric
When monitoring replication through the prometheus endpoint metrics, replication_slots_max_lag_bytes is the more important metric to track for replication lag.
Subscription states in pg_subscription_rel.srsubstate
The srsubstate column in pg_subscription_rel can have four values: i (Initializing - subscription is being initialized), d (Data Synchronizing - doing initial copy), s (Synchronized - subscription is synchronized), r (Replicating - replicating data).
Query WAL size with pg_ls_waldir function
WAL size can be checked using the pg_ls_waldir() function. Example query: select * from pg_ls_waldir();
Query last replayed LSN on subscriber with pg_last_wal_replay_lsn
The last replayed LSN on the subscriber can be checked using: select pg_last_wal_replay_lsn();
Query subscription status on subscriber
Subscription status on the subscriber can be checked by joining pg_subscription and pg_subscription_rel tables. The query retrieves subscription_name, table_name, replication_state (i/d/s/r), state_description, and last_synced_lsn for each subscribed table.
Query current LSN on primary with pg_current_wal_lsn
The current LSN on the primary can be checked using: select pg_current_wal_lsn();
Publication changes require pipeline restart
After modifying a Postgres publication, the replication pipeline must be restarted for changes to take effect.
Do not manually delete Pipelines-managed destination tables
Do not delete or modify tables or views managed by Pipelines. For BigQuery, deleting a managed object can stop replication and may require a new, billable initial sync. Pipelines does not guarantee that it will automatically repair or fully resynchronize a destination object that was removed manually.
Two options to permanently remove a table from destination
To permanently remove a table from the destination: Option 1 (end replication permanently): delete the destination to permanently end replication and pipeline-hour billing (stopping the pipeline alone does not end pipeline-hour billing), delete the table at the destination, and do not restart the pipeline while the table remains in its publication. Option 2 (remove from publication first): remove the table from the Postgres publication using ALTER PUBLICATION ... DROP TABLE, restart the replication pipeline to apply the change (the table at the destination will remain but stop receiving new changes), and delete the table at the destination. Removing a table from the publication and restarting the pipeline does not delete the table downstream, it only stops replicating new changes to it.
Stopping pipeline keeps changes queued in WAL
Stopping replication causes changes to queue up in the WAL. Stopping requests a graceful shutdown, so the pipeline can remain Stopping for up to five minutes while in-flight work finishes. Configured pipeline-hour billing continues while the pipeline is stopped. Delete the destination to end that charge.
Free plan downgrade deletes all Pipelines
When a project is downgraded to the Free Plan, all replication pipelines created with Pipelines for that project are deleted.
Inactive project stops pipelines without auto-resume
If a project becomes inactive, Pipelines stops any running pipelines and does not automatically resume them after the project is restarted. After restarting the project, each replication pipeline must be restarted manually from the Database > Replication section of the Dashboard.
Table error state occurs during initial sync
Table errors occur during the initial sync. To recover, click View pipeline, find the affected table, and click its restart action. This restarts that table's initial sync from the beginning, deletes its existing destination data, and bills the successfully processed row data again.
Replication lag causes and investigation
Lag increases when Postgres produces WAL faster than the pipeline can confirm it has processed. Common causes include a slow or rate-limited destination, a pipeline issue, heavy source database activity, long transactions, network latency between the pipeline and source database, or a stopped/disconnected pipeline. Check Waiting to sync, WAL retention remaining, Last check-in, Connected, and Slot status in the Database > Replication section.
Pipeline Stopped state causes from manual stop or inactive project
A pipeline is stopped when you select Stop pipeline, when its project becomes inactive, or while an operation requires it to restart. Non-retryable configuration, schema, or data errors put the pipeline in Failed state instead. Transient connection and destination errors retry automatically when possible.
Pipeline Failed state indicates non-retryable error
A pipeline enters Failed state when it encounters a non-retryable pipeline-level error during startup or ongoing replication. The pipeline stops instead of silently skipping the failure. To recover: check the error message by hovering over the Failed status, click View pipeline for detailed information, fix the underlying issue (e.g., schema mismatches, destination connectivity), and restart the pipeline.
Replica identity requirements for updates and deletes
If inserts replicate but updates or deletes fail, the source table might not be sending enough old-row identity through Postgres logical replication. Every table sent to BigQuery must have a primary key with every primary-key column included in the publication. For updates and deletes, use the primary key replica identity or REPLICA IDENTITY FULL. Full replica identity is recommended for tables with large text, jsonb, bytea, or other values that Postgres can store out of line. Use: ALTER TABLE public.your_table REPLICA IDENTITY FULL. Full replica identity increases WAL volume and only affects new WAL records. Fix the setting before generating more changes.
Partitioned tables replication depends on publish_via_partition_root
Whether partitioned tables replicate as separate tables depends on the publication's publish_via_partition_root setting. If false or not set in manually created publications, Postgres publishes changes from leaf partitions and Pipelines creates destination tables for those leaf partitions. If publish_via_partition_root = true, Postgres publishes changes as the partition root and the partition hierarchy is treated as the published partition root. Publications created from the Dashboard replication flow use publish_via_partition_root = true.
Lost slot status means WAL files were removed
Lost slot status means Postgres has already removed WAL files that the pipeline's replication slot needed. The pipeline cannot continue from that slot. Recovery options: recreate the pipeline, or open Advanced settings, set Invalidated slot behavior to Recreate, and start the pipeline again. The recreate option resets saved table-sync state, creates a new replication slot, and replaces each destination table through a new initial sync. This destructive restart is required for consistency because the old slot can no longer provide every change the pipeline missed, and the data processed during the new initial sync is billed again.
Generated columns are skipped during replication
Generated columns are skipped during replication to Pipelines destinations.
BigQuery requires tables to have primary keys
BigQuery requires each source table to have a primary key and requires the publication to include its columns. This is a BigQuery Pipelines requirement, not a general requirement for publishing Postgres inserts.
Disabling Pipelines removes database objects
Disabling Pipelines removes the database objects installed in the source database, including the replication state tables in the etl schema and the DDL event trigger. To remove Pipelines from a project, delete all pipeline destinations first, then the disable action becomes available. Disabling Pipelines stops Supabase from managing replication but does not delete tables or data already written to the destination.
Pipelines schema changes supported for BigQuery
Schema change support is currently in beta and limited to BigQuery destination. Supported changes are: adding a scalar, top-level column (created as NULLABLE in BigQuery), removing a column, renaming a column, dropping a NOT NULL constraint, and setting or dropping supported column default metadata. Initial BigQuery table creation preserves whether each scalar, non-array Postgres column allows NULL. Arrays use BigQuery's REPEATED mode. DROP NOT NULL relaxes a BigQuery REQUIRED column to NULLABLE. SET NOT NULL leaves an existing BigQuery column nullable and logs a warning. Newly added scalar, top-level columns are always nullable in BigQuery. Supported defaults are applied separately as destination metadata and do not populate existing destination rows. Pipelines does not currently support changing column data types.
Pipelines installs event trigger and etl schema tables
When Pipelines is enabled, Supabase installs an event trigger that runs on every ALTER TABLE statement to support schema change handling, and a set of tables in the etl schema to track replication state for pipelines. The replication state tables are not updated very often, especially after the initial sync is complete.
Pipelines requires Pro, Team, or Enterprise plan
Pipelines is only available on Pro, Team, or Enterprise plans. During public alpha, access is rolled out gradually, so an eligible plan does not guarantee that Pipelines is enabled for every organization. If not available, request access from the Database > Replication page or contact your account manager.
BigQuery is the only managed Pipelines destination
BigQuery is currently the only supported managed destination for Supabase Pipelines. ClickHouse, Snowflake, and DuckLake are available for early access request only. Supported destinations can be Supabase-managed or third-party systems as support expands.
Replication logs location in Dashboard
Navigate to the Logs > Replication section of the Dashboard to see all pipeline logs. Logs contain diagnostic information useful for troubleshooting issues.
Replication lag metric: Slot status
The 'Slot status' metric shows how safely Postgres is keeping the WAL files the pipeline still needs. 'Unreserved' and 'Lost' status values require action.
Replication lag metric: Connected
The 'Connected' metric indicates whether the pipeline's replication slot is active and currently being used. 'Not connected' while the pipeline should be running usually means you should check pipeline status and logs.
Replication lag metric: Last check-in
The 'Last check-in' metric shows how long it has been since the pipeline last sent replication feedback to Postgres. An old value can mean the pipeline is stopped, disconnected, overloaded, or unable to make progress.
Replication lag metric: WAL retention remaining
The 'WAL retention remaining' metric indicates how much WAL can still accumulate before the replication slot is at risk of becoming unusable, controlled by the max_slot_wal_keep_size parameter. A small or shrinking value means you should investigate before required WAL is removed. 'Unlimited' means Postgres is not reporting a slot WAL retention limit.
Replication lag metric: Waiting to sync
The 'Waiting to sync' metric shows bytes of WAL between the pipeline's confirmed flush position and the current Postgres WAL position, representing the main byte-based replication lag. A value that keeps growing indicates the pipeline is receiving changes more slowly than Postgres produces them.
Pipeline states and their meanings
Supabase Pipelines can be in these states: Stopped (pipeline is not running), Starting (pipeline is being started), Running (pipeline is actively replicating data), Stopping (pipeline is being stopped), Restarting (pipeline settings or table state are being applied before replication restarts), Failed (pipeline has encountered an error), or Unknown (the Dashboard cannot currently determine the pipeline status).