Replication lag metric: Slot status
The 'Slot status' metric shows how safely Postgres is keeping the WAL files the pipeline still needs. 'Unreserved' and 'Lost' status values require action.
Supabase · Database · all subjects
database/replication
127 notes in this subject, read out of this brain and free to use. This is page 2 of 3.
Pipeline replication slot architecture
Pipelines uses one main pipeline replication slot for ongoing replication. During the initial sync, it can also create temporary table-sync replication slots. These temporary slots let multiple tables sync in parallel, make large initial syncs faster, and allow individual tables to be retried without restarting the whole pipeline. Temporary table-sync slots show the same kind of lag and slot health metrics while they are active. After a table finishes its initial sync and catches up, its temporary slot is removed and ongoing replication continues through the main pipeline slot.
Replication slot statuses explained
Replication slot statuses are: Reserved (healthy; Postgres is keeping the WAL files this pipeline's replication slot needs within the normal WAL size limit), Extended (healthy but growing; the slot holds more WAL than usual but Postgres is keeping everything needed), Unreserved (at risk; Postgres is no longer reserving all WAL files this pipeline's replication slot needs), Lost (broken; some WAL files needed have already been removed and the pipeline cannot continue from this slot), or Unknown (Postgres reported an unknown or unavailable state).
Table states during replication
During replication, individual tables can be in these states: Queued (table is waiting for the pipeline to begin its initial sync), Copying (existing rows are being copied during the initial sync), Copied (initial sync is complete and the table is preparing to replicate changes), Live (table is now receiving ongoing replication), Error (table has experienced an error during replication), Restarting (the table's initial sync is being restarted), Not Available (table state is temporarily unavailable while the pipeline changes state), or Unknown (the Dashboard received a table state it does not recognize).
Sources of replication lag in pipelines
Replication lag can come from: destination throughput (the destination is slow, rate-limited, unavailable, or rejecting writes), pipeline throughput (the pipeline is overloaded, processing a very large transaction, or not performing as expected), source database activity (Postgres is producing WAL faster than the pipeline can consume it during bulk writes or migrations), network latency (latency or instability between pipeline and source database), stopped or disconnected pipeline (Postgres keeps WAL for the slot until retention limit is reached), or slow initial sync (a temporary table-sync slot can fall behind if copying is slower than the table's write rate).
Initial sync bottleneck: table-sync slot issues
A common initial sync issue occurs when a large or busy table is still in Copying state while new rows keep being inserted or updated. The temporary table-sync slot retains changes during the initial sync. If copying is too slow compared to the table's write rate, the slot can move to Unreserved and then Lost if Postgres removes changes the sync still needs. When a table-sync slot is lost, the affected table needs to run its initial sync again.
Tuning pipeline initial sync performance
To improve initial sync performance: increase 'Copy connections per table' when one large table is the bottleneck to let the pipeline copy chunks of that table over multiple source connections, up to the point where the source database, network, or destination becomes the limit. Increase 'Table sync workers' when several tables need to copy at the same time; each worker can copy one table and uses an additional temporary replication slot during initial sync. Run the initial sync during a quieter write period or reduce bulk writes until the table reaches Live state if possible.
Investigating replication lag steps
To investigate replication lag: Open Database > Replication and check the destination's lag column. Click View pipeline and check Waiting to sync, WAL retention remaining, Last check-in, Connected, and Slot status. Check table states; tables in Copying can create temporary lag while initial sync catches up to ongoing changes. If a table-sync slot is Unreserved or Lost, tune copy parallelism and retry the affected table's initial sync. Open Logs > Replication and look for destination errors, retries, rate limits, schema errors, or repeated restarts. Compare the lag trend with recent database activity such as imports, migrations, bulk updates, or long transactions.
Handling Reserved slot status
When slot status is Reserved: if 'Waiting to sync' is stable or decreasing, continue monitoring. If it keeps increasing, check destination write performance, logs, and whether the publication includes more tables or write volume than expected.
Handling Extended slot status
When slot status is Extended: treat it as an early warning. Confirm the pipeline is connected, check logs for retries or destination slowness, and reduce avoidable write bursts if possible until the pipeline catches up.
Handling Unreserved slot status
When slot status is Unreserved: act soon as the slot is at risk of losing required WAL. Check whether the pipeline is connected and making progress, fix destination or pipeline errors, and contact support if the lag continues to grow.
Handling Lost slot status
When slot status is Lost: the pipeline cannot continue from the existing slot because required WAL has been removed. Recreate the pipeline, or set 'Invalidated slot behavior' to 'Recreate' in the pipeline's advanced settings and restart the pipeline. This creates a new slot and starts replication from scratch for all tables.
Handling Unknown slot status
When slot status is Unknown: check replication logs for errors or missing slot details. If the status remains unknown while the pipeline should be running, contact support with the pipeline ID and recent log details.
Strategies to reduce future replication lag risk
To reduce future lag risk: keep publications focused on the tables and operations you need at the destination. Avoid leaving pipelines stopped for long periods while the source database is still receiving writes. Schedule bulk updates, imports, and migrations during lower-traffic windows when possible. For BigQuery, verify that service account permissions, table requirements, and replica identity settings match the BigQuery destination guide. If the initial sync is the bottleneck, review Table sync workers and Copy connections per table in the pipeline's advanced settings; increasing either can use more source database connections and increasing table sync workers can also use more temporary replication slots.
Table errors in pipeline replication
Table errors occur during the initial sync and affect individual tables. These errors can be retried without stopping the entire pipeline. When a table encounters an error during initial sync, you can reset the table state to restart that table's initial sync from the beginning.
Pipeline-level errors in replication
Pipeline errors can occur during startup or ongoing replication and affect the entire pipeline. If a non-retryable pipeline-level error occurs, the entire pipeline stops and enters a Failed state instead of silently skipping the failure. To recover from a pipeline error, investigate the root cause using error details and logs, fix the underlying issue such as destination connectivity or schema compatibility, then restart the pipeline from the destinations list.
Accessing pipeline monitoring dashboards
To monitor pipelines, navigate to Database > Replication section of the Dashboard to see a list of all destinations with their pipeline status. Click 'View pipeline' on a destination to access the pipeline status page where you can monitor replication performance and table states. Access Logs > Replication section of the Dashboard for detailed error logs and diagnostic information for all pipelines.
Pipeline initial sync process
A pipeline first performs an initial sync of existing rows, then uses ongoing replication with change data capture (CDC) to send subsequent database changes to the destination.
Replicate only specific columns from a table
To replicate only a subset of columns from a table, use: `create publication pub_users_subset for table users (id, email, created_at);` This only replicates the id, email, and created_at columns from the users table.
Supabase Pipelines uses Postgres logical replication
Supabase Pipelines is a managed CDC product that uses Postgres logical replication with the open-source Supabase ETL engine to move data from Supabase Postgres to supported destination systems.
Pipelines replication phases: initial sync and ongoing replication
Supabase Pipelines has two replication phases: initial sync (a one-time copy of existing rows in published tables) and ongoing replication or CDC (continuously captures and applies subsequent inserts, updates, deletes, and truncates).
Managed Pipelines run in AWS eu-central-1 Frankfurt
Managed Pipelines run in AWS eu-central-1 (Frankfurt). This region cannot be changed. To reduce network latency and replication lag, choose a destination region as close as possible to Frankfurt.
Pipelines requires Postgres publication and destination
Pipelines requires two main components: a Postgres publication (defines what to replicate) and a destination (where data is sent). Supabase runs the managed pipeline that reads from the publication and writes to the destination.
Create publication for specific tables
To create a publication for specific tables, use: `create publication pub_users_orders for table users, orders;` This publication tracks all changes (INSERT, UPDATE, DELETE, TRUNCATE) for both the users and orders tables.
Create publication for all tables in schema
To create a publication for all tables in the public schema, use: `create publication pub_all_public for tables in schema public;` This tracks changes for all existing and future tables in the public schema.
Create publication for all tables
To create a publication for all tables in database, use: `create publication pub_all_tables for all tables;` This tracks changes for all tables in your database.
FOR ALL TABLES includes Supabase-managed schemas pitfall
FOR ALL TABLES includes tables in Supabase-managed schemas, including the internal etl tables that Pipelines creates. Prefer FOR TABLES IN SCHEMA public or list the application tables explicitly unless you intend to replicate every eligible table in the database.
Filter rows with WHERE clause in publication
To filter which rows to replicate using a WHERE clause, use: `create publication pub_active_users for table users where (status = 'active');` or `create publication pub_recent_orders for table orders where (created_at > '2024-01-01');`
publish_via_partition_root publication setting for partitioned tables
The publish_via_partition_root publication setting controls whether changes from partitions are emitted as the partition root or as the leaf partitions. When publish_via_partition_root = true, rows from the published partition root are replicated, resulting in one table matching the published partition root at the destination. When publish_via_partition_root = false, rows from leaf partitions are replicated, resulting in one table per replicated leaf partition. If not set in SQL, Postgres defaults to false.
Publishing individual leaf partition ignores publish_via_partition_root
When publishing an individual leaf partition, only that leaf partition gets replicated, regardless of the publish_via_partition_root setting.
Create publication for partition root
To replicate the whole partition hierarchy as the parent table, use: `create publication pub_orders_root for table orders with (publish_via_partition_root = true);` This replicates all rows from the published partition root, including rows stored in leaf partitions, as a single table.
Create publication for leaf partitions
To replicate each leaf partition as its own table, use: `create publication pub_orders_leaves for table orders with (publish_via_partition_root = false);` This replicates each leaf partition as its own destination table.
Use publish_via_partition_root true for single destination table
Use publish_via_partition_root = true when you want analytics queries to read from a single destination table that has the parent table's schema.
Use publish_via_partition_root false for separate partition tables
Use publish_via_partition_root = false when each partition should remain a separate destination table.
Dashboard publications use publish_via_partition_root true
Publications created from the Dashboard replication flow use publish_via_partition_root = true. If you create or alter a publication manually with SQL, set this option explicitly so the destination shape matches what you expect.
Row filters on partition publications in Postgres 15+
On Postgres 15 and newer, row filters on partition publications apply during both the initial sync and ongoing replication. Pipelines uses the row filter attached to the effective publication table entry: the published partition root when publish_via_partition_root = true, and the published leaf relation when publish_via_partition_root = false.
publish_via_partition_root does not copy physical partitioning to destination
The publish_via_partition_root setting controls which Postgres relation becomes a destination table. It does not copy the source table's physical partitioning configuration, partition key, or partition bounds to BigQuery.
Truncating leaf partition with publish_via_partition_root true
With publish_via_partition_root = true, truncating an individual leaf partition is not replicated as a truncate event for the published parent. This is useful for append-only data such as events: you can truncate old leaf partitions to keep Postgres storage bounded while retaining the rows already copied to the destination. If you want the destination to be truncated too, run TRUNCATE on the published partition root.
Enable Pipelines for project before creating pipeline
Before creating a managed replication pipeline, enable Pipelines for your project by navigating to Database > Replication in the Dashboard, clicking Add destination, selecting a Pipelines destination such as BigQuery, and clicking Enable Pipelines.
Pipeline destination configuration details
When configuring a destination in Supabase Pipelines, you must provide: Destination name (a name to identify this destination), Publication (select an existing publication or create a new one by choosing a name and at least one table), and Region (select a nearby dataset, warehouse, or storage bucket region in your destination provider, as managed Pipelines run in fixed AWS eu-central-1 Frankfurt).
Pipeline advanced settings configuration table
Pipeline advanced settings are: Batch wait time (default 10000 milliseconds, allowed whole milliseconds 0 or greater, maximum time after first buffered initial-sync row or ongoing change before pipeline flushes a partially filled batch), Table sync workers (default 4 workers, allowed whole number greater than 0, maximum number of tables synced in parallel during initial sync, each active table sync uses one additional replication slot up to N+1 slots including pipeline's main slot), Copy connections per table (default 4 connections, allowed whole number greater than 0, maximum source database connections used to copy one table in parallel), and Invalidated slot behavior (default Error, allowed Error or Recreate, what happens when main replication slot can no longer continue from retained WAL).
Invalidated slot behavior Error setting
When Invalidated slot behavior is set to Error, the pipeline blocks startup for manual recovery if the main replication slot can no longer continue from retained WAL.
Invalidated slot behavior Recreate setting
When Invalidated slot behavior is set to Recreate, if the pipeline starts after Postgres has invalidated the main replication slot, the pipeline resets its saved table-sync state, rebuilds the slot on the next start, and runs the initial sync again for every replicated table.
Invalidated slot behavior Recreate caution
Use Invalidated slot behavior Recreate carefully. If Recreate is selected and the pipeline starts after Postgres has invalidated the main replication slot, the pipeline performs a destructive restart by resetting its saved table-sync state, creating a new slot, and replacing each destination table through a new initial sync. This destructive restart is required for consistency because the old slot can no longer provide every change the pipeline missed, and the data processed during the new initial sync is billed again.
Pipeline actions: start, stop, restart, edit, delete
Available pipeline actions from the destinations list actions menu are: Start pipeline (begin replication for a stopped pipeline), Update available (review and apply latest managed pipeline version when available), Stop pipeline (request graceful stop, pipeline can remain stopping for up to five minutes while in-flight work finishes, new changes queue in WAL, configured pipeline-hour billing continues while stopped), Restart pipeline (stop and start the pipeline, required after publication changes), Edit destination (modify destination settings like credentials or advanced options), and Delete destination (remove the destination and permanently stop replication).
Add multiple tables to existing publication
To add multiple tables at once to an existing publication, use: `alter publication pub_users_orders add table products, categories;`
Remove single table from publication
To remove a single table from a publication, use: `alter publication pub_users_orders drop table orders;`
Remove multiple tables from publication
To remove multiple tables at once from a publication, use: `alter publication pub_users_orders drop table orders, products;`
Restart pipeline after removing tables from publication
After removing tables from a publication using ALTER PUBLICATION, you must restart the replication pipeline using the actions menu for the changes to take effect.
FOR ALL TABLES or FOR TABLES IN SCHEMA auto-includes new tables
If your Postgres publication uses FOR ALL TABLES or FOR TABLES IN SCHEMA, new tables in that scope are automatically included in the publication. However, you still must restart the replication pipeline for the changes to take effect.
Schema change support in Pipelines
Schema change support depends on the destination. BigQuery is currently the only destination with beta schema change support. See BigQuery schema change support documentation for supported and unsupported changes.
How Pipelines works: capture, send, load
Once configured, a replication pipeline captures changes from your Postgres database using Postgres publications and logical replication, sends the changes through the managed pipeline, and loads the data to your destination. Pipelines automatically optimizes how changes are delivered to the destination. It maps published source columns and values to destination-compatible names and types, but doesn't provide user-defined transformations.
Pipelines limitation: primary keys are destination-specific
Primary key requirements are destination-specific. Postgres can publish inserts for a table without a primary key, but BigQuery Pipelines always requires a source primary key and requires the publication to include its columns.
Pipelines limitation: custom data types replicate as strings
Custom data types in Pipelines replicate as strings. Check that your destination can interpret those string values correctly.
Pipelines limitation: generated columns are skipped
Generated columns are skipped in Pipelines replication. Use triggers to store derived values in regular columns if you need them in the destination.
Pipelines limitation: replica identity requirements are destination-specific
Replica identity requirements are destination-specific. Updates and deletes need enough row identity to apply safely. See BigQuery source table requirements documentation for the supported modes.
Pipelines limitation: no user-defined transformations
Pipelines performs destination-compatible type and name mapping, but doesn't run custom transformations.
Pipelines limitation: at-least-once processing
Pipelines uses at-least-once processing. Failed destination write attempts that Pipelines retries are not counted. In rare cases, Pipelines can count an acknowledged batch but crash or be interrupted before its replication checkpoint is persisted. Recovery can then process and count that batch again. BigQuery uses primary-key-based CDC to converge on the current table state.
Disable Pipelines for project
To turn off Pipelines for a project, delete all Pipelines destinations first. After all destinations are removed, open the three-dot actions menu on the Replication page and click Disable Pipelines.
Logical replication prerequisites for Supabase
To set up manual logical replication, an instance size of XL or greater is recommended. The IPv4 add-on should be enabled. To create a replication slot, you need to use the postgres user and follow the logical replication example instructions. In Postgres 17 or higher, you can create a new user and grant replication permissions with the postgres user, but for versions below 17, you must use the postgres user.