observability

243 notes in this subject, read out of this brain and free to use. This is page 3 of 5.

Tail Worker use cases

Tail Workers can process logs for alerts, debugging, or analytics. You can filter, change the format of the data, and send events to any HTTP endpoint. For quick debugging, Tail Workers can be used to send logs to KV or any database.

Tail handler basic example

The following code shows a Tail Worker that sends events data to an HTTP endpoint: export default { async tail(events) { fetch("https://example.com/endpoint", { method: "POST", body: JSON.stringify(events), }); }, };

Tail events object structure

The events object passed to the tail() handler is an array containing event objects. Each event object has the following fields: scriptName (string), outcome (string), eventTimestamp (number), event (object containing request details with url, method, headers, and cf object), logs (array of log objects with message, level, and timestamp), exceptions (array of exception objects with name, message, and timestamp), and diagnosticsChannelEvents (array of objects with channel, message, and timestamp).

Tail Worker vs OpenTelemetry export

If exporting logs and errors to observability tools like Sentry, Grafana, or Honeycomb, you may not need to use Tail Workers. Instead, you can configure a Worker to export OpenTelemetry (OTEL) format logs and traces to these tools. With OTEL, logs and traces are sent in batches to your destination rather than sent after each invocation. Tail Workers should be considered the advanced-mode option for custom requirements not built into the Workers observability platform.

Tail Worker configuration steps

To configure a Tail Worker: 1. Create a Worker to serve as the Tail Worker. 2. Add a tail() handler to your Worker. The tail() handler is invoked every time the producer Worker to which a Tail Worker is connected is invoked. 3. Add the following to the Wrangler file of the producer Worker: {"tail_consumers": [{"service": "<TAIL_WORKER_NAME>"}]}. Workers added to the tail_consumers array must have a tail() handler defined.

MCP server enables AI agent integration with observability

The Cloudflare MCP server allows AI agents to access and interact with Workers observability data, enabling automated analysis and insights.

MCP server for Workers observability

Cloudflare provides an MCP (Model Context Protocol) server that connects AI agents to Workers observability data. The MCP server is available at https://github.com/cloudflare/mcp-server-cloudflare/tree/main/apps/workers-observability.

CPU time per execution metric

The CPU Time per execution chart shows historical CPU time data broken down into relevant quantiles using reservoir sampling. In some cases, higher quantiles may appear to exceed CPU time limits without generating invocation errors because of a mechanism in the Workers runtime that allows rollover CPU time for requests below the CPU limit.

Duration per request (GB-seconds) metric

The Duration per request chart shows historical duration per Worker invocation broken down into quantiles. Understanding duration on a Worker is especially useful when performing significant computation on the Worker itself.

Workers metrics overview and access

Workers metrics aggregate request data for an individual Worker and help diagnose issues and understand workloads by showing performance and usage. If a Worker runs across multiple domains and on *.workers.dev, metrics aggregate requests across all of them. To view metrics, go to the Cloudflare dashboard Workers & Pages page and select your Worker in Overview.

Zone analytics overview

Zone analytics aggregate request data for all Workers assigned to any routes defined for a zone. Zone data can be scoped by time range within the last 30 days. Access zone analytics from the Cloudflare dashboard Workers Analytics page for your zone.

Total requests metric definition

Total requests is the count of all incoming requests registered by a Worker. Requests blocked by WAF or other security features will not count toward this metric.

Success requests metric definition

Success requests are counted when a request returned a Success or Client Disconnected invocation status.

Error requests metric definition

Error requests are counted when a request returned a Script Threw Exception, Exceeded Resources, or Internal Error invocation status.

Subrequests metric definition

Subrequests are requests triggered by calling fetch from within a Worker. A subrequest that throws an uncaught error will not be counted. The subrequests metric shows total subrequests, cached responses, and uncached responses.

Wall time per execution definition

Wall time represents the elapsed time in milliseconds between the start of a Worker invocation and when the Workers runtime determines no more JavaScript needs to run. It includes time spent waiting on I/O and time spent executing in a Worker's waitUntil() handler. Wall time is measured from when the JavaScript context remains open, not from when the final response byte is sent to the client. The metric uses reservoir sampling to show historical data broken down into quantiles.

Memory usage metric and limits

The Memory usage chart shows how much V8 isolate memory a Worker uses at the time of each invocation, broken down into P50, P90, P99, and P999 percentiles. Workers run in V8 isolates with a 128 MB memory limit. A single isolate can handle many concurrent requests and shares memory across them. Memory usage metric reflects how much of this shared memory is in use at each invocation. Deployment markers on the chart let you correlate memory changes with code deployments. Memory trending upward over time may indicate a memory leak.

Invocation status table

Invocation statuses indicate whether a Worker executed successfully or failed. The statuses are: Success (Worker executed successfully, no error code), Client disconnected (HTTP client disconnected before request completed, no error code), Worker threw exception (Worker threw an unhandled JavaScript exception, error code 1101), Exceeded resources (Worker exceeded runtime limits, error codes 1102 or 1027), Internal error (Workers runtime encountered an error, no error code). Invocation statuses differ from HTTP status codes.

Exceeded Resources invocation status causes

The Exceeded Resources invocation status appears when a Worker exceeds runtime limits. The most common cause is excessive CPU time, but it can also be caused by a Worker exceeding startup time or free tier limits.

Internal Error invocation status details

The Internal Error status appears when the Workers runtime fails to process a request due to an internal failure in the system. These errors are not caused by any issue with Worker code or resource limits. While rare, some Internal Error requests may appear during normal operation. Requests with Internal Error status are not counted towards usage for billing purposes. If you notice an elevated rate of Internal Error requests, review www.cloudflarestatus.com.

Request duration metric and Smart Placement

The request duration chart shows how long it took a Worker to respond to requests, including code execution and time spent waiting on I/O. This chart is currently only available when a Worker has Smart Placement enabled. Request duration measures from when a request comes into a data center until a response is delivered, in contrast to execution duration which measures only the time a Worker is active. The chart shows a histogram comparing duration for requests with Smart Placement enabled versus disabled (by default, 1% of requests are routed with Smart Placement disabled).

Metrics retention period

Worker metrics can be inspected for up to three months in the past in maximum increments of one week.

Request traffic data aggregation delay

Request traffic data may display a drop off near the last few minutes displayed in the graph for time ranges less than six hours. This does not reflect a drop in traffic but a slight delay in aggregation and metrics delivery.

Zone analytics subrequests breakdown

The subrequests chart in zone analytics shows requests triggered by calling fetch from within a Worker, broken down by cache status into Uncached (requests answered directly by origin server or other servers responding to subrequests) and Cached (requests answered by Cloudflare's cache).

Zone analytics bandwidth metric

The bandwidth chart in zone analytics shows historical bandwidth usage for all Workers on a zone broken down by cache status.

Zone analytics status codes metric

The status codes chart in zone analytics shows historical requests for all Workers on a zone broken down by HTTP status code.

Zone analytics total requests metric

The total requests chart in zone analytics shows historical data for all Workers on a zone broken down by successful requests, failed requests, and subrequests. Request types are categorized by HTTP status code where 200-level requests are successful and 400 to 500-level requests are failed.

Worker metrics powered by GraphQL

Worker metrics are powered by GraphQL. You can learn more about querying data sets in the Querying Workers Metrics with GraphQL tutorial.

Analytics Engine for custom analytics

For custom, application-specific analytics beyond standard Worker metrics, use Workers Analytics Engine. It is useful for custom business metrics (track events like signups or purchases), per-customer analytics (record data with high-cardinality dimensions like customer IDs), usage-based billing (count API calls or billable events per customer), and performance tracking (measure response times or error rates with custom dimensions). Writes to Analytics Engine are non-blocking and do not add latency to a Worker. Query data using SQL through the Analytics Engine SQL API or visualize in Grafana.

Source map retrieval does not impact Worker performance

When an uncaught exception occurs, Cloudflare fetches the source map after the Worker invocation completes. This is an asynchronous process that does not impact the Worker's CPU utilization or performance. Source maps are not accessible inside the Worker at runtime—if you `console.log()` the stack property within a Worker, you will not get a deobfuscated stack trace.

View stack traces with source maps in real-time logs and Tail Workers

When your Worker throws an uncaught exception, the source map is used to map the stack trace back to lines of your Worker's original source code. You can then view the deobfuscated stack trace when streaming real-time logs or in Tail Workers.

Stack trace remapping behavior with source maps

When Cloudflare attempts to remap a stack trace to the Worker's source map, it does so line-by-line, remapping as much as possible. If a line of the stack trace cannot be remapped for any reason, Cloudflare leaves that line unchanged and continues to the next line.

Query example: debugging 5xx errors by path

To find and debug all paths that respond with 5xx errors: create a base query visualizing by raw event count, add a filter for $workers.event.response.status greater than 500, then group by $workers.event.request.path and $workers.event.response.status to identify the number of requests affected. This shows which paths have 4xx and 5xx errors. After identifying problematic paths, apply an additional filter for that specific path and investigate further using the Invocations tab to see logged invocations of the error, then expand individual invocations to view relevant logs.

Query Builder Group By feature

Group By combines rows that have the same value into summary rows. For example, adding $workers.event.request.cf.country as a Group By field will group results by country.

Query Builder Order By feature

Order By affects how results are sorted in the summary table. If asc is selected, results are sorted in ascending order from least to greatest. If desc is selected, results are sorted in descending order from greatest to least.

Query Builder Limit feature

Limit restricts the number of results returned. When paired with Order By, it can be used to return the 'top' or 'first' N results.

Query Builder time range selection

When selecting a time range, you specify the time interval where you want to look for matching events. The retention period depends on your plan type.

Query Builder result view tabs

There are three views for query results: Visualizations tab (shows graphs and a summary table for the query), Invocations tab (shows all logs grouped by invocation and ordered by timestamp, with only invocations matching query criteria returned), and Events tab (shows all logs ordered by timestamp, with only events matching query criteria returned, and can be customized to add additional fields).

Saving queries in Query Builder

Queries can be saved with a name, description, and custom tags by selecting Save Query. Saved queries are stored at the account-level and are accessible to all users in the account. Saved queries can be re-run by selecting them from the Queries tab and can be edited and saved again. Individual users can star queries, and starred queries are unique to the user, not the account.

Deleting saved queries

Saved queries can be deleted from the Queries tab in the Observability page. To delete: go to Observability page, select Queries tab, select the three dots on the right-hand side for additional actions, select Delete Query, and follow the instructions. Deleted queries are removed for all users in the account.

Sharing saved queries

Saved queries are assigned a unique URL and can be shared with any user in the account.

Workers Logs Wrangler configuration

To enable Workers Logs in Wrangler, use this configuration: { "observability": { "enabled": true, "logs": { "invocation_logs": true, "head_sampling_rate": 1 } } }. The head_sampling_rate is optional with a default value of 1.

Query Builder availability and enablement

The Query Builder is available to all developers and requires no enablement. Queries search all Workers Logs stored by Cloudflare. To enable Workers Logs, add the observability configuration to the Wrangler file with observability.enabled set to true and observability.logs.invocation_logs set to true, then redeploy the Worker.

Query Builder visualization functions

The Query Builder supports the following visualization functions: Count (returns total number of rows matching query conditions), Count Distinct (number of occurrences of unique values, requires any field), Min (smallest value, requires numeric field), Max (largest value, requires numeric field), Sum (total of all values, requires numeric field), Average (average of field, requires numeric field), Standard Deviation (requires numeric field), Variance (requires numeric field), P001 (value below which 0.1% of data falls, requires numeric field), P01 (value below which 1% of data falls, requires numeric field), P05 (value below which 5% of data falls, requires numeric field), P10 (value below which 10% of data falls, requires numeric field), P25 (value below which 25% of data falls, requires numeric field), Median/P50 (value below which 50% of data falls, requires numeric field), P75 (value below which 75% of data falls, requires numeric field), P90 (value below which 90% of data falls, requires numeric field), P95 (value below which 95% of data falls, requires numeric field), P99 (value below which 99% of data falls, requires numeric field), P999 (value below which 99.9% of data falls, requires numeric field). All methods are aggregate functions, with Count being an exception that works without a specific field. Multiple visualizations can be added in a single query, each rendering a graph, and a single summary table shows raw query results.

Query Builder filter operators by data type

Filters in the Query Builder have three components: key (any field in a log event), operator (logical condition), and value. For numeric fields, valid operators are: Equals, Does not equal, Greater, Greater or equals, Less, Less or equals, Exists, Does not exist. For string fields, valid operators are: Equals, Does not equal, Includes, Does not include, Regex, Exists, Does not exist, Starts with. The value for numeric fields is an integer; for string fields, it is any string. Multiple filters are combined with an AND operator, so only events matching all filters are returned.

Query Builder filter example

When filtering with key $workers.cpuTimeMs, operator Greater than, and value 100, only log events where $workers.cpuTimeMs > 100 will be returned.

Query Builder location in Cloudflare dashboard

The Query Builder can be found in the Observability page of the Cloudflare dashboard at the Workers & Pages section.

Query Builder search feature

Search is a text filter that returns only events containing the specified text. It can be used as a quick filtering mechanism or to search for unique identifiable values in logs.

Sentry integration with Cloudflare Workers

You can connect a Sentry project from your Cloudflare Worker to automatically send errors and uncaught exceptions to Sentry for observability and error tracking.

Traces show "Trace in Progress" status while being recorded

While a trace is in progress, the event will show "Trace in Progress" on the root span. Users should wait a few moments for the full trace to become available.

Use $metadata.service for consistent Worker name filtering

Some attributes only apply to certain spans (for example, service.name and faas.name). When filtering or grouping by Worker name across traces and logs, use $metadata.service instead, as it will apply consistently across all event types.

Span and attribute names subject to change during beta

As Workers tracing is currently in beta, span names and attribute names are not yet finalized. These names may be refined during the beta period to improve clarity and align with OpenTelemetry semantic conventions. Users should review the spans and attributes documentation periodically for updates.

Incomplete span attributes in Workers tracing

Workers tracing currently has incomplete span attributes. More detailed attributes are planned to be added on each span. Users should review the spans and attributes documentation periodically for updates, and can provide feedback on missing attributes via the Workers tracing GitHub discussion.

Non-I/O operations report 0 ms duration in traces

Due to security measures to prevent Spectre attacks, the Workers Runtime does not update time until I/O events take place. This means that some spans will return a length of 0 ms even when the operation took longer. The Cloudflare Workers team is exploring security measures that would allow exposing time lengths at millisecond-level granularity in these cases.

Workers tracing is in open beta

Workers tracing is currently in open beta. Users can provide feedback and send feature requests via the Workers tracing GitHub discussion.

Trace context not propagated to external services

When exporting traces to external platforms, trace IDs are not propagated to services outside of Cloudflare. This means traces from Workers will not link with traces from non-Cloudflare services in observability tools. The Cloudflare team is working on automatic trace context propagation using W3C Trace Context standards to enable end-to-end visibility across tools and services.

Example: Nested spans with platform operations

Example showing nested spans with KV and fetch: import { tracing } from 'cloudflare:workers'; async function handleOrder(env: Env, orderId: string) { return tracing.enterSpan('handleOrder', async (span) => { span.setAttribute('order.id', orderId); const order = await env.ORDERS_KV.get(orderId, 'json'); const total = tracing.enterSpan('calculateTotal', (innerSpan) => { innerSpan.setAttribute('item.count', order.items.length); return order.items.reduce( (sum: number, item: any) => sum + item.price, 0, ); }); await fetch('https://api.example.com/notify', { method: 'POST', body: JSON.stringify({ orderId, total }), }); return new Response(JSON.stringify({ orderId, total })); }); } The KV read and fetch are automatically children of 'handleOrder'. The nested 'calculateTotal' span is also a child of 'handleOrder'.

Custom spans enable tracing of application logic

Custom spans extend visibility into application logic alongside Cloudflare's automatic instrumentation of platform operations like fetch calls, KV reads, and D1 queries. They allow you to trace custom code paths by wrapping sections of code in named spans.

Two ways to access custom spans API

The custom spans API is available through: (1) importing tracing from 'cloudflare:workers', which works anywhere in the codebase including utility functions and libraries without handler context access, and (2) ctx.tracing, available on the ExecutionContext passed to the handler. Both provide identical methods and behavior.

Two span creation methods: enterSpan and startActiveSpan

enterSpan() creates a span that automatically ends when the callback returns or its returned promise settles, suitable for most instrumentation. startActiveSpan() creates a span you must end manually by calling span.end(), used when the span must outlive the callback such as when instrumenting streams or other long-lived operations.

Give your agent this brain