nexus_requests metric (Frontend Service)
The nexus_requests metric is a counter representing the number of Nexus requests received by the Frontend Service.
reference
188 notes in this subject, read out of this brain and free to use. This is page 2 of 4.
nexus_completion_requests metric (Frontend Service)
The nexus_completion_requests metric is a counter representing the number of Nexus completion (callback) requests received by the Frontend Service.
nexus_completion_latency metric (Frontend Service)
The nexus_completion_latency metric is a histogram representing the latency of Nexus completion (callback) requests received by the Frontend Service.
nexus_completion_request_preprocess_errors metric (Frontend Service)
The nexus_completion_request_preprocess_errors metric is a counter representing the number of Nexus completion requests for which pre-processing failed in the Frontend Service.
Monitoring infrastructure metrics alongside Temporal Service metrics
Apart from metrics emitted by the Temporal Service, you should also monitor infrastructure-specific metrics like CPU, memory, and network for all hosts that are running Temporal Service services.
Grafana dashboards for Temporal metrics
The dashboards repository at https://github.com/temporalio/dashboards contains community-driven Grafana dashboard templates that can be used as a starting point for monitoring the Temporal Service and SDK metrics. These templates can be used as references to build custom dashboards. For metrics missing in the dashboards, use metric_defs.go as a reference.
schedule_rate_limited metric
The schedule_rate_limited metric reflects instances where the creation of Workflows by a Schedule is throttled due to rate limiting policies within a Namespace. This metric is crucial for identifying scheduling patterns that frequently hit rate limits, potentially causing missed catchup windows. Example Prometheus query: sum(rate(schedule_rate_limited{namespace="$namespace"}[5m]))
schedule_action_success metric
The schedule_action_success metric measures the successful execution of Workflows as per their schedules or through manual triggers. This metric confirms that Workflows are running as expected without delays or errors. Example Prometheus query: sum(rate(schedule_action_success{namespace="$namespace"}[5m]))
workflow_cancel metric
The workflow_cancel metric shows the number of Workflows canceled before completing execution.
workflow_continued_as_new metric
The workflow_continued_as_new metric shows the number of Workflow Executions that were Continued-As-New from a past execution.
workflow_failed metric
The workflow_failed metric shows the number of Workflows that failed before completion.
workflow_success metric
The workflow_success metric shows the number of Workflows that successfully completed.
workflow_timeout metric
The workflow_timeout metric shows the number of Workflows that timed out before completing execution.
dynamic_worker_pool_scheduler_dequeued_tasks metric (Nexus)
The dynamic_worker_pool_scheduler_dequeued_tasks metric is a counter that is incremented when a task is dequeued from the in-memory buffer in the Nexus machinery of the History Service.
ApplicationFailure SDK implementations
ApplicationFailure implementations across SDKs: TypeScript ApplicationFailure, Java ApplicationFailure, Go ApplicationError, Python ApplicationError, PHP ApplicationFailure.
TemporalFailure base class SDK implementations
Most SDKs have a base Failure class that other Failures extend: TypeScript TemporalFailure, Java TemporalFailure, Python FailureError, PHP TemporalFailure.
Failure proto message base fields
The base Failure proto message has these fields: string message, string stack_trace, string source (the SDK this Failure originated in, e.g., 'TypeScriptSDK'), Failure cause (the Failure message of the cause if applicable), Payload encoded_attributes (contains encoded message and stack_trace fields when using a Failure Converter).
CancelledFailure SDK implementations
CancelledFailure implementations across SDKs: TypeScript CancelledFailure, Java CanceledFailure, Go CanceledError, Python CancelledError, PHP CanceledFailure.
TypeScript isCancellation helper for checking cancellations
TypeScript has an isCancellation helper function to check both cases where a Cancelled Failure is thrown directly by a Workflow API function and where it is wrapped in a different Failure.
ActivityFailure SDK implementations
ActivityFailure implementations across SDKs: TypeScript ActivityFailure, Java ActivityFailure, Go ActivityError, Python ActivityError, PHP ActivityFailure.
Nexus Operation Failure SDK implementations
Nexus Operation Failure implementations: Go NexusOperationError, Proto NexusOperationFailureInfo.
ChildWorkflowFailure SDK implementations
ChildWorkflowFailure implementations across SDKs: TypeScript ChildWorkflowFailure, Java ChildWorkflowFailure, Go ChildWorkflowExecutionError, Python ChildWorkflowError, PHP ChildWorkflowFailure.
TimeoutFailure SDK implementations
TimeoutFailure implementations across SDKs: TypeScript TimeoutFailure, Java TimeoutFailure, Go TimeoutError, Python TimeoutError, PHP TimeoutFailure.
TerminatedFailure SDK implementations
TerminatedFailure implementations across SDKs: TypeScript TerminatedFailure, Java TerminatedFailure, Go TerminatedError, Python TerminatedError, PHP TerminatedFailure.
ServerFailure SDK implementations
ServerFailure implementations across SDKs: TypeScript ServerFailure, Java ServerFailure, Go ServerError, Python ServerError, PHP ServerFailure.
Available reference resources
The Temporal Platform provides references for: SDK Metrics, Commands, Events, Web UI environment variables, Temporal Service and Web UI configurations, and API guides for Go, Java, Python, TypeScript, .NET, and PHP.
Reference documentation hub structure
Temporal documentation includes the following reference sections: API reference, SDK metrics reference, Commands reference, Events reference, Web UI environment variables reference, Temporal Service configuration reference, Temporal Web UI configuration reference, Temporal Cloud Operation reference, and Glossary.
cors configuration for Web UI
The cors configuration manages Cross-Origin Resource Sharing. It includes the following fields: cookieInsecure (boolean, defaults to false) and allowOrigins (array of authorized domain URLs that can access the UI Server APIs).
publicPath configuration for Web UI Server
The publicPath configuration specifies the path used by the Temporal Web UI Server and its APIs. It defaults to an empty string.
port configuration for Web UI Server
The port configuration specifies the port used by the Temporal Web UI Server and its APIs. It defaults to 8080.
Temporal Web UI configuration file structure
The Temporal Web UI Server uses a YAML configuration file for UI settings. An example development.yaml file is available in the temporalio/ui-server repository. Multiple configuration files can be created for configuring specific areas of the UI, such as Auth or TLS.
batchActionsDisabled configuration
The batchActionsDisabled configuration controls whether batch actions can be executed. It is a boolean field that defaults to false.
cloudUi configuration
The cloudUi configuration enables the Cloud UI. It is a boolean field that defaults to false.
codec configuration for Web UI
Codec Server configuration includes the following fields: endpoint (URL to the codec server), passAccessToken (boolean, defaults to false, determines whether to pass the access token to the codec server), includeCredentials (boolean, defaults to false), and decodeEventHistoryDownload (boolean, defaults to false).
defaultNamespace Web UI configuration
The defaultNamespace configuration specifies which Namespace the UI loads data for by default. It defaults to 'default'.
disableWriteActions configuration for Workflow Actions
The disableWriteActions configuration prevents users from executing Workflow Actions on the Web UI. This affects both Bulk Actions for Recent Workflows and Workflow Actions on the Workflow Details page. When set to true, it overrides individual Workflow Action configuration values and disables all Workflow Actions on the Web UI. It defaults to false.
enableUi configuration
The enableUi configuration enables the browser UI and defaults to true. This configuration can be set dynamically with the TEMPORAL_UI_ENABLED environment variable. If disabled (set to false), the UI server APIs remain available but the UI itself will not be accessible.
feedbackUrl configuration
The feedbackUrl configuration specifies the URL to direct users to when they click the Feedback button in the UI. If not specified, it defaults to the UI's GitHub Issue page at https://github.com/temporalio/ui/issues/new/choose.
forwardHeaders configuration
The forwardHeaders configuration allows you to configure headers for forwarding.
hideLogs configuration
The hideLogs configuration disables server logs from being printed to the console when enabled. It defaults to false.
hideWorkflowQueryErrors configuration
The hideWorkflowQueryErrors configuration hides errors resulting from a Query to the Workflow when enabled. It defaults to false.
notifyOnNewVersion configuration
The notifyOnNewVersion configuration, when set to true, displays a notification in the UI when a newer version of the Temporal Server is available. It defaults to false.
refreshInterval configuration
The refreshInterval configuration specifies how often the UI Server reads the configuration file for new values. Currently, only TLS configuration values are propagated during a refresh. It defaults to 1m (one minute).
showTemporalSystemNamespace configuration
The showTemporalSystemNamespace configuration, when set to true, makes the Temporal System Namespace visible in the UI. The Temporal System Namespace lists Workflow Executions used by the Temporal Platform. It defaults to false.
temporalGrpcAddress configuration
The temporalGrpcAddress configuration specifies the frontend address for the Temporal Cluster. The default address is localhost at 127.0.0.1:7233.
workflowCancelDisabled configuration
The workflowCancelDisabled configuration prevents users from canceling Workflow Executions from the Web UI when enabled. It defaults to false.
workflowResetDisabled configuration
The workflowResetDisabled configuration prevents users from resetting Workflows from the Web UI when enabled. It defaults to false.
workflowSignalDisabled configuration
The workflowSignalDisabled configuration prevents users from signaling Workflow Executions from the Web UI when enabled. It defaults to false.
workflowTerminateDisabled configuration
The workflowTerminateDisabled configuration prevents users from terminating Workflow Executions from the Web UI when enabled. It defaults to false.
temporal_worker_task_slots_used metric
The total number of Workflow, Activity, Local Activity, or Nexus Tasks execution slots in current use. Use the worker_type tag to differentiate execution slots. Type: Gauge. Available in: Core, Go, Java. Tags: namespace, task_queue, worker_type.
temporal_workflow_active_thread_count metric
Total amount of Workflow threads in the Worker Process. Type: Gauge. Available in: Java.
temporal_workflow_canceled metric
Workflow Execution ended because of a cancellation request. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type.
temporal_workflow_completed metric
A Workflow Execution completed successfully. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type.
temporal_workflow_continue_as_new metric
A Workflow ended with Continue-As-New. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type.
temporal_workflow_endtoend_latency metric
Total Workflow Execution time from schedule to completion for a single Workflow Run. A retried Workflow Execution is a separate Run. Type: Histogram. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type.
temporal_workflow_failed metric
A Workflow Execution failed. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type.
temporal_workflow_task_execution_failed metric
A Workflow Task Execution failed. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type, failure_reason. Failure reason values: NonDeterminismError (Workflow Task failed due to non-determinism error), GrpcMessageTooLarge (response exceeded gRPC message size limit), WorkflowError (failed for any other reason).
temporal_workflow_task_execution_latency metric
Workflow Task Execution time. Type: Histogram. Available in: Core, Go, Java. Tags: namespace, task_queue, workflow_type.
temporal_workflow_task_queue_poll_empty metric
A Workflow Worker polled a Task Queue and timed out without picking up a Workflow Task. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue.
temporal_workflow_task_queue_poll_succeed metric
A Workflow Worker polled a Task Queue and successfully picked up a Workflow Task. Type: Counter. Available in: Core, Go, Java. Tags: namespace, task_queue.