new·The score now tells you which way it movedA brain's exam only ever grows: its own material writes questions, and so does every question a real caller asked and did not get answered. The score is a percentage over that growing set, so a brain that learned more could post a smaller number — and this week three did. One of them answered two MORE questions than the week before and showed eighteen points less. Printed as a single percentage, that reads as decline to a reader and as punishment to anyone who contributes material.all news →
mozg.beta
Sign in

Temporal · Concepts · all subjects

failures & error handling

14 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Side Effect failure behavior

Side Effects should not fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task.

Workflow Task execution latency high alert

Alert on workflow_task_execution_latency metric with condition task_queue and workflow_type. Threshold is p99 above 10s. for duration is 5m. Default severity is Critical.

Transient errors during Worker deployment

Several alert conditions appear briefly during Worker deployment. A rolling restart drops pollers to zero on each pod, produces NOT_FOUND on respond operations for in-flight Tasks, and can produce short-lived non-determinism errors while two Worker versions overlap. These clear on their own once rollout finishes, so the fix is a for duration longer than the deploy takes, not a quieter threshold.

NOT_FOUND on respond operations alert

Alert on request_failure metric with condition status_code=NOT_FOUND and operation in RespondWorkflowTaskCompleted, RespondWorkflowTaskFailed, RespondActivityTaskCompleted, RespondActivityTaskFailed. Threshold is any occurrence. for duration is 5m. Default severity is Critical.

NOT_FOUND on Activity heartbeat alert

Alert on request_failure metric with condition status_code=NOT_FOUND and operation=RecordActivityTaskHeartbeat. Threshold is any occurrence. for duration is 5m. Default severity is Warning.

RESOURCE_EXHAUSTED on respond operations alert

Alert on request_failure metric with condition status_code=RESOURCE_EXHAUSTED and operation in RespondWorkflowTaskCompleted, RespondWorkflowTaskFailed, RespondActivityTaskCompleted, RespondActivityTaskFailed. Threshold is any occurrence. for duration is 5m. Default severity is Critical.

RESOURCE_EXHAUSTED on poll operations alert

Alert on long_request_failure metric with condition status_code=RESOURCE_EXHAUSTED and operation in PollWorkflowTaskQueue, PollActivityTaskQueue. Threshold is any occurrence. for duration is 5m. Default severity is Warning.

UNIMPLEMENTED from Temporal Service alert

Alert on request_failure metric with condition status_code=UNIMPLEMENTED and any operation. Threshold is any occurrence. for duration is 2m. Default severity is Critical.

INTERNAL from Temporal Service alert

Alert on request_failure metric with condition status_code=INTERNAL and any operation. Threshold is any occurrence. for duration is 2m. Default severity is Critical.

Request latency high on user-facing operations alert

Alert on request_latency metric with condition operation in StartWorkflowExecution, SignalWithStartWorkflowExecution, SignalWorkflowExecution, UpdateWorkflowExecution, ExecuteMultiOperation. Threshold is p99 above 2s. for duration is 5m. Default severity is Critical.

gRPC message too large alert

Alert on workflow_task_execution_failed metric with condition failure_reason=GrpcMessageTooLarge. Threshold is any occurrence. for duration is 1m. Default severity is Critical.

Workflow Task execution failures elevated alert

Alert on workflow_task_execution_failed metric with condition failure_reason=WorkflowError. Threshold is rate above 10/s. for duration is 2m. Default severity is Warning.

Activity execution failures elevated alert

Alert on activity_execution_failed metric with condition activity_type. Threshold is rate above 10/s. for duration is 2m. Default severity is Warning.

Critical alert severity for NOT_FOUND Activity heartbeat requires idempotency

A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start. If the Activity is not idempotent, treat the alert as Critical instead of the default Warning.

Give your agent this brain