Skip to main content

How to troubleshoot SDK request failures

View Markdown

This guide covers gRPC failures and elevated latency on requests that Temporal SDK Workers and Clients make to the Temporal Service. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

Each section names the metric to watch, explains the impact, and walks through triage. For recommended alert thresholds, see SDK Worker alerting. For metric definitions, see the Temporal SDK metrics reference.

temporal_request_failure increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. temporal_long_request_failure covers poll operations and long-poll GetWorkflowExecutionHistory. Both carry namespace, operation, status_code, and task_queue labels. Severity depends on which status code appears on which operation.

info

UpdateWithStartWorkflowExecution appears in SDK metrics under the gRPC operation name ExecuteMultiOperation.

NOT_FOUND on respond operations​

Metric: temporal_request_failure with status_code=NOT_FOUND on RespondWorkflowTaskCompleted, RespondWorkflowTaskFailed, RespondActivityTaskCompleted, or RespondActivityTaskFailed

A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists. There are three causes:

  • The task timed out. The Worker ran past the Workflow Task timeout, or past the Activity startToClose or scheduleToClose timeout, and the Service discarded the in-flight task.
  • The Workflow Execution is no longer running. It completed, was terminated, or hit its Workflow Run Timeout before the task finished.
  • The Worker restarted mid-execution. The in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up.

Why it matters​

The result the Worker just produced was discarded. For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. For Workflow Tasks, the Service writes a WorkflowTaskTimedOut event to Event History and reschedules the task on the normal Task Queue, which forces a Sticky Execution cache eviction and a cold replay on the retry.

Repeated NOT_FOUND on respond operations means Workers are consistently finishing too late. Every discarded result is Worker capacity spent on work that was thrown away, and every rescheduled task adds to Workflow end-to-end latency.

If you run Local Activities, a Workflow Task timeout causes them to re-execute from scratch on the retried task. Local Activity results are not written to Event History between Workflow Task heartbeats, so re-execution means running them again. If they are not idempotent, this produces duplicate side effects with real business impact.

Triage​

  1. Check task execution latency. For Workflow Tasks, check temporal_workflow_task_execution_latency. For Activities, check temporal_activity_execution_latency for the affected activity_type. If p99 is at or above the corresponding timeout, that is the direct cause.
  2. Check replay latency. If Workflow Task execution latency is elevated, check temporal_workflow_task_replay_latency next. High replay latency means the Worker is spending its time re-executing Event History rather than running new commands — check for large histories and slow Data Converter execution during replay.
  3. Check Worker resources. High CPU on the Worker slows task execution directly. Look at the identity field in the WorkflowTaskStarted or ActivityTaskStarted event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays.
  4. Confirm the Workflow Execution is still running. Check the Execution status in the Temporal UI or with temporal workflow describe. If it completed, was terminated, or hit its Run Timeout, the NOT_FOUND is expected and the Execution lifecycle is what to investigate.
  5. Check for Worker restarts. Frequent restarts lose in-flight Task Tokens. Check pod restart counts in your infrastructure observability stack.
  6. Check for throttling on respond operations. See RESOURCE_EXHAUSTED on respond operations below. Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands.
Self-hosted Temporal Service

If SDK-side metrics look normal and the Execution was not terminated or timed out, check server-side latency: Frontend Service latency filtered to the affected respond operation, and persistence latency filtered to UpdateWorkflowExecution.

NOT_FOUND on Activity heartbeat​

Metric: temporal_request_failure with status_code=NOT_FOUND on RecordActivityTaskHeartbeat

A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists. The Service has already cancelled the in-flight Activity Task: either the heartbeatTimeout fired before the next heartbeat call arrived, the startToClose timeout expired while the Activity was still executing, or the Workflow Execution is no longer running.

Normal Workflow-side cancellation is not a cause. Cancellation returns CancelRequested=true in the heartbeat response body rather than a gRPC error, so NOT_FOUND on this operation is a reliable signal of a timeout or forced closure.

Why it matters​

If heartbeatTimeout is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it. The Activity re-executes from scratch on the next attempt, so a non-idempotent Activity produces duplicate side effects.

Repeated NOT_FOUND on heartbeat calls means the Worker is consistently failing to heartbeat within the configured interval. The Activity will keep timing out on every attempt until the cause is fixed, holding Task slots and generating retry tasks the whole time.

Triage​

  1. Compare the heartbeat interval against heartbeatTimeout. The Worker must call heartbeat more frequently than the timeout. If the Activity slows down between heartbeat calls because of CPU pressure, blocking I/O, or downstream throttling, the effective interval grows past the timeout even though the code is calling heartbeat.
  2. Check Worker CPU. A CPU-starved Worker slows down between heartbeat calls even when the Activity is making progress. If utilization is consistently high, reduce per-Worker concurrency or scale out horizontally.
  3. Check for throttling on heartbeat calls. If the Temporal Service is throttling RecordActivityTaskHeartbeat, the effective heartbeat interval grows past heartbeatTimeout even when the Worker calls on time.
  4. Check the startToClose timeout. If the Activity has run longer than startToClose, the Service times it out while the Activity is still executing, and the next heartbeat returns NOT_FOUND. Compare temporal_activity_execution_latency for the affected activity_type against the configured timeout.
  5. Check heartbeat payload size. The last heartbeat details payload is held in memory for the life of the Activity attempt. Large payloads on high-throughput Activity Workers contribute to memory pressure on the Temporal Service. Store only the minimum progress state needed to resume on retry.

RESOURCE_EXHAUSTED on user-facing operations​

Metric: temporal_request_failure with status_code=RESOURCE_EXHAUSTED on StartWorkflowExecution, SignalWithStartWorkflowExecution, SignalWorkflowExecution, UpdateWorkflowExecution, or ExecuteMultiOperation

The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates. The SDK retries these automatically for up to 60 seconds. Beyond that, the call fails and the error propagates to your caller.

Why it matters​

These operations are on your application's critical path. Within the retry window, callers experience elevated latency. Past it, calls fail outright and your application must handle the error.

If it does not, starts and Signals are silently dropped. A dropped start means the Workflow never runs. A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely. Log these failures in your application code so you can backfill starts and Signals afterward.

The Temporal Service throttles these operations last. Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread.

Triage​

  1. Identify the throttle cause. The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, or an open circuit breaker are different problems.
  2. Check your traffic against your Namespace limits. On Temporal Cloud, compare current throughput against your Namespace's service limits and open a support request if you need them raised.
  3. Add backoff in your application. If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure.
Self-hosted Temporal Service

Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to CreateWorkflowExecution and UpdateWorkflowExecution — slow persistence is the most common root cause of cascading throttling. If the cause is a rate limit, frontend.namespaceRPS may be set too low for your traffic, but only raise it after confirming persistence is healthy. If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself and needs capacity, not a higher limit.

RESOURCE_EXHAUSTED on respond operations​

Metric: temporal_request_failure with status_code=RESOURCE_EXHAUSTED on RespondWorkflowTaskCompleted, RespondWorkflowTaskFailed, RespondActivityTaskCompleted, or RespondActivityTaskFailed

The Temporal Service is throttling Workers reporting task results. The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands.

Why it matters​

This is a leading indicator of NOT_FOUND on respond operations. If throttling persists long enough, the task times out, the Service writes a WorkflowTaskTimedOut or ActivityTaskTimedOut event, and the task is rescheduled — with the cache eviction, cold replay, and Local Activity re-execution consequences described in that section.

Meanwhile every in-flight task holds its slot, reducing the concurrency available for new work. Sustained throttling here escalates into Worker Task slots exhausted.

Triage​

  1. Identify the throttle cause, as in the section above.
  2. Check whether timeouts have already started. If NOT_FOUND on respond operations is also firing, throttling has already cascaded into task timeouts and Executions are losing work.
  3. Check Task slot availability. See Worker Task slots exhausted — slots are not released until the respond call succeeds, so throttling here drains the slot pool.
Self-hosted Temporal Service

Check persistence latency filtered to UpdateWorkflowExecution — slow persistence on this operation is the most common root cause of throttling on respond operations.

UNIMPLEMENTED or INTERNAL from the Temporal Service​

Metric: temporal_request_failure with status_code=UNIMPLEMENTED or status_code=INTERNAL, on any operation

These two status codes point at the Temporal Service rather than at your application, and they behave differently in the SDK.

UNIMPLEMENTED means the Service does not recognize an operation the Worker called. By the time a Worker reaches steady-state polling it has already called GetSystemInfo and DescribeNamespace successfully, so this is rarely a plain version mismatch on a freshly deployed Worker. Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down.

INTERNAL means the Service encountered an error it could not attribute to the request. Short bursts during Service restarts and rolling deploys are expected — set the for duration long enough that your own deploy process does not page you. The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency.

Triage​

  1. Check SDK and Temporal Service version compatibility. For UNIMPLEMENTED, confirm your SDK version is not calling an API that has been removed or changed in your Service version.
  2. Check whether recent deploys correlate. Both codes commonly appear immediately after a Service upgrade or a Worker deploy. If the timing lines up, consider rolling back while you investigate.
  3. Check whether the errors are Namespace-scoped or cluster-wide. Errors isolated to one Namespace point at Namespace configuration. Cluster-wide errors point at infrastructure.
  4. Check downstream effects. Sustained errors on poll operations cause Workers to back off. Cross-check All pollers disconnected and Task completions dropped to zero.
Self-hosted Temporal Service

Check service panics first — any panic is critical and is almost always the root cause of sustained INTERNAL errors. Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL. For UNIMPLEMENTED, verify every Frontend, History, and Matching pod is running the intended binary — a wrong or corrupted binary on a subset of pods produces UNIMPLEMENTED on valid operations, usually alongside panics.

Request latency high on user-facing operations​

Metric: temporal_request_latency on StartWorkflowExecution, SignalWithStartWorkflowExecution, SignalWorkflowExecution, or ExecuteMultiOperation

p99 latency on the operations your application calls synchronously has risen above your threshold.

Why it matters​

These calls block your application code while they wait on the Temporal Service, so the latency is felt directly by your users and by anything downstream of the call completing.

The SDK retries transient errors but does not hide the latency cost: every retry adds to the total time this metric observes. If throttling is the cause and retries exhaust the 60-second budget, the call fails outright.

Triage​

  1. Check for throttling on the same operations. See RESOURCE_EXHAUSTED on user-facing operations. If both are firing, throttling is the cause of the latency and retries are what you are measuring.
  2. Check payload sizes. This metric includes serialization and network time. Large Workflow inputs or Signal payloads, or an expensive Payload Codec, raise it without any Service-side slowdown.
  3. Check network path and region. Clients in a different region from the Temporal Service pay that round trip on every call.
Self-hosted Temporal Service

Check Frontend Service latency filtered to the affected operations — server-side latency is the more precise signal, since the SDK metric includes serialization and network time. Then check persistence latency filtered to CreateWorkflowExecution and UpdateWorkflowExecution, the usual driver of elevated Frontend latency on starts and Signals. If persistence is healthy and there is no throttling, check Frontend pod CPU.