:boom: Payload limit enforcement by jmaeagle99 · Pull Request #1363 · temporalio/sdk-rust

jmaeagle99 · 2026-06-30T04:25:56Z

What was changed

Builds on the existing payload-field classification codegen to add end-to-end payload/memo
size-limit enforcement in the client and worker.

Validate every outbound request via generated dispatcher; error-level violation fails with an invalid argument status carrying the PayloadLimitViolation as its source.
Warning thresholds configured on ConnectionOptions as payload_limits PayloadLimitsOptions field. Default is to warn payloads at 512 KiB and memos at 2 KiB, same defaults as server.
Error limits are read by workers when describing the namespace and set on the PayloadLimitsClient decorator that wraps the existing connection decorator. Each request has the error limits (if set) attached as a request extension.
On error limit violation, worker converts requests to their corresponding failures and the operations are retryable:
- WFT completion → RespondWorkflowTaskFailed
- Activity completion / cancel → RespondActivityTaskFailed
- Activity heartbeat → RespondActivityTaskFailed and report activity as cancelled
- Nexus completion → RespondNexusTaskFailed
Logs and task-failure messages render the same TMPRL1103 string.
Richer completion-span context (namespace/workflow_id/type/attempt/worker_id) threaded through for log forwarding.
New FailureReason::PayloadsTooLarge metric reason.

Why?

Prevent server from hard-failing workflows, activities, and operations due to payloads/memo size limit violations. Allows customers to potentially update their workflows, activities, and operations to reduce the oversized payloads/memos.

Checklist

How was this tested: New unit tests and integration tests
Any docs updates needed? Yes

jmaeagle99

Some self-review comments. If you have ideas/opinions, would love to hear them.

jmaeagle99 · 2026-07-01T06:48:26Z

+pub struct PayloadLimitsOptions {
+    /// Warning threshold (bytes) for the size of an outbound payload-bearing field; over-threshold
+    /// fields are logged but still sent to server. Defaults to 512 KiB. Set to `0` to disable.
+    pub payloads_size_warn: u64,


I might flip the naming of these fields to:

payloads_warn_size

memo_warn_size
I think it would allow better grouping of related settings together if we ever need to add more for each severity level e.g. payload_warn_fancy_feature and I think it reads better when saying them because there aren't two S's next to each other like there is in payloads_size_warn.

This suggestion is contrary to what we did for Python and Go. Also note that the payloads field is plural here where it is singular in Python and Go; I intentionally did that since the sizing is applied to payload-bearing fields, which include single payload fields, repeated payload fields (a sequence), and the Payloads proto message.

So options are:

Leave as-is.

Match Python and Go naming for consistency sake.

Rename to payloads_warn_size and memo_warn_size for (IMO) maringally better naming and conceptualization.

Some other better suggestion.

Naming is hard.

Option 3 works for me

jmaeagle99 · 2026-07-01T06:52:28Z

        F: Send + Sync + Unpin + 'static,
    {
+        // Validate payload sizes after any request mutation but before encoding/metrics.
+        validate_request_payload_limits(


This is situated right before the gRPC message is sent so that it is the last thing in the gRPC client chain that could modify payloads and memos. Things I need to check here are:

This needs to be checked before gRPC request metrics are emitted so that we aren't double counting attempts e.g. a WFT completion that is rejected due to payload sizes should not emit a gRPC request metric for that message.

I recall there is a grpc override callback. This probably should occur before that callback is invoked.

Sushisource · 2026-07-02T18:27:27Z

+/// Per-call payload/memo size error limits, attached to a request's extensions by a caller that
+/// wants error-level enforcement on this call.
+#[derive(Debug, Clone, Copy)]
+pub struct PayloadErrorLimits {
+    /// Blob (payload) size error threshold, in bytes.
+    pub blob: usize,
+    /// Memo size error threshold, in bytes.
+    pub memo: usize,
+}


Nit: There is request_extensions.rs that this could go in, but, I'm fine with it here too.

Sushisource · 2026-07-02T18:29:02Z

+pub struct PayloadLimitsClient<C> {
+    inner: C,
+    error_limits: Arc<RwLock<Option<PayloadErrorLimits>>>,
+}


I think this should probably be an interceptor, rather than a client wrapper?

There are a couple of issues that I think prevent this from being an interceptor:

tonic interceptors receive type-erased request bodies. We need the typed body to decide if it should be validated and how. I do not think there is a technical workaround as an interceptor. It could be done as a tower layer, but that would require decoding the already encoded body, which seems wasteful.

The connection is namespace unaware. I don't think we'd be able to construct the interceptor at channel creation time and be able to plumb through namespace limit information. We'd have to mutate the interceptor later but not for the client because that'll lead to inconsistent behavior depending on when a worker is attached to the channel. We could copy of the channel for each worker, but then we lose pooling/sharing.

We could eliminate this wrapper and just have the call sites attach the error limit information per caller; that would be ripe for forgetting to do that for new callers, but its workable and could be made with a simple convenience method.

If opposed to the client wrapper, I could do the per-call site attachment.

Ah, yeah, good points. I forgot that there isn't really a kind of interceptor that would allow us to have all that in one place.

I suppose my only other comment is I don't think this needs to be fully pub?

Sushisource · 2026-07-02T18:30:20Z

+/// The violation, if `status` is the client proactively rejecting an outbound request for exceeding a
+/// payload/memo error size limit.


Suggested change

/// The violation, if `status` is the client proactively rejecting an outbound request for exceeding a

/// payload/memo error size limit.

/// Returns the violation, if `status` is the client proactively rejecting an outbound request for exceeding a

/// payload/memo error size limit.

Sushisource · 2026-07-02T18:45:06Z

+pub struct PayloadLimitsOptions {
+    /// Warning threshold (bytes) for the size of an outbound payload-bearing field; over-threshold
+    /// fields are logged but still sent to server. Defaults to 512 KiB. Set to `0` to disable.
+    pub payloads_size_warn: u64,


Option 3 works for me

Sushisource · 2026-07-02T18:46:38Z

+/// Warn/error thresholds for both limit classes. A `None` threshold disables that check for that
+/// class: `None` warn = no warnings, `None` error = no error enforcement (warnings only).
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 pub struct PayloadLimits {


If the resolver thing earlier resolves 0 to None, I'd kinda vote for these just being 0 also means disabled? 0-but-respected seems like a bit of a nonsense state

Sushisource · 2026-07-02T18:49:32Z

+                                    Err(e) if payload_limit_violation_from(&e).is_some() => {
+                                        let violation = payload_limit_violation_from(&e)
+                                            .expect("violation present per guard");


I want to say this can be combined with an if-let (forget if those work in match arms)

Sushisource · 2026-07-02T18:50:09Z

+                        Err(e) if payload_limit_violation_from(&e).is_some() => {
+                            let violation = payload_limit_violation_from(&e)


Same here on if-let

Sushisource · 2026-07-02T18:52:12Z

+                                Err(e) => {
+                                    if let Some(violation) = payload_limit_violation_from(&e) {
+                                        act_metrics.act_execution_failed();
+                                        client


Does this make sense? Should we instead just truncate the cancel details or something?

Sushisource · 2026-07-02T18:53:43Z

+    /// Issues outbound gRPC calls, automatically attaching this worker's payload/memo error limits
+    /// (set via `set_payload_error_limits`) so the gRPC layer can enforce them. Wraps a clone of
+    /// `connection`, so a client replacement on `connection` is reflected here too.
+    client: PayloadLimitsClient<SharedReplaceableClient<Connection>>,


Per my bit about interceptors, I think ideally we wouldn't need to store a separate client here..

Sushisource · 2026-07-02T18:58:59Z

+                        task_queue=%self.config.task_queue, worker_id=%self.client.identity(),
+                        workflow_id, run_id, workflow_type, activity_type, attempt))]


Adding these additional attributes could be nice, could also be noisy, was this to match other SDKs?

jmaeagle99 force-pushed the payload-limits branch 6 times, most recently from c42f207 to aebcc45 Compare July 1, 2026 02:46

feat: payload limit enforcement

a9f694c

jmaeagle99 force-pushed the payload-limits branch from aebcc45 to a9f694c Compare July 1, 2026 05:01

jmaeagle99 marked this pull request as ready for review July 1, 2026 05:20

jmaeagle99 requested a review from a team as a code owner July 1, 2026 05:20

jmaeagle99 commented Jul 1, 2026

View reviewed changes

jmaeagle99 changed the title ~~feat: payload limit enforcement~~ 💥 Payload limit enforcement Jul 1, 2026

Sushisource reviewed Jul 2, 2026

View reviewed changes

		/// The violation, if `status` is the client proactively rejecting an outbound request for exceeding a
		/// payload/memo error size limit.

		Err(e) if payload_limit_violation_from(&e).is_some() => {
		let violation = payload_limit_violation_from(&e)

		task_queue=%self.config.task_queue, worker_id=%self.client.identity(),
		workflow_id, run_id, workflow_type, activity_type, attempt))]

Uh oh!

Conversation

jmaeagle99 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

jmaeagle99 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmaeagle99 commented Jun 30, 2026 •

edited

Loading