Skip to content

Fix sync_compatible resolution in synchronous transaction hooks#22626

Open
hsusul wants to merge 1 commit into
PrefectHQ:mainfrom
hsusul:fix-transactions-sync-rollback-hooks
Open

Fix sync_compatible resolution in synchronous transaction hooks#22626
hsusul wants to merge 1 commit into
PrefectHQ:mainfrom
hsusul:fix-transactions-sync-rollback-hooks

Conversation

@hsusul

@hsusul hsusul commented Jul 25, 2026

Copy link
Copy Markdown

Summary

This PR fixes an issue where @sync_compatible functions (such as Block.load or prefect.variables.get) returned unawaited coroutine objects instead of synchronous values when called inside synchronous transaction hooks (on_rollback, on_commit, on_completion) during an asynchronous flow run.

Problem

When a synchronous transaction hook (e.g. @task.on_rollback) is executed during an async flow, the transaction runner executes the hook in a synchronous thread (or directly as a sync function call). However, @sync_compatible functions inspect the enclosing run context (FlowRunContext), see that the parent flow is asynchronous (flow.isasync == True), and assume that the caller will await the returned coroutine. Because synchronous hook functions are not async def and cannot await coroutines, the hook receives a raw unawaited coroutine object (causing AttributeError or TypeError when accessing properties on the returned value).

Root cause

  1. sync_compatible checked parent_obj.isasync without checking if the current thread has an active event loop or if RUNNING_ASYNC_FLAG was explicitly set to False.
  2. RUNNING_ASYNC_FLAG defaulted to False, making RUNNING_ASYNC_FLAG.get() is False indistinguishable from the default unset state.
  3. Transaction hook handlers did not set RUNNING_ASYNC_FLAG to False when invoking synchronous hook callables.

Fix

  1. Updated RUNNING_ASYNC_FLAG default to None so that RUNNING_ASYNC_FLAG.get() is False uniquely identifies contexts explicitly marked as synchronous.
  2. Updated sync_compatible to verify that an active event loop exists in the calling thread and respect RUNNING_ASYNC_FLAG.get() is False.
  3. Updated Transaction.run_hook and AsyncTransaction.run_hook to execute synchronous hooks with RUNNING_ASYNC_FLAG.set(False) (using run_sync_in_worker_thread for async transaction runners).

Testing

  • uv run pytest tests/test_transactions.py tests/utilities/test_asyncutils.py — Passed (148 passed, 8 skipped).
  • Verified deterministic reproduction script repros/17656.py fails before the fix and passes cleanly after the fix.

Compatibility

Preserves backward compatibility across all client and server APIs.

Related issue

Closes #17656

@github-actions github-actions Bot added the bug Something isn't working label Jul 25, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68b38476f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

await hook(self)
else:
await anyio.to_thread.run_sync(hook, self)
await run_sync_in_worker_thread(hook, self)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep synchronous transaction hooks shielded from cancellation

When an AsyncTransaction is unwound inside an active cancellation scope (for example, during a flow or task timeout), this helper uses abandon_on_cancel=True, so cancellation stops awaiting the synchronous commit or rollback hook while its worker thread continues running. The previous anyio.to_thread.run_sync call retained the default shielding behavior and waited for the hook to finish; this change can therefore abort rollback()/commit() and reset() while cleanup side effects continue after the transaction context has unwound. Preserve the prior non-abandoning cancellation behavior while propagating the synchronous context flag.

Useful? React with 👍 / 👎.

@codspeed-hq

codspeed-hq Bot commented Jul 25, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark1


Comparing hsusul:fix-transactions-sync-rollback-hooks (68b3847) with main (d2d873b)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

on_rollback runs in async context if failing task is async

1 participant