-
Notifications
You must be signed in to change notification settings - Fork 30
Add azure-functions-durable v2: a durabletask-based rewrite of Durable Functions for Python #155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
andystaples
wants to merge
62
commits into
microsoft:main
Choose a base branch
from
andystaples:andystaples/add-functions-support
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
7ff1525
Storing changes commit
andystaples 552a2dd
Working orchestrators + activities
andystaples af0e3c2
Nitpicks and cleanup
andystaples 86ee081
Merge branch 'main' into andystaples/add-functions-support
andystaples 3497148
Save-all nits
andystaples 57de878
Add entity support (needs extension change)
andystaples 18145f8
Refine entity support
andystaples 9965ba4
Finish entity support
andystaples 209443e
Fixes and improvements
andystaples cc005ae
Bump durabletask version, fix metadata
andystaples bf6d6f2
Use Protocol for stubs
andystaples 0a03bd1
Merge branch 'main' into andystaples/add-functions-support
andystaples 1176d03
Update to new workflow pattern
andystaples 7bf763a
Rename stub file
andystaples 811653e
Fix import
andystaples 068cb43
Merge branch 'main' into andystaples/add-functions-support
andystaples 827d201
Experimental dependency revision
andystaples fde02c5
Update to match changes in functions SDK
andystaples 5b9644d
Merge branch 'main' into andystaples/add-functions-support
andystaples 2df96dc
Merge issue fix
andystaples eac9efd
Various
andystaples 20aacab
Add Functions to requirements
andystaples 5df87b1
Rename to azure-functions-durable v2
andystaples ea5c2b0
Re-add Orchestrator object/model
andystaples 2d5fd7a
Merge remote-tracking branch 'origin/main' into andystaples/add-funct…
andystaples 0505a0d
Modernize pipelines for functions package
andystaples c9ea2fb
Cleanup pyright errors
andystaples 5b906bd
Remove non-existent extension call
andystaples 643e6ea
Merge remote-tracking branch 'origin/main' into andystaples/add-funct…
andystaples df9932d
Serialization compat, fix typing, working again
andystaples a7bdd01
Use the async client (match old behavior)
andystaples 1891d94
Compat layer V1
andystaples d4a84bc
Reorganize compat files
andystaples a1b54bd
Add shims for old orchestration and entity call arg patterns
andystaples 3c05264
Add remaining orchestration surface
andystaples 46570e9
Add missing type coersion, rich client, tests
andystaples af18afe
Fix gaps found via integration tests
andystaples 013f674
Fix old-name interceptor lookup
andystaples 60b6195
More compat layer stuff
andystaples 16f3cf1
Merge remote-tracking branch 'origin/main' into andystaples/add-funct…
andystaples 2c7d5b2
Return an emtpy function context instead of raising
andystaples 69cbc45
Asyncio in test matrix
andystaples 95e0b9e
CI Tweaks
andystaples b69eeeb
Add Durable HTTP feature
andystaples 905a400
CHANGELOG for now
andystaples 8de1900
Add unit testing coverage
andystaples 7d48ad3
Add E2E harness
andystaples 4a0ed32
Expand E2E coverage
andystaples ff203b7
More E2E test edge cases, fixes, add scheduled tasks
andystaples 25f21df
Add history export to afd
andystaples 914a083
Merge remote-tracking branch 'origin/main' into andystaples/add-funct…
andystaples f0fdf05
Add afd rewind
andystaples 483c4aa
lint fix
andystaples eea3c46
Pyright fixes
andystaples 469de15
Merge remote-tracking branch 'origin/main' into andystaples/add-funct…
andystaples 43dcd0f
Review feedback
andystaples a38f4ab
Review Feedback 2
andystaples 6baa78a
2.0.0b1 CHANGELOG rewrite
andystaples 3cb8f63
Review Feedback 3
andystaples 72cea0e
Fix CI
andystaples a0668a3
Review Feedback 4
andystaples c10c6f4
History export pagination fix
andystaples File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,4 +3,7 @@ ignore = E501,C901,W503 | |
| exclude = | ||
| .git | ||
| *_pb2* | ||
| __pycache__ | ||
| __pycache__ | ||
| .venv | ||
| .nox | ||
| .python_packages | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| name: Durable Task Scheduler SDK (azure-functions-durable) | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - "main" | ||
| tags: | ||
| - "azurefunctions-v*" # Only run for tags starting with "azurefunctions-v" | ||
| pull_request: | ||
| branches: | ||
| - "main" | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| lint: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Set up Python 3.14 | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: 3.14 | ||
| - name: Install dependencies | ||
| working-directory: azure-functions-durable | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install setuptools wheel tox | ||
| pip install flake8 | ||
| - name: Run flake8 Linter | ||
| working-directory: azure-functions-durable | ||
| run: flake8 . | ||
| - name: Run flake8 Linter | ||
| working-directory: tests/azure-functions-durable | ||
| run: flake8 . | ||
|
|
||
| run-tests: | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.13", "3.14"] | ||
| needs: lint | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
|
|
||
| - name: Install durabletask dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install flake8 pytest | ||
| pip install -r requirements.txt | ||
|
|
||
| - name: Install durabletask locally | ||
| run: | | ||
| pip install . --no-deps --force-reinstall | ||
|
|
||
| - name: Install azure-functions-durable locally | ||
| working-directory: azure-functions-durable | ||
| run: | | ||
| pip install . --no-deps --force-reinstall | ||
|
|
||
| - name: Run unit tests | ||
| working-directory: tests/azure-functions-durable | ||
| run: | | ||
| pytest -m "not dts and not azurite and not functions_e2e" --verbose | ||
|
|
||
| e2e-tests: | ||
| needs: run-tests | ||
| runs-on: ubuntu-latest | ||
| # The end-to-end suite launches a real Azure Functions host, which requires | ||
| # ``azure-functions>=2.2.0b6``. That build is not yet published to PyPI, so | ||
| # this job is skipped by default. Enable it by setting the repository | ||
| # variable ``RUN_FUNCTIONS_E2E`` to ``true`` (Settings > Secrets and | ||
| # variables > Actions > Variables) once 2.2.0b6 is available on PyPI, or | ||
| # provide the library build via ``AZURE_FUNCTIONS_PYTHON_LIBRARY``. | ||
| if: ${{ vars.RUN_FUNCTIONS_E2E == 'true' }} | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.13" | ||
|
|
||
| - name: Set up Node.js (needed for Azurite and Functions Core Tools) | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: "20.x" | ||
|
|
||
| - name: Install Azurite and Azure Functions Core Tools | ||
| run: | | ||
| npm install -g azurite | ||
| npm install -g azure-functions-core-tools@4 --unsafe-perm true | ||
|
|
||
| - name: Start Azurite | ||
| shell: bash | ||
| run: | | ||
| azurite --silent --location /tmp/azurite --blobPort 10000 --queuePort 10001 --tablePort 10002 & | ||
| sleep 2 | ||
|
|
||
| - name: Install nox | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install nox | ||
|
|
||
| - name: Run end-to-end tests | ||
| # nox provisions each sample app's in-app virtual environment (a link to | ||
| # a single editable install) so the Functions worker loads our | ||
| # grpc/protobuf rather than its bundled copies. | ||
| run: nox -s functions_e2e | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| # Changelog | ||
|
|
||
| All notable changes to this project will be documented in this file. | ||
|
|
||
| The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), | ||
| and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). | ||
|
|
||
| ## 2.0.0b1 | ||
|
|
||
| First preview (beta) release of `azure-functions-durable` 2.x — a ground-up | ||
| rewrite of the Azure Durable Functions Python SDK built on top of the | ||
| [`durabletask`](https://pypi.org/project/durabletask/) SDK. This is a preview | ||
| release; APIs may change before the stable 2.0.0. | ||
|
|
||
| ### Why the rewrite | ||
|
|
||
| The 1.x SDK implemented the Durable Functions out-of-process programming model | ||
| directly, with its own orchestration action/state model and a JSON protocol | ||
| tailored to the classic Durable Functions host extension. 2.x instead builds on | ||
| the `durabletask` Python SDK — the same gRPC-based runtime that powers the | ||
| Durable Task Scheduler (DTS) and the modern Durable Functions host. Building on | ||
| durabletask means: | ||
|
|
||
| - a single orchestration/entity execution core, serialization pipeline, and | ||
| retry/versioning implementation shared with the broader durabletask | ||
| ecosystem, instead of a Functions-specific reimplementation; | ||
| - Functions users can adopt durabletask-native APIs and patterns directly, | ||
| while existing v1 code keeps working through the compatibility layer; and | ||
| - less protocol drift between the Python worker and the durable backend. | ||
|
|
||
| ### Underlying packages | ||
|
|
||
| - [`durabletask`](https://pypi.org/project/durabletask/) — orchestration and | ||
| entity client/worker, executed over gRPC. | ||
| - [`azure-functions`](https://pypi.org/project/azure-functions/) — the | ||
| decorator/binding programming model (`DFApp` / `Blueprint`). | ||
| - [`azure-identity`](https://pypi.org/project/azure-identity/) — Managed | ||
| Identity token acquisition for durable HTTP calls. | ||
|
|
||
| ### Breaking changes (from `azure-functions-durable` 1.x) | ||
|
|
||
| - **Python 3.13+ is now required** (1.x supported 3.10+). On Functions Python | ||
| workers older than 3.13, worker dependencies are not isolated from your app's | ||
| dependencies, which causes a `grpc` version conflict with the durable runtime | ||
| at load time. Python 3.13 enables worker dependency isolation, avoiding the | ||
| collision. | ||
| - **The classic (v1) programming model has been dropped.** Only the | ||
| decorator-based application model (`DFApp` / `Blueprint`, the Python v2 | ||
| programming model) is supported; the `function.json`-based model is not. | ||
| - **The OpenAI Agents integration has been removed.** The | ||
| `azure.durable_functions.openai_agents` package (durable OpenAI Agents SDK | ||
| orchestration) is not part of 2.x. | ||
| - **Runtime target.** 2.x speaks the durabletask gRPC protocol used by the | ||
| Durable Task Scheduler and the modern Durable Functions host, rather than the | ||
| classic Durable Functions extension protocol. | ||
| - **Primary client and context APIs use durabletask names.** The main surface | ||
| is now durabletask's (e.g. `schedule_new_orchestration`, | ||
| `get_orchestration_state`, `wait_for_orchestration_completion`). The v1 | ||
| `DurableOrchestrationClient` method names remain available as deprecated | ||
| aliases that emit `DeprecationWarning` (see [Deprecated](#deprecated)). | ||
|
|
||
| ### Added | ||
|
|
||
| New capabilities beyond the v1 surface, most inherited from `durabletask`: | ||
|
|
||
| - **durabletask-native authoring.** Two-argument orchestrator and entity | ||
| functions (`def orchestrator(ctx, input)`, `def entity(ctx, input)`) and | ||
| class-based entities (`DurableEntity`) are first-class, alongside the | ||
| supported v1-style single-argument functions. | ||
| - **`DurableFunctionsClient.rewind_orchestration(...)`** rewinds a failed | ||
| orchestration to its last known good state (inherited from durabletask). | ||
| - **`DFApp.configure_scheduled_tasks()`** opts an app in to durabletask | ||
| scheduled (recurring) tasks by registering the schedule entity and operation | ||
| orchestrator. Schedules are then managed from a client via | ||
| `durabletask.scheduled.ScheduledTaskClient`. Scheduled tasks are not | ||
| registered unless this method is called. | ||
| - **`DFApp.configure_history_export()`** opts an app in to durabletask history | ||
| export by registering the export-job entity, driving orchestrator, and | ||
| activities. Export jobs are driven from a client via | ||
| `durabletask.extensions.history_export.ExportHistoryClient`; supply the | ||
| activities' runtime dependencies with `history_export.bind_context(...)`. The | ||
| instance-enumeration activity uses a Functions-specific implementation based | ||
| on `QueryInstances`, because the Durable Functions host extension does not | ||
| implement the `ListInstanceIds` gRPC call the core activity relies on. | ||
| - **`DurableOrchestrationContext.call_http(...)`** makes durable HTTP calls from | ||
| orchestrators, restoring the v1 API. The request is executed by a built-in | ||
| activity and, when the endpoint responds with `202 Accepted` and a `Location` | ||
| header, is automatically polled to completion (honoring `Retry-After`). | ||
| `ManagedIdentityTokenSource` can be supplied to attach a Managed Identity | ||
| bearer token to the request. `DurableHttpRequest` and `DurableHttpResponse` | ||
| are exported from `azure.durable_functions`. | ||
| - **`orchestration_trigger(..., input_type=...)`** decodes a v1-style | ||
| `context.get_input()` to the declared type; a call-site `expected_type` on | ||
| `get_input` takes precedence. | ||
|
|
||
| ### Compatibility with v1 | ||
|
|
||
| To ease migration, 2.x ships a compatibility layer over the durabletask | ||
| surface: | ||
|
|
||
| - **v1-style single-argument functions** (`def orchestrator(context)`, | ||
| `def entity(context)`) are supported. The worker detects the function shape | ||
| and, for single-argument functions, delivers a functional | ||
| `DurableOrchestrationContext` / `DurableEntityContext` that wraps the | ||
| durabletask context and exposes the v1 API — for orchestrations: | ||
| `get_input`, `call_activity`/`call_activity_with_retry`, | ||
| `call_sub_orchestrator`/`call_sub_orchestrator_with_retry`, `create_timer`, | ||
| `wait_for_external_event`, `continue_as_new`, `set_custom_status`, | ||
| `task_all`/`task_any`, `call_entity`/`signal_entity`, `new_uuid`/`new_guid`, | ||
| `custom_status`, `will_continue_as_new`, `parent_instance_id`, and | ||
| `function_context`; and for entities: `entity_name`, `entity_key`, | ||
| `operation_name`, `get_input`, `get_state` (with `initializer`), `set_state`, | ||
| `set_result`, and `destruct_on_exit`. The operation result is taken from | ||
| `set_result(...)`, falling back to the function's return value. | ||
| - **v1 return-type wrappers** `DurableOrchestrationStatus`, | ||
| `PurgeHistoryResult`, and `EntityStateResponse` are returned by the deprecated | ||
| client methods and exported from `azure.durable_functions`. | ||
| - **`HttpManagementPayload`** subclasses `dict`, so it is directly | ||
| JSON-serializable via `json.dumps(payload)` and supports mapping-style access, | ||
| matching v1 usage. | ||
| - **`create_http_management_payload`** accepts either the durabletask | ||
| `(request, instance_id)` or the v1 `(instance_id)` signature. | ||
|
|
||
| ### Deprecated | ||
|
|
||
| These v1 names are retained as shims that delegate to their durabletask | ||
| equivalents and emit `DeprecationWarning`; prefer the durabletask names in new | ||
| code: | ||
|
|
||
| - `DurableOrchestrationClient` (alias for `DurableFunctionsClient`) and its | ||
| method names: `start_new`, `get_status`, `get_status_all`, `get_status_by`, | ||
| `raise_event`, `terminate`, `purge_instance_history`, | ||
| `purge_instance_history_by`, `suspend`, `resume`, `restart`, | ||
| `read_entity_state`, `get_client_response_links`, and | ||
| `wait_for_completion_or_create_check_status_response`. | ||
| - `rewind(...)` — delegates to `rewind_orchestration(...)`. | ||
| - `signal_entity(..., operation_input=...)` — `operation_input` is an alias for | ||
| `input`; `task_hub_name` / `connection_name` are accepted and ignored. | ||
| - `RetryOptions` — maps the v1 millisecond-based constructor onto durabletask | ||
| `RetryPolicy` (which uses `timedelta`). `RetryPolicy` is also exported from | ||
| `azure.durable_functions`. | ||
| - Compatibility aliases exported from `azure.durable_functions`: | ||
| `DurableOrchestrationContext`, `DurableEntityContext`, `EntityId`, | ||
| `ManagedIdentityTokenSource`, `TokenSource`, `Entity`, and | ||
| `OrchestrationRuntimeStatus`. | ||
|
|
||
| ### Known limitations | ||
|
|
||
| - Orchestration history is not exposed on the context; | ||
| `DurableOrchestrationContext.histories` raises `NotImplementedError`. Use the | ||
| client's `get_orchestration_history(...)` instead. | ||
| - The client status methods accept the v1 `show_history` / | ||
| `show_history_output` flags for signature compatibility but ignore them, so | ||
| the returned status has no `historyEvents`. Use | ||
| `get_orchestration_history(...)` to retrieve history. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| MIT License | ||
|
|
||
| Copyright (c) Microsoft Corporation. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.