feat: add top level async streaming by jakelorocco · Pull Request #655 · generative-computing/mellea

jakelorocco · 2026-03-16T13:57:19Z

Misc PR

Type of PR

Bug Fix
New Feature
Documentation
Other

Description

Link to Issue: Fixes fix: make asynchronous functions / session functions return uncomputed model output thunks #381

Top level async functions (aquery, ainstruct, aact) can now return uncomputed ModelOutputThunks. These will be uncomputed by default if strategy=None.

Other top level functions (sync and atransform/achat, etc...) can only return ComputedModelOutputThunks (since their actions rely on the computed result).

ComputedModelOutputThunks are a new type that are just mots + is_computed == True. This allows for better type hinting of results. Sampling results are typed with these computed mots for the same reason as the sync functions.

Added infrastructure for supporting type-hinting tests to ensure future changes don't result in regressions. (Note: only added for the new type hinting, will need to retroactively add for older type hints.)

I've additionally added an m fix async command that can analyze python files with mellea code and ensure changes caused by the top level async change are fixed if necessary. I'd advise looking at the command description to see all the options; but it will correctly fix all changes in the current version of the code. In the future, if someone relies on passing an uncomputed mot around from those top-level async functions, there may be false positives that get fixed. The command --help message adds this context.

Adds an example for top level async streaming / lazy computation.

Testing

Tests added to the respective file if code was changed
New code has 100% coverage if code as added
Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

github-actions · 2026-03-16T13:57:35Z

The PR description has been updated. Please fill out the template for your PR to be reviewed.

mergify · 2026-03-16T13:57:56Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

psschwei

minor nits but otherwise looks good to me

nrfulton

The main thing we're still missing is an articulation of the motivation for this change (even in the PR description). Otherwise looks good. I did not review the CLI tool.

nrfulton · 2026-03-25T22:20:23Z

Not reviewed.

nrfulton · 2026-03-25T22:20:29Z

Not reviewed.

nrfulton · 2026-03-25T22:20:38Z

Not reviewed.

nrfulton · 2026-03-25T22:29:06Z

not reviewed.

nrfulton · 2026-03-26T00:49:05Z

                    strategy=extracted.strategy,
                    format=self._response_model,
                    model_options=extracted.model_options,
+                    await_result=True,


might want to document in the docstring that setting this to false is probably a bad idea unless you really understand what you're doing...

This specific invocation site will always be True (since generative slots rely on parsing the output).

When we chatted about the implementation of this, we agreed that the default should actually be lazy. So it's False by default. Most people won't be impacted though since strategies / sync will auto-compute this anyways.

jakelorocco · 2026-03-26T13:38:06Z

The main thing we're still missing is an articulation of the motivation for this change (even in the PR description). Otherwise looks good. I did not review the CLI tool.

This needs to be added if we want to enable lazily computed model output thunks and streaming when utilizing mfuncs and sessions. Otherwise, streaming and lazy compute is limited to backend.generate calls.

~~I will find somewhere in the docs / examples to add this explanation.~~ Done

Co-authored-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>

nrfulton

lgtm

* feat: add top level async streaming Co-authored-by: Akihiko Kuroda <akihikokuroda2020@gmail.com> * feat: add examples for top level streaming * fix: pr comments * fix: pr comments and add simpler example * fix: issues with docs gen / quality * feat: add test files for mypy to catch overload typing issues --------- Co-authored-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>

jakelorocco force-pushed the jal/finalize-async branch from 123cbc8 to 7209da3 Compare March 16, 2026 14:36

jakelorocco mentioned this pull request Mar 16, 2026

feat: add stream support to mfuncs and session #395

Closed

8 tasks

jakelorocco marked this pull request as ready for review March 16, 2026 15:04

jakelorocco requested review from a team and nrfulton as code owners March 16, 2026 15:04

psschwei reviewed Mar 16, 2026

View reviewed changes

Comment thread mellea/core/base.py Outdated

Comment thread mellea/core/base.py Outdated

planetf1 added documentation Improvements or additions to documentation and removed documentation Improvements or additions to documentation labels Mar 17, 2026

jakelorocco force-pushed the jal/finalize-async branch 2 times, most recently from 0bb530f to 7cfb1ce Compare March 17, 2026 19:15

psschwei approved these changes Mar 18, 2026

View reviewed changes

Comment thread test/core/test_streaming_sync_functions.py Outdated

Comment thread mellea/core/base.py Outdated

Comment thread mellea/core/base.py Outdated

nrfulton reviewed Mar 26, 2026

View reviewed changes

github-actions Bot added the enhancement New feature or request label Mar 26, 2026

jakelorocco and others added 4 commits March 26, 2026 13:49

feat: add top level async streaming

7a9a36f

Co-authored-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>

Merge branch 'main' into jal/finalize-async

14b00ce

feat: add examples for top level streaming

fa9b43c

fix: pr comments

9768fd9

jakelorocco force-pushed the jal/finalize-async branch from 25dfae7 to 889aa90 Compare March 26, 2026 17:54

fix: pr comments and add simpler example

e6216e8

jakelorocco force-pushed the jal/finalize-async branch 4 times, most recently from 812124b to d23fb9e Compare March 26, 2026 18:28

fix: issues with docs gen / quality

fffa659

jakelorocco force-pushed the jal/finalize-async branch from d23fb9e to fffa659 Compare March 26, 2026 19:55

jakelorocco enabled auto-merge March 26, 2026 19:56

feat: add test files for mypy to catch overload typing issues

4652cdf

nrfulton self-requested a review April 2, 2026 17:34

nrfulton approved these changes Apr 2, 2026

View reviewed changes

jakelorocco added this pull request to the merge queue Apr 2, 2026

Merged via the queue into main with commit deb9d24 Apr 2, 2026
8 checks passed

jakelorocco deleted the jal/finalize-async branch April 2, 2026 18:14

Uh oh!

Conversation

jakelorocco commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Misc PR

Type of PR

Description

Testing

Uh oh!

github-actions Bot commented Mar 16, 2026

Uh oh!

mergify Bot commented Mar 16, 2026

Merge Protections

🟢 Enforce conventional commit

Uh oh!

Uh oh!

Uh oh!

psschwei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nrfulton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nrfulton Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

nrfulton Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

nrfulton Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

nrfulton Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nrfulton Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

jakelorocco Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakelorocco Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakelorocco commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nrfulton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakelorocco commented Mar 16, 2026 •

edited

Loading

jakelorocco Mar 26, 2026 •

edited

Loading

jakelorocco commented Mar 26, 2026 •

edited

Loading