Skip to content

25523: Adds suite of tools to provide a more unified progress experience - MINOR#692

Open
mkoistinen wants to merge 7 commits into
mainfrom
25523-adds-task_id_to-relevant-methods
Open

25523: Adds suite of tools to provide a more unified progress experience - MINOR#692
mkoistinen wants to merge 7 commits into
mainfrom
25523-adds-task_id_to-relevant-methods

Conversation

@mkoistinen

@mkoistinen mkoistinen commented Jul 10, 2026

Copy link
Copy Markdown
Member

This PR is all about providing transparent progress reporting and a suite of facilities for instrumenting code within this repo and things downstream from it to provide and display progress:

  • HSE-based progress is implemented here
  • Batch-based (batch react, batch train) implemented here
  • Python-layer facilities for anything that should show progress for downstream use
  • Facilities to provide console "Keep Alive" to prevent Databricks from hanging up on long-running tasks
  • Tools to alter the defaults: force on, force off, etc.

The HSE-based progress is automatically applied to:
train, analyze, react, react_series, react_series_stationary, impute, react_group, react_into_features, react_aggregate, and reduce_data where the engine provides support.

The progress reporting degrades gracefully in less capable environments. Part of this is provided by rich, with a RichProgressReporter but also by a separate SimpleProgressReporter which was designed for use in notebooks. This PR tries to be smart about detecting the run-time environment to decided which reported to use.

For manual control over the automatic progress, we have these in the public API:

  • Thread-local override: auto_progress_scope() context manager, enable_/disable_/reset_auto_progress().
  • Client config auto_progress (on / off / auto).
  • Env HOWSO_PROGRESS (plus HOWSO_SIMPLE_PROGRESS, HOWSO_HEARTBEAT_INTERVAL).

@mkoistinen
mkoistinen requested review from a team as code owners July 10, 2026 20:50
@mkoistinen mkoistinen changed the title 25523 adds task id to relevant methods 25523: Adds suite of tools to provide a more unified progress experience - MINOR Jul 10, 2026
Comment thread howso/client/base.py Outdated
Comment thread howso/engine/trainee.py
Comment thread howso/engine/trainee.py Outdated
Comment thread howso/engine/trainee.py Outdated
Comment thread howso/utilities/progress.py
Comment thread howso/client/base.py
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py Outdated
Comment thread howso/utilities/progress.py Outdated
Comment thread howso/utilities/progress.py Outdated
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py
Comment thread howso/utilities/progress.py
@mkoistinen
mkoistinen force-pushed the 25523-adds-task_id_to-relevant-methods branch from c64e67b to ed0a06f Compare July 14, 2026 03:17
@mkoistinen
mkoistinen force-pushed the 25523-adds-task_id_to-relevant-methods branch from b71ef23 to d99157f Compare July 14, 2026 16:03
mkoistinen and others added 7 commits July 24, 2026 09:51
Mechanical, no-behavior-change refactor split out from the task_id/progress
feature so that feature diff stays reviewable:

- Remove `import typing as t`; convert `t.Optional[X]` -> `X | None`,
  `t.Any` -> `Any`, `t.Union[...]` -> `... | None`, `t.overload`/`t.Protocol`/
  `t.TypeVar` -> direct imports across client/base.py, engine/trainee.py, and
  utilities/internals.py.
- Assorted lint/style cleanup (import ordering, signature formatting,
  ConsoleFeedback stop-event check, trailing newline).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ework

Adds a `task_id` keyword to the long-running client/engine operations and a
transparent-progress framework (howso/utilities/progress.py) that surfaces
engine + batch progress via an `@auto_progress` decorator, with
`AbstractHowsoClient.get_progress()` for concurrent progress polling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mkoistinen
mkoistinen force-pushed the 25523-adds-task_id_to-relevant-methods branch from 6b56adc to c31c525 Compare July 24, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants