hydra.mini(): capability-based application entry point with opt-in behavior

[omry]

hydra.mini(): capability-based application entry point with opt-in behavior

Summary

Hydra should provide a new application entry point, provisionally named
hydra.mini(), with a minimal, predictable default and explicit opt-in runtime
capabilities.

@hydra.main() is a batteries-included application entry point. In addition
to composing a configuration and processing command-line overrides, it manages
output directories, configuration snapshots, logging, callbacks, working
directory changes, and multirun execution. These features are valuable for
experiment-oriented applications, but they make Hydra feel heavy for command
line tools, services, inference applications, data utilities, and other
programs that primarily want Hydra's configuration and CLI support.

Today these applications must opt out of several independent behaviors through
Hydra configuration. The result is difficult to discover and still cannot
fully disable Hydra-managed artifacts in all modes. For example, a single run
can be configured to leave no files by combining several overrides, while a
Basic Sweeper multirun still writes multirun.yaml.

The proposed direction is a new entry point where Hydra-managed capabilities
are opt-in instead of opt-out.

This is post-1.4 design work. This issue defines the intended shape and design
constraints, not a final API.

Goals

• Make Hydra attractive as a general configuration and CLI framework, not only
  as an experiment runner.
• Provide an application entry point that has no Hydra-managed filesystem,
  logging, environment, or working-directory side effects by default.
• Let applications explicitly declare which Hydra runtime capabilities they
  support.
• Move application and invocation state toward explicit, instance-scoped
  ownership instead of process-global singleton coordination where practical.
• Preserve the current behavior of @hydra.main() for compatibility.
• Reuse one internal composition and execution pipeline where practical, while
  keeping the Compose API and application entry points semantically distinct.

Proposed model

The new entry point would always provide the essential application path:

1. Discover the application, config sources, and config search path.
2. Parse Hydra's CLI and task overrides.
3. Compose the application configuration.
4. Invoke the application function.

Hydra-managed runtime behavior beyond that baseline would be enabled through
declared capabilities. The exact API and names remain open, but possible
capability areas include:

• Help and shell completion.
• Multirun with a sweeper and launcher.
• Output-directory management.
• Saving composed config, Hydra config, and override snapshots.
• Hydra and application logging configuration.
• Changing the application's working directory.
• Callbacks and other lifecycle hooks.
• Environment-variable management.

Capability declaration should describe what the application supports. Runtime
configuration can still control how an enabled capability behaves. This
separates availability from configuration: for example, an application can
declare multirun support, the user can select it with --multirun, and the
composed Hydra config can select the sweeper and launcher.

Enabled capabilities should contribute their own CLI surface, configuration,
and lifecycle behavior. The minimal entry point should not load a registry of
features that were not enabled or contain special handling for their absence.
For example, without the multirun capability, --multirun is simply not part
of the application's CLI.

The same principle may apply to Hydra's own configuration schema.
hydra.mini() does not necessarily need to compose the existing HydraConf
with fields for every legacy feature. It may use a new HydraMiniConfig with a
different structure containing only the baseline composition and invocation
state. Enabled capabilities can contribute their own configuration schema and
defaults to that object.

This means an absent capability does not need a disabled setting or a missing
placeholder. For example, without output management there may be no
hydra.runtime.output_dir field at all. The name HydraMiniConfig and its
structure are provisional, but compatibility with the existing HydraConf
shape should not constrain the design prematurely.

Some capabilities may be composed from lower-level capabilities. Examples:

• Working-directory management can include output-directory management.
• File logging can include storage.
• Config snapshots can include storage.
• Multirun requires a sweeper and launcher, but should not inherently require
  Hydra-managed output directories.
• A launcher or plugin that needs persistent storage should bring that behavior
  into its enabled execution path instead of relying on an implicit global
  output directory.

Relationship to existing APIs

@hydra.main()

@hydra.main() should remain backward-compatible as the full Hydra application
entry point.

hydra.mini()

The new entry point should default to the minimal application path and allow
capabilities to be added explicitly. hydra.mini() is the working name. Its
exact decorator or application-object syntax remains open.

There is no guarantee that every @hydra.main() capability will be available
through hydra.mini(). Capabilities should be added only when they fit the
minimal, composable model. Applications that need the complete existing Hydra
lifecycle should continue to use @hydra.main().

Compose API

The Compose API should remain the programmatic composition interface. It does
not own the process CLI or invoke an application function. The new entry point
fills the gap between pure programmatic composition and the full
@hydra.main() lifecycle.

The three surfaces would therefore be:

• Compose API: programmatic configuration composition.
• hydra.mini(): CLI application lifecycle with selected capabilities.
• @hydra.main(): compatible full Hydra lifecycle.

Architectural direction

The current implementation couples composition, job execution, output
directory creation, logging, callbacks, sweeping, and launcher behavior across
_run_hydra(), Hydra.run(), Hydra.multirun(), run_job(), sweepers, and
launchers.

A capability-based entry point likely requires an explicit execution plan:

1. Assemble the application from its enabled capabilities.
2. Build and parse only the CLI surface contributed by that application.
3. Compose the config.
4. Execute the resulting lifecycle.

This should make side effects visible in the execution model instead of
assuming them inside run_job(), the Basic Sweeper, or the Basic Launcher.

Plugin discovery should follow the same layered model. Plugins required for
composition, such as config source and search-path plugins, belong to the
baseline and should be available to every hydra.mini() application. Other
plugin types should be discovered only when their corresponding capability is
enabled. For example, enabling multirun can discover sweeper and launcher
plugins, while a minimal single-run application should not scan for them.

This is also an opportunity to modernize Hydra plugin discovery using Python
package entry points. Entry points can identify plugins by type without
importing and scanning the entire hydra_plugins namespace. The migration is
tracked by #3252. The
hydra.mini() design should align with that work so each baseline or enabled
capability queries only the relevant entry-point group.

Instance-scoped state is a desired architectural direction for
hydra.mini(). Ideally, an application instance owns its config loader,
enabled capability state, runtime state, and invocation context instead of
taking exclusive ownership of mutable process-global state. This would make
multiple, nested, and eventually concurrent applications easier to support.

However, current Hydra internals rely on singletons including GlobalHydra,
HydraConfig, JobRuntime, Plugins, and callback caches. Removing all such
dependencies may conflict with the design of the rest of Hydra and may be too
large a prerequisite for the initial implementation. The design should avoid
introducing new singleton dependencies, reduce mutable global coordination
where practical, and document any remaining process-global constraints.

Process-wide immutable registrations or caches are less concerning than
mutable invocation state. Compatibility adapters for @hydra.main() may also
continue to use existing global mechanisms.

Callbacks and plugins must not assume that legacy Hydra configuration fields
or output artifacts are universally available. Capabilities that need those
facilities should contribute and consume their own explicit configuration.

Initial acceptance criteria

• A minimal single-run application creates no files or directories, does not
  reconfigure logging, does not change the working directory, and does not
  mutate the process environment through Hydra.
• A minimal multirun-capable application can execute a local sweep without
  creating output directories, job logs, config snapshots, or
  multirun.yaml.
• A minimal application discovers composition plugins but does not discover or
  import plugins belonging only to capabilities it did not enable.
• The minimal Hydra configuration contains no disabled or missing placeholders
  for capabilities that were not enabled.
• Existing @hydra.main() applications remain compatible.
• The documentation clearly compares the Compose API, the new entry point,
  and @hydra.main().

Non-goals for the initial design

• Splitting out and publishing a separate minimal Hydra distribution.
• Changing the default behavior of @hydra.main().
• Finalizing the decorator or application-object syntax in this issue.
• Requiring every launcher to operate without persistent storage.
• Finalizing all capability boundaries before an implementation prototype.

Open design questions

• Should capabilities always be declared individually, or should Hydra config
  composition also offer reusable capability bundles? Config composition can
  express such bundles, but it is not yet clear that adding this indirection is
  desirable or how capability-driven CLI assembly would interact with it.
• Which CLI functions belong to the minimal baseline, especially help,
  completion, config inspection, and diagnostic commands?
• How should entry-point groups represent plugin types and capabilities while
  preserving compatibility with existing plugin packages?
• How should launchers and sweepers declare required capabilities?
• Should hydra.mini() use a new HydraMiniConfig, and how should enabled
  capabilities contribute schemas and defaults without recreating a monolithic
  Hydra configuration?
• How should capability declarations interact with Hydra config groups without
  making availability depend on configuration that has not yet been composed?
• Can @hydra.main() be implemented as a compatibility capability set over
  the new execution model, or should it initially remain a separate adapter?
• Should hydra.mini() return a reusable application object, decorate a single
  function, or support both while preserving instance-scoped state?
• Which existing singletons can be removed or isolated in the first version,
  and which process-global constraints must initially remain?

Related issues

• [Feature Request] Conditionally disable the creation of folders in output/multirun via the CLI #1937
• [Feature Request] Minimal Version of Hydra With Bare-Bones Config Support #2000
• [Feature Request] Conditionally disable the creation of output folders via callback #2846
• Merge compose API and @hydra.main #2794
• Migrate Hydra plugin discovery from namespace scanning to entry points #3252
• Support disabling working directory management #910
• [Feature Request] Promote Compose API to hydra.compose() #953