Skip to content

Issues with Artifact handling and mapping to Python Optuna package #73

Description

@AndreasHofmann217

Problematic multi-trial behavior in src/artifacts.jl:

function get_all_artifact_meta(study::Study)
    return stack([get_all_artifact_meta(study, trial) for trial in study.study.trials])[1, :]
end

This function uses stack() to combine artifact metadata from all trials. The indexing [1, :] is problematic in several cases:

  • If the study contains multiple trials without artifacts, or trials have differing numbers of artifacts, the stacking and slicing will throw an error (e.g., BoundsError or shape mismatch).
  • This means get_all_artifact_meta does not robustly handle studies with more than one trial, since the returned structure is not correctly shaped if trials differ in their artifact metadata.

Download artifact path issue:

function download_artifact(study::Study, artifact_id::String, file_path::String)
    return optuna.artifacts.download_artifact(;
    artifact_store=study.artifact_store.artifact_store,
    artifact_id=artifact_id,
    file_path=abspath(file_path) * "$artifact_id.jld2",
)
end

Here, line file_path=abspath(file_path) * "$artifact_id.jld2" means the file is always put within a directory, not at the exact path specified, possibly leading to confusion or mismatches when the path is treated as a file rather than a directory.

Test coverage does not exercise multiple-trial cases:
The test for get_all_artifact_meta in test/artifacts.jl only creates and tests a single trial:

create_test_study(; study_name="artifact_test") do study, test_dir
    trial = ask(study)
    ...
    metas = get_all_artifact_meta(study)
    @test length(metas) == 1
end

This would not catch issues with multiple trials or errors thrown due to stacking in get_all_artifact_meta.

API usability issue for single-trial metadata lookup:
The function signature for get_all_artifact_meta(study::Study, trial) expects the second argument to be a Julia Trial object. However, internally this calls Python's optuna.artifacts.get_all_artifact_meta(trial; ...), which requires a Python trial object, i.e., trial.trial from a Julia Trial.
This can cause confusion and errors: calling get_all_artifact_meta(study, trial) with a Julia Trial works, but attempting to call it with a Python trial or a Trial.trial object (as needed by the Python API) will not work as expected.

Summary:

  • Using get_all_artifact_meta on a study with multiple trials will throw errors, limiting its utility for real-world use cases where studies have many trials.
  • The handling of artifact paths may also cause confusion or unintended results when downloading artifacts.
  • The interface for single-trial artifact metadata is inconsistent: the Julia wrapper accepts a Julia Trial, but the call to the Python backend must get .trial, which is easy to overlook and breaks API composability.
  • More robust solutions for both multi-trial artifact collection, explicit path handling, and clearer/convenient API for cross-language trial identification would improve usability and predictability, especially for users migrating between Python Optuna and Optuna.jl or running studies with multiple trials.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions