Export-import between workspace-enabled servers

[frankcaoyun]

Hi,

Context

To follow up on the discussion here

We currently have a number of mlflow databases (Cloud SQL) for each team without workspace enabled, and we would like to migrate all team's data into the new single mlflow database (Cloud SQL) and separate them by workspaces. The curernt tool doesn't support this scenario.

The Ask

Move content from server A database A workspace A into server B database B, keep or rename the workspace name in the target database.

For example:
export-experiment --experiment sklearn-wine --workspace workspace_old
import-experiment --experiment sklearn-wine --workspace workspace_new

[mprahl]

@a-gajam what are your thoughts here?

[a-gajam]

@mprahl i see env variable which we can set https://mlflow.org/docs/latest/self-hosting/workspaces/#workspace-resolution. Every time we run export/import we can set this env variable.

If neither mlflow.set_workspace() nor MLFLOW_WORKSPACE is set, MLflow falls back to the provider default workspace (typically default). If no workspace can be resolved and the provider does not support a default workspace, API calls will return an error.

[mprahl]

@a-gajam I hadn't looked at the code, so I didn't realize it was just using the MLflow client under the hood, so that should work for most of this.

For the migration in this issue (team DBs without workspaces → one workspace-enabled server):

# Export from the old non-workspace server
export MLFLOW_TRACKING_URI=<source>
export-experiment --experiment sklearn-wine --output-dir /tmp/exp

# Import into the new shared server under a team workspace
export MLFLOW_TRACKING_URI=<destination>
export MLFLOW_WORKSPACE=workspace_new
import-experiment --experiment sklearn-wine --input-dir /tmp/exp

I had an agent take a look and it saw a few code paths use a custom HTTP client (not MlflowClient), and that client does not currently attach X-MLFLOW-WORKSPACE. If we hit gaps there, we should fix that client to honor MLFLOW_WORKSPACE. We probably need to improve the documentation as well.

@frankcaoyun can you try the env-var approach above and report back if anything fails?