[Roadmap] Improvements needed on main after the 2026-09-06 live sweep and backlog triage

[Shashankss1205]

Summary

Consolidated list of improvements still needed on main (686b04b) after the 2026-09-06 triage: a live sweep of all 26 tools over the wire with tiny datasets, a re-verification of the 2026-08-20 audit (MCP_BUG_REPORT_2026-08-20.md), and a full pass over every open issue and PR. Everything below was reproduced on current main unless marked code-reading. Items already tracked in a dedicated open issue are referenced, not repeated. Intended to be split into one issue/PR per line item.

P0 — data loss, security, transport

1. Auto-format on load reindexes to a mismatched anchor and wipes data. 12 monthly values dated on the 15th with one gap → load_data_source returns success: true, missing_values: 0, changes_made.missing_filled: 12; inspect_data then shows all 12 values null, count 0, start shifted to the next month. Same for weekly-Monday and 30-minute data. runtime/executor.py format_data_handle must only reindex when the existing index is a subset of the generated range, otherwise leave freq unset and return a frequency_warning; changes_made counts must be ints; missing_values must describe stored data. PR fix: skip silent daily fallback for unknown freq #553 fixes only the "D" catch-all ([BUG] format_data_handle silently reindexes data to daily frequency for unrecognized time intervals #313). (audit F-01, F-63)
2. HTTP/SSE transport is unusable with the official MCP client. uvicorn sktime_mcp.app:app + mcp.client.sse.sse_client → initialize() never completes; server logs RuntimeError: Unexpected ASGI message 'http.response.start' sent, after response already completed (handlers return None after driving the transport with request._send). Rewrite app.py per the SDK pattern (Mount("/messages/", app=sse.handle_post_message), return Response()), add an end-to-end test using mcp.client.sse against a uvicorn subprocess. (F-02; Feat: Add an API for sktime-mcp that can help connect to browser based LLMs #459 closed as shipped, this is the follow-up)
3. HTTP app has no auth, no DNS-rebinding protection, no CORS, and exposes run_command; README tells users to expose it through ngrok. Add a bearer-token gate (SKTIME_MCP_HTTP_TOKEN, refuse to start without it unless --insecure), TransportSecuritySettings, an explicit CORS allow-list, README warning; consider disabling run_command over HTTP. (F-03, F-41)

P1 — wrong results / broken workflows

4. Bare NaN/Infinity on the wire (invalid JSON). NaiveForecaster fit on an all-NaN handle → "2025-02": NaN in the response text. sanitize_for_json passes builtin float('nan') through and json.dumps uses allow_nan=True. Map NaN/±Inf → null centrally (builtin float and np.floating) and set allow_nan=False as a guard. PR fix: map NaN/Inf to null in sanitize_for_json #347 covers only np.floating. (F-04)
5. Exogenous data in a handle is never used; X_handle resolves to that handle's target. fit(YfromX(LinearRegression()), y_handle=h) ignores the stored X; predict(X_handle=test) returns a constant (target passed as X); evaluate(y=h, X=h) gives MAPE 0.0 (target regressed on itself). tools/fit_predict.py:59-66 uses _data_handles[X_handle]["y"]; _resolve_source ignores prefer for handles. Resolve X_handle/X= to the handle's "X" (error if none), auto-use a handle's own X when only y_handle is given, refuse when X is y. (F-05)
6. Ambiguous inputs are silently resolved. fit(y_handle=<handle>, y_dataset="airline") succeeds and the dataset wins (predictions start at 1961-01). Same pattern in predict/update. Reject when both a handle and a dataset are given for the same slot. (new, N-1)
7. Forecasts cannot be plotted, saved, or scored: predict returns JSON only; no prediction_handle. The horizon-truncation note says "use save_data" but there is nothing to save. Register the prediction (point forecast; intervals/quantiles as a DataFrame) as a data handle and return its id. (F-06)
8. evaluate is hard-wired to 1-step-ahead: ExpandingWindowSplitter(step_length=1, fh=[1]); cv_folds=3 on airline gives three folds scoring one point each. Expose fh and step_length; stop echoing cv_folds_requested when initial_window overrides it. (F-07, F-53)
9. Demo dataset catalogue: fit(DummyClassifier, X_dataset="unit_test", y_dataset="unit_test") fails because X/y are swapped (load_dataset decides (y, X) vs (X, y) from a hard-coded name list; also japanese_vowels, acsf1, covid_3month, tecator). list_available_data puts classification/regression/detection loaders and network-only loaders (m5, solar, unit_test_tsf) under "forecasting". PR fix: exclude loaders with required args from demo dataset registry #227 removes the loaders that need arguments; the rest needs introspection of return types. (F-08, F-09; narrowed [BUG] Classification workflows are discoverable but not executable end-to-end via MCP #405)
10. Scitype-specific execution crashes: detectors predict → BaseDetector.predict() got an unexpected keyword argument 'fh' ([ENH] Add support for anomaly/outlier detection workflow #83); panel/hierarchical predictions → TypeError: Setting a MultiIndex dtype (use index.map(str)); transform_data(format) on a multivariate DataFrame → The truth value of a Series is ambiguous; transformer predict(y_handle=…) → bogus 'BoxCoxTransformer' object has no attribute 'predict' (only X_handle works); clusterer predict without X → raw fh error. (F-10, F-11, F-12, F-31, F-54)
11. Adapters: integer time column year → index 1970-01-01 00:00:00.000002018, frequency "<Nano>", success (pandas/file/SQL all call pd.to_datetime unconditionally); SQL adapter with no time_column converts the RangeIndex to epoch-ns and crashes on <3 rows; an explicit frequency that does not align ("M" on mid-month dates) wipes rows to NaN with success: true (PR FIX: raise error for invalid explicit frequency in data adapters #451 raises on invalid strings but not misaligned ones); pandas time-column auto-detection is unreachable dead code ([BUG] PandasAdapter auto time-column detection never runs _detect_time_column is unreachable dead code #220, PR fix(#220): PandasAdapter auto time-column detection was unreachable dead code #238); mixed-timezone timestamps produce an object index reported as "Integer"; exog_columns="promo" (string) is iterated per character (PR fix: raise error for missing explicit exogenous columns #441 makes it error instead of silent; still needs str → [str]). (F-13, F-14, F-15, F-34, F-35, F-37)
12. URL adapter: synchronous download with no timeout and no size cap blocks the whole event loop (no tool, not even check_job_status, responds); file:// and arbitrary schemes accepted; UrlAdapter.load_async (aiohttp) is dead code but aiohttp is a hard dependency. Stream with urlopen(timeout=…), cap bytes, allow only http/https, run in a thread; drop the dead path and the dependency. (F-16, F-39, F-40)
13. query_registry applies different filter semantics with and without query: task="metric_forecasting" → 25 results without query, 0 with query="e"; list-valued tag filters and y_inner_mtype differ likewise. One filter path (sktime all_estimators filtering, then substring). Tag values are not coerced by value_type: {"capability:pred_int": "true"} silently returns 0 results; {"python_dependencies": "statsmodels"} → raw unhashable type: 'list'. Tag-name suggestions lack a substring fallback (pred_int → null; PR [FIX] : Add substring fallback for tag suggestions in list_estimators #366 needs re-targeting). 14 skchange objects get task="interval_scorer", which is not a filterable task. (F-17, F-42, F-47)
14. Env parsing: SKTIME_MCP_JOB_CLEANUP_INTERVAL=abc logs "using default" but the periodic cleanup task dies immediately with ValueError because config.py re-parses with unguarded int(); _get_int_env in server.py is dead code; SKTIME_MCP_JOB_MAX_AGE_HOURS=0 would wipe all jobs every interval ([BUG] cleanup_old_jobs_tool accepts zero/negative max_age_hours silently deletes all jobs #249). (F-18)
15. CI health: lint on main is red (7 ruff PTH errors in three August test files; ruff format --check would reformat 6 files) because .pre-commit-config.yaml pins ruff 0.3.4 while dev installs 0.15.x — pin ruff in dev extras to the same version and run ruff format. Test jobs take 1h40m–2h (25 s locally) because KNeighborsTimeSeriesClassifier on arrow_head runs without numba in tests/test_export_code_scitype.py and tests/test_nits_roundup.py — add numba to dev extras or switch to DummyClassifier; add pytest --durations=10 to CI. (F-19, F-20)

P2 — misleading responses, UX, security-adjacent

16. plot_series returns images as base64 text inside JSON, never as MCP ImageContent ([ENH] Visual Feedback & Plotting API Integration via MCP (ImageContent) #99). call_tool only ever emits TextContent.
17. SKTIME_MCP_MAX_RESPONSE_TOKENS truncation slices the JSON text mid-string and appends a notice → unparseable response. Truncate structurally (cap lists/dicts) and keep the envelope valid. (F-22)
18. predict(mode="predict_proba") returns the repr() of an skpro distribution ("Normal(columns=Index([...]), ... mu= ... sigma= ..."); serialise mean/var keyed by index or reject with a hint. predict(horizon=100000) computes all 100 000 steps before truncating to 500 — add a compute cap or reject. (F-23, F-24)
19. describe_component evaluates its name with craft(): describe_component("int") → success: true, task "estimator", module builtins.int; arbitrary expressions execute. Restrict to sktime BaseObject subclasses from all_estimators; also reject non-string names without an AttributeError (narrowed [BUG] estimator discovery tools crash or mis-handle invalid top-level input types #341). (F-25)
20. Discovery payload bloat: every tests:*/authors/maintainers tag per estimator (~60 lines × default limit=50); describe_component duplicates parameters/hyperparameters and tags/tag_explanations, cuts the docstring at 500 chars (losing the Parameters section), and gives params no type/description; no dependency-availability flag ([ENH] Add dependency availability check to describe_estimator tool #117). query_registry schema says limit/offset are ignored for task='tag' but tags are paginated (50 of 135). list_available_data(is_demo=false) embeds full metadata+validation for every handle; call_method(split) returns every fold's full index arrays with no cap. (F-26, F-27, N-4, N-5)
21. export_code output does not run outside the server for np./pd. specs (craft("CurveFitForecaster(function=np.exp)") → NameError; the server injects np/pd into craft's namespace) and for tuple-returning datasets (y = load_longley(); model.fit(y)). (F-28)
22. save_model with a file:// URI writes into a literal file: directory and reports success; save_data(format="json") fails for MultiIndex handles and for a column named time; transform_data(action="convert", to_mtype="pd.DataFrame") drops the handle's exogenous X silently; the schema advertises to_mtype='np.ndarray' but the tool rejects it. save_data writes the time column as date (csv) vs time (json) and the response does not say which column to pass back as time_column — a save/load round trip without time_column yields a RangeIndex with date as an object exog column. (F-29, F-30, F-32, F-39, N-2)
23. Secrets: _sanitize_connection_string masks only user:pass@ (?password=, PWD=, tokens survive into metadata.connection); URL adapter echoes full URLs in metadata/errors; server.py logs full tool arguments and results at INFO (connection strings, inline data, base64 images) to SKTIME_MCP_LOG_PATH. (F-33)
24. load_data_source(run_async=true) with no running loop runs synchronously but still says "job started… monitor progress"; fit/predict/evaluate raise RuntimeError in the same situation; three coroutines use get_event_loop() instead of get_running_loop() (Async tool scheduling uses deprecated/fragile event-loop access (asyncio.get_event_loop) in sync entrypoints #65). (F-38)
    24b. fit, predict and evaluate with run_async=true return success: true and a job id for a nonexistent estimator handle; the job fails a second later with "Handle not found". The except Exception: estimator_name = "Unknown" blocks in tools/fit_predict.py and tools/evaluate.py should fail fast instead (PR fix: fail fast in fit_predict_async for invalid estimator handles #105 asked to be re-targeted). (new)
25. instantiate accepts a bare class name without parentheses ("NaiveForecaster") and returns a handle wrapping the class ("estimator": "type"); fit then fails with missing 1 required positional argument: 'y'. _is_sktime_object accepts classes because hasattr(cls, "get_params"). (new)
26. describe_component/instantiate/evaluate/release_handle error ergonomics: no "did you mean" (ThetaForcaster), full module paths rejected, make_reduction(...) → bare name 'make_reduction' is not defined (factory functions unsupported, no hint), evaluate(metric="mape") aliases not accepted, release_handle/call_method on a data_… id → "Handle not found" without pointing to the data-handle tools. (F-66)
27. Docs/schema drift: docs/source/tool-reference.md documents the url source with key path (code requires url); README tool table omits plot_series and run_command; README config table omits SKTIME_MCP_MAX_DATA_HANDLES and SKTIME_MCP_MAX_RESPONSE_TOKENS; list_available_data(is_demo=true) still returns active_handles: []; load_data_source.config is an untyped object ([ENH] Add structured config schemas for load_data_source by source type #350); query_registry.task has no enum and evaluate.cv_folds no minimum (Add JSON Schema constraints for task and cv_folds in MCP server #407). (F-39)

P3 — nits (one roundup PR per area)

• Server/registry: serverInfo.version reports the mcp SDK version, app.py hardcodes 0.1.0; undeclared list_data_sources/auto_format_on_load still dispatchable (the latter mutates server state); unknown-tool response lacks success: false; limit=5.0 → "slice indices must be integers"; query_registry.tags JSON-string branch unreachable; sanitize_for_json turns set into the string "{1, 2}"; run_command with a backgrounded child holding stdout always reports a 120 s timeout, and reports success: true for chained commands where an earlier command failed; asyncio.create_task(_periodic_job_cleanup()) result not retained; cleanup_old_jobs evicts still-RUNNING jobs by created_at. (F-43–F-51, N-7)
• Executor/tools: split_data(fh=[12]) reserves 12 rows where sktime reserves 1, test_size uses floor vs sktime's ceil; transform_data(format) on a 2-row series → raw Need at least 3 dates to infer frequency; plot_series(figsize=[5]) → raw unpack error, no expanduser/mkdir, silent overwrite, figure leak on error, extra labels silently truncated; a failed re-fit leaves fitted=True from the previous fit; fit_async race when the handle is released mid-fit reports success for a non-existent handle (code-reading); split_data can evict its own fresh train_handle at tiny caps (code-reading); frequency displayed as "<MonthEnd>" repr; predict_var column named "0"; predict_quantiles without alpha echoes "alpha": null; classifier predict returns a bare list of stringified labels with no instance index. (F-52, F-55–F-60, F-65, N-3)
• Adapters/infra: file adapter ignores parse_dates: false, crashes on string csv_options, mutates the caller's csv_options; SQL component-built URL does not escape password (code-reading), list-valued filter → raw SQLAlchemy error echoing SQL and parameters; Dockerfile installs no [sql,files] extras although load_data_source advertises SQL/Excel/Parquet, .xls needs xlrd. (F-61, F-62, F-64)

Process

• MCP protocol-level tests are absent (MCP protocol layer has no test coverage — stdout corruption, silent tool loss, and serialization failures go undetected #310); PR Add MCP protocol test suite — stream cleanliness, tool registration, error contracts, JSON serializability #309 would provide them after a rebase and would have caught items 2 and 4.
• Keep the "one issue → one focused PR with a failing-then-passing test" convention; squash-merge in the order above (P0 first).

[Abhishek9639]

Hi @Shashankss1205,
Thanks for the detailed sweep the F-XX cross-references back to the 2026-08-20 audit make this very easy to pick up.

I'd like to pick this up starting with P0 item 1 (F-01, F-63) format_data_handle reindexing to a mismatched anchor and wiping data. Since the issue is explicitly intended to be split one-issue-per-line-item, could you assign me that item (or let me know if you'd prefer I open a dedicated issue for it and link it back here)?

Scope I'm planning for that item:

• In runtime/executor.py, only reindex when the existing index is a strict subset of the generated range; otherwise leave freq unset and return a frequency_warning instead of silently rewriting the index.
• Fix the reporting side so changes_made counts are ints and missing_values reflects the data actually stored, not the pre-reindex view.
• Cover the monthly-15th-with-a-gap case, weekly-Monday, and 30-minute data as regression tests, following the "one failing-then-passing test per PR" convention from the Process section.

On overlap: I've looked at #553 as noted, it only addresses the "D" catch-all for unknown frequencies, so the mismatched-anchor path is still open. I'll rebase on top of it rather than duplicate it, and I'll link #313 in the PR since this is the same root cause.

Once that's merged I'm happy to take P0 item 3 (F-03, F-41) the missing auth / DNS-rebinding / CORS gate on the HTTP app as a follow-up, since it's self-contained and doesn't touch the executor.

A couple of questions before I start:

1. For the mismatched-anchor case, do you want a hard failure (success: false) or success: true with frequency_warning and the original index preserved? The issue text reads as the latter, but confirming since it's a behaviour change for existing callers.
2. Should the changes_made / missing_values reporting fix ship in the same PR, or would you rather keep the data-loss fix minimal and split the reporting correctness into its own PR?

I'm working from current main (686b04b). Will hold off on opening a PR until scope is confirmed.

[samudraneel05]

hey @Shashankss1205 , this appears to be a duplicate of #556 produced by accident, best to close.