[core][state] preserve StateSchema column order in filter_fields

[KuongB]

Description

Item 2 of #30805 asks that task_id appear first in ray list tasks --detail, so that tasks are easier to track.

Checking against current master, that symptom no longer reproduces: task_id is already the first field of the TaskState dataclass, and all CLI output paths (table, YAML, --format json) follow the dataclass field order. The CLI is correct today.

The root cause behind it, however, is still live one layer down. filter_fields() in python/ray/util/state/common.py builds the payload that the State API server returns, and it iterated over columns() / base_columns() — both of which return a set. Iterating a set[str] yields an order that depends on the per-process string hash seed, so the key order of /api/v0/* responses is non-deterministic and changes on every restart.

This is invisible from the CLI because StateApiClient.list() rebuilds each dict into a TaskState dataclass, which restores the schema order. Anything consuming the HTTP API directly — the dashboard, external tooling — sees the raw, shuffled
order.

StateSchema.list_columns() already defines the canonical order, so this PR simply uses it in filter_fields(). It selects the same set of columns as before, only ordered.

Reproduction (before this change)

base_columns() returns a different order in every process:

for i in 1 2 3; do
  python -c "from ray.util.state.common import TaskState as T; print(list(T.base_columns())[:6])"
done

['attempt_number', 'actor_id', 'task_id', 'state', 'name', 'job_id']
['job_id', 'error_type', 'attempt_number', 'task_id', 'type', 'name']
['task_id', 'job_id', 'node_id', 'worker_id', 'parent_task_id', 'worker_pid']

task_id lands at position 3, 4, and 1 across three consecutive runs. That intermittency is likely why this went unnoticed for so long.

After

The same three commands, with this change applied.

for i in 1 2 3; do
  python -c "from ray.util.state.common import TaskState as T; print(T.list_columns(detail=False)[:6])"
done

['task_id', 'attempt_number', 'name', 'state', 'job_id', 'actor_id']
['task_id', 'attempt_number', 'name', 'state', 'job_id', 'actor_id']
['task_id', 'attempt_number', 'name', 'state', 'job_id', 'actor_id']

Identical every run, and matching the TaskState field declaration order.

Related issues

Addresses the second item in #30805 ("the ordering of the results ... Task Id should be the first"), by fixing the underlying cause rather than reordering columns at the presentation layer.

Supersedes #64283 — per @edoakes' review comment there, the column order is defined by StateSchema instead of being special-cased in state_cli.py.

[gemini-code-assist]

Code Review

This pull request updates the filter_fields function in python/ray/util/state/common.py to use state_dataclass.list_columns(detail=detail) and adds a unit test to verify that filter_fields preserves the schema's column order. The reviewer suggested a valuable improvement to the new test: reversing the order of the input dictionary's keys to ensure the test robustly verifies that the schema's column order is enforced regardless of the input dictionary's key order.

[Copilot]

Pull request overview

This PR makes State API response payloads deterministic by ensuring filter_fields() emits keys in the canonical StateSchema column order (dataclass field order) instead of iterating over unordered set-based column collections.

Changes:

• Update filter_fields() to iterate via StateSchema.list_columns(detail=...) to preserve schema-defined ordering.
• Add a unit test asserting filter_fields() preserves StateSchema column order for both detail=True and detail=False.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
python/ray/util/state/common.py	Switches filter_fields() to iterate ordered columns from StateSchema.list_columns() for deterministic API dict key order.
python/ray/tests/test_state_api.py	Adds a regression test verifying filter_fields() preserves schema column order.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[martinlhw]

Hey thanks for the contribution. If the CLI is correct today, could you elaborate on why this fix is needed?

[KuongB]

Hey thanks for the contribution. If the CLI is correct today, could you elaborate on why this fix is needed?

Hi @martinlhw, thanks for taking a look.

The CLI is correct today — this isn't a user-visible bug fix. The case for it is consistency: StateSchema.list_columns() is the declared canonical column order, and it's already what the table renderer and asdict() follow. filter_fields() was the single place that ignored it, iterating a set instead, so the HTTP API is the one surface where the schema's order doesn't hold.

That inconsistency is what makes #30805 confusing to reason about: the CLI happens to be correct not because the ordering is enforced, but because StateApiClient.list() rebuilds the dicts into dataclasses downstream and incidentally restores it. The invariant isn't actually held anywhere — it just looks like it is. Two curl /api/v0/tasks calls against restarted clusters return
the same fields in different key orders.

To be clear on scope: JSON objects are unordered by spec, and I'm not aware of anything in-tree that breaks today. The value is having one canonical order that every consumer honours, rather than one that happens to survive by accident.

[KuongB]

Hey thanks for the contribution. If the CLI is correct today, could you elaborate on why this fix is needed?

Hi @martinlhw, thanks for taking a look.

The CLI is correct today — this isn't a user-visible bug fix. The case for it is consistency: StateSchema.list_columns() is the declared canonical column order, and it's already what the table renderer and asdict() follow. filter_fields() was the single place that ignored it, iterating a set instead, so the HTTP API is the one surface where the schema's order doesn't hold.

That inconsistency is what makes #30805 confusing to reason about: the CLI happens to be correct not because the ordering is enforced, but because StateApiClient.list() rebuilds the dicts into dataclasses downstream and incidentally restores it. The invariant isn't actually held anywhere — it just looks like it is. Two curl /api/v0/tasks calls against restarted clusters return the same fields in different key orders.

To be clear on scope: JSON objects are unordered by spec, and I'm not aware of anything in-tree that breaks today. The value is having one canonical order that every consumer honours, rather than one that happens to survive by accident.

One correction to my own comment above, now that I've looked at the history: item 2 was deliberately fixed by #32353, which prepended task_id in state_cli.py. So the CLI ordering isn't accidental — and today it's enforced properly, since get_table_output() iterates list_columns() directly and asdict() follows the dataclass order. I overstated it by saying the invariant isn't held anywhere.

The narrower point stands: filter_fields() is the one place that doesn't follow list_columns(), so the HTTP API is the only surface where the schema's order doesn't hold.

Incidentally, #32353's review thread is a small argument for the consistency approach . This PR is the same idea: while tracing that issue I found the one place still inconsistent with it, and patched it.