mcp-server: route agent discovery through Gravitino metadata

[markhoerth]

mcp-server: route agent discovery through Gravitino metadata

What changes were proposed in this pull request?

Agents given both the Gravitino MCP server and a SQL query engine were routing metadata questions to SQL, enumerating catalogs and schemas one level at a time, and reading data through whichever catalog answered first. Two changes keep discovery in the metadata layer.

Add a server-level FastMCP instructions string, defined once and applied to both server construction paths, stating that these tools are the first stop for what exists, where it lives, and how it is governed, that they return metadata and never rows, and that tag and policy tools should be consulted before reading potentially sensitive data.

Add a find_metadata tool that resolves a table name to its fully-qualified locations across every relational catalog and schema in a single call, so an agent no longer crawls to locate a table. The crawl is composed from the existing catalog, schema, and table listing operations in a testable helper. A catalog or schema that cannot be listed is recorded under skipped rather than aborting the search.

Why are the changes needed?

Without a routing steer, an agent holding both surfaces treats SQL as the general-purpose tool for questions the metadata layer answers directly. Listing tables became a fan-out of one query per catalog and per schema against live sources, and locating a table by name was the same crawl. Reads landed on whichever catalog resolved first, so a governed object could be read through an ungoverned path without the agent recognizing the difference. Steering discovery and governance questions to the metadata tools, and giving the agent a single-call name lookup, removes the reason it reaches for the query engine.

Does this PR introduce any user-facing change?

Yes. The MCP server now advertises a server-level instructions string to connected clients, and exposes one new tool, find_metadata, tagged under table. No existing tool, API, or property key is changed or removed.

How was this patch tested?

A new unit test, tests/unit/tools/test_metadata.py, covers the crawl helper with a fake client: case-insensitive and case-sensitive matching, substring matching, relational-only catalog filtering, skip-on-error for a schema that cannot be listed, the searched counts, and the empty-name guard. Seven cases, all passing:

cd mcp-server && uv run pytest tests/unit/tools/test_metadata.py -v

Also validated end to end on the Gravitino playground against a live agent: a "find the employees table" prompt resolved to a single find_metadata call returning the table across catalog_hive, catalog_hive_ranger, and catalog_postgres, surfaced the Ranger-governed twin, degraded gracefully on a schema reported as managed by multiple catalogs, and prompted a tags/policy check before reading rows.

Note on scope

The find_metadata crawl issues one listing call per catalog and per schema, which is fine at playground and mid scale but heavier on a large metalake. The durable successor is an indexed metadata search endpoint in Gravitino core that this tool would wrap, reducing the lookup to a single call. That is server-side work, out of scope here.

[yuqi1129]

The new instructions are useful, but I don't think the current find_metadata implementation is suitable for a global metadata lookup.

This moves the catalog/schema/table crawl from the agent into the MCP server without removing it. A lookup performs 1 + catalog_count + schema_count REST/catalog operations, sequentially, and fetches every table name before applying a substring match in Python. On a large metalake this can cause high latency and substantial load on Gravitino and the underlying live catalogs. It also has no pagination, result limit, timeout, or scan bound.

Calling this a “single call” is therefore misleading: it is one MCP call but potentially thousands of downstream calls.

I suggest keeping the routing instructions as a separate change and implementing metadata discovery through a Gravitino Server search API backed by an indexed metadata inventory. The API should define search scope, match mode, pagination, authorization filtering, and consistency semantics. If external, non-imported metadata must be searchable, that likely requires an asynchronous indexing/synchronization mechanism.

A Gravitino endpoint that performs the same nested crawl internally would not solve the scalability problem.

[yuqi1129]

If this capability is genuinely needed now, I suggest adding a dedicated search API to Gravitino Server first, and then having the MCP tool act as a thin wrapper around that API. This keeps metadata discovery semantics, authorization, pagination, and performance controls in Gravitino rather than implementing them independently in the MCP server. The server API should be backed by an appropriate metadata index or inventory; moving the same nested crawl behind a new endpoint would not address the scalability issue.