Add a generic, extensible rest-endpoint provider SPI - #5656
Conversation
Add a leading `rest <endpoint>` command that exposes a curated, read-only, fixed-schema set of in-cluster management endpoints as a PPL table, modeled as a system row source bridged through visitRelation (the same seam as describe and the system-index family), so it runs on the Calcite engine without the unsupported table-function path. - Grammar/AST: REST/TIMEOUT tokens, restCommand rule, RestRelation, AstBuilder visit, query anonymizer. - Execution: RestSourceTable -> CalciteLogicalRestScan / CalciteEnumerableRestScan; RestEndpointRegistry (read-only allow-list + fixed schema + accepted args); RestEnumerator/RestRequest dispatch via OpenSearchClient (NodeClient in-cluster, RestClient standalone). - 9 endpoints: cluster health/state/settings, cat indices/nodes/cluster_manager/ plugins/shards, resolve/index. - Output shaping: numeric type normalization, id-to-name resolution, role-name expansion, structural flattening, graceful null degrade. - Args: count caps emitted rows; timeout reserved but rejected with 400; get-args applied server-side with per-arg value validation (local on health, health on cat/indices, expand_wildcards on resolve/index). Undeclared arg or out-of-domain value is rejected with a 400. level and include_defaults are deferred to a later release; flat_settings is dropped as redundant. - Error handling: blank endpoint, negative count, disallowed arg, and uncoercible value all surface clean 400s rather than 500s. Tests: CalcitePPLRestIT 25/25, RestEndpointRegistryTest 16, RestSourceTableTest 10. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
…xes; comment cleanup - /_cluster/settings: run the persistent and transient tiers through the node SettingsFilter (published via RestSettingsFilterHolder from SQLPlugin#getRestHandlers) so Property.Filtered and plugin-registered pattern settings are redacted exactly as the native GET /_cluster/settings endpoint. Remove the dead secretFields column-filter, which was the wrong shape for the (setting, value, tier) rows. - Parser: add TIMEOUT to searchableKeyWord so a bare 'timeout' term still matches searchLiteral. - coerce(): narrow the catch to IllegalArgumentException | ClassCastException; add empty-string guards in toNumber/toBoolean. - spotlessApply formatting; drop outdated and redundant comments. Tests: RestEndpointRegistryTest, RestSourceTableTest, OpenSearchNodeClientClusterSettingsFilterTest green. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
- clusterSettings: fail closed (throw IllegalStateException) when the node SettingsFilter is unavailable, instead of returning unredacted settings - collectSettings: handle list-type settings via getAsList fallback - decodeRestSpec: reject a blank/missing endpoint with a clear error - docs: correct rest.md allow-list table (9 endpoints + accepted args), quote endpoint literals, fix timeout/get-arg descriptions, add security note - register docs/user/ppl/cmd/rest.md in docs/category.json (deterministic single-node examples: number_of_nodes=1, cluster_manager count=1) Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
decodeRestSpec is only ever called behind an isRestSource gate today, but as a public decoder it must not assume its precondition. Without the guard a malformed token would surface an opaque StringIndexOutOfBoundsException from substring; now it throws a clear IllegalArgumentException instead. Addresses the PR opensearch-project#5599 Code Suggestions finding (importance 8). Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
…fetch - CalciteExplainIT.explainRestCommand: single-quote the endpoint literal; the explain harness inlines the query into a JSON body without escaping, so a double-quoted literal produced an invalid payload and a 400 (integration CI failure). - OpenSearchNodeClient.clusterSettings: resolve the SettingsFilter and fail closed BEFORE fetching cluster state, so settings are never read into memory when the redaction filter is unavailable. - OpenSearchRestClient.clusterState: narrow filter_path to nodes.*.name so node IPs/attributes are not over-fetched; manager-name resolution is preserved. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
… decode - AnalyticsEngineCompatIT: assert | rest '/_cluster/health' behaves identically with the analytics engine enabled (rest is never routed to DataFusion). - SystemIndexUtils.fromHex: reject an odd-length hex body so a crafted source name that passes the isRestSource suffix check fails clearly rather than silently dropping the trailing half-byte. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
On a cluster started with cluster.pluggable.dataformat=composite, RestUnifiedQueryAction.isAnalyticsIndex() routed every non-system-catalog PPL query to the analytics engine. The rest command's reserved in-cluster source (REST...__REST_SOURCE) has no backing index and only resolves on the Calcite path, so it was routed to DataFusion and failed with "Table 'REST...__REST_SOURCE' not found". Fix: exclude isRestSource(name) alongside isSystemCatalog(name) so a rest source falls back to the default (Calcite) pipeline and is never routed to the analytics engine. - RestUnifiedQueryActionTest: unit repro under cluster-composite. - integ-test analyticsEngineCompat testcluster set composite-default so the existing rest coexistence IT exercises this routing exclusion. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Collapse the two near-duplicate Calcite scan hierarchies -- SHOW/DESCRIBE system tables and the rest command -- into one generic OpenSearchCatalogTable whose per-endpoint behavior is supplied by a pluggable CatalogSource. - OpenSearchSystemIndex + RestSourceTable -> one OpenSearchCatalogTable backed by SystemIndexCatalogSource / RestCatalogSource. - Two Abstract/Logical/Enumerable scans + two enumerators + two converter rules -> AbstractCalciteCatalogScan / CalciteLogicalCatalogScan / CalciteEnumerableCatalogScan (+ rest-only CalciteScannableCatalogScan) / OpenSearchCatalogEnumerator / EnumerableCatalogScanRule. - Concerns stay per-source: system tables keep the real V2 implement() path; rest is Calcite-only (implement() throws) and opts into Scannable for the collect short-circuit. No behavior change: dispatch, schemas, V2 support, and the Scannable marker are preserved; this is pure de-duplication of the Calcite scan plumbing. Verified: opensearch compileJava/compileTestJava green; ppl and integ-test test-compile green; affected unit tests pass. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Two dynamic cluster settings gate the rest command per deployment: - plugins.ppl.rest.redaction.enabled (default false): mask network identifiers in _cat/* cells and availability-zone names in _cluster/settings values. - plugins.ppl.rest.allowed_endpoints (default all): restrict which endpoints are served; an empty list disables the rest command. Both default to open-source parity: all endpoints served, no masking. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
The shared language-grammar carried REST and TIMEOUT lexer tokens and the restCommand/restArgument parser rules with no consumer: async-query-core has no rest visitor, and the rest command grammar lives in the ppl module. Remove the dead rules. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Verify the rest command is subject to the security plugin fine grained access control: a caller without cluster:monitor privilege is denied the cat and cluster endpoints, a caller holding the privilege can run them, and the resolve index endpoint is filtered to the caller authorized indices. Test indices are created idempotently, and denials are asserted by the security denial reason in the response body because a denied action on the Calcite only rest path surfaces as a wrapped error carrying that reason. Calcite fallback is disabled so the denial reason is not replaced by an unsupported command error. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
plugins.ppl.rest.redaction.enabled and plugins.ppl.rest.allowed_endpoints were dynamic cluster settings, so they could be changed at runtime through _cluster/settings or the _plugins/_query/settings endpoint, which could override an operator-configured value. Drop Setting.Property.Dynamic so both are node-level settings set in the node config; the engine rejects runtime updates on both paths. Register them without an update consumer and read the node-configured value. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
Flip plugins.ppl.rest.allowed_endpoints default from ["*"] to empty so open source ships with the rest command closed. Deployments opt specific endpoints in via the setting (AOS enables the ones it supports; AOSS leaves it empty and stays disabled). Enforcement already treats an empty or missing list as disabled. Opt the integration-test clusters into all endpoints so the rest ITs still exercise the enabled path. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 84f5773.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
PR Reviewer Guide 🔍(Review updated until commit 7637f4d)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 7637f4d Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 84f5773
Suggestions up to commit 5ed738d
Suggestions up to commit 8bb45cd
|
…r/health sample) Rebuild of the reverted PPL rest command as a generic framework. A RestEndpointProvider ExtensiblePlugin SPI (in a thin ppl-rest-spi module) lets any plugin contribute read-only, fixed-schema endpoints resolved through one allow-list choke point and run as a lazy relational scan; the grammar stays a single generic rest <name> command. Ships one built-in endpoint /_cluster/health as the reference provider, allow-listed by default; any other endpoint stays disabled until explicitly added to plugins.ppl.rest.allowed_endpoints. Redaction is applied centrally at the choke point rather than through the endpoint SPI. Additional endpoints are left to follow-up changes. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
8bb45cd to
2c6901d
Compare
…ework Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com> # Conflicts: # ppl/src/main/antlr/OpenSearchPPLLexer.g4 # ppl/src/main/antlr/OpenSearchPPLParser.g4 # ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java
- ArgSpec: drop the unused csv-arg path (csvArg builder, csvArgs field, csv branch); no endpoint declares a csv arg (leftover from the deferred _cat endpoints). - RedactionRegistry: drop unused isEmpty() (consumers call mask()). - RestRequest: drop the unused 3-arg convenience ctor; fix a stale comment left by the ctx.client() refactor. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 5ed738d |
|
Persistent review updated to latest commit 84f5773 |
…e centralized Redactor The framework now exposes one Redactor seam ((endpoint, row) -> row, with a NONE no-op) published through RedactorHolder and applied once per raw row at the rest choke point before coercion; the default is NONE. Removes RedactionClass, RedactionRegistry, RedactionRegistryHolder and the per-column redaction tag on Column, so the framework stays field- and facet-agnostic, one coordinated masking pass avoids independent maskers interfering on the same value, and redaction has a single central owner. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
84f5773 to
7637f4d
Compare
|
Persistent review updated to latest commit 7637f4d |
Description
Rebuild of the reverted PPL
restcommand (#5599, reverted in #5635) as a separated, generic, extensible framework.rest <name>exposes read-only, fixed-schema management endpoints as a relational row source behind one operator allow-list, so results feed straight intowhere/stats/head.Key design points
ppl-rest-spimodule exposesRestEndpointProvider(anExtensiblePluginloadExtensionsSPI) plusRestEndpointDefinition/RestEndpointHandler/RestEndpointContextandColumn/ColumnType/ArgSpec. New endpoints are registrations, not new grammar keywords — the language stays a single genericrest <name>command./_cluster/healthships viaCoreEndpointsProvider, which fetches through the SPI context's transport node client (ctx.client()) exactly like an external plugin — there is no privileged in-repo path.ColumnTypekeeps the SPI module free of any dependency on the sql engine'sExprType.plugins.ppl.rest.allowed_endpointsis the singleresolve()gate; it defaults to["/_cluster/health"](health on, every other endpoint off until an operator opts in). Health stays gated by the cluster's owncluster:monitor/healthpermission.EXPLAINis side-effect free andwhere/head/statscompose and push down.RestRequestmirrors the system-indexOpenSearchSystemIndexScanbehind one shared catalog enumerator.Redactor((endpoint, row) -> row) at the choke point before coercion; the default isRedactor.NONE(no-op). Redaction lives at the platform choke point rather than in the endpoint SPI./_cluster/healthreference endpoint; further endpoints are left to follow-up PRs.Related Issues
Rebuild of #5599 (reverted in #5635) as a separated, extensible framework.
How it works:
handler+ctxflow (bootstrap to fetch)Each endpoint's
handleris captured into itsEndpointat bootstrap and travels bundled inside it; the per-queryctx(args + transport client) is assembled atsearch(); the two meet atEndpoint.toRows->handler.fetch(ctx). The registry and redactor cross the bootstrap-to-query Guice gap through static holders.flowchart TB subgraph BOOT["Phase 1 · Plugin bootstrap (once per node; loadExtensions runs before the Guice injector)"] direction TB prov["RestEndpointProvider SPI<br/>CoreEndpointsProvider.getEndpoints()<br/>→ RestEndpointDefinition(name, schema, argSpec, handler)"] load["SQLPlugin.loadExtensions<br/>loader.loadExtensions(RestEndpointProvider.class)"] build["publishRestCommandRegistries()<br/>new RestEndpointRegistry(providers)<br/>each Definition → new Endpoint(def)<br/>★ Endpoint.handler = def.handler()"] hold["RestEndpointRegistryHolder.set(registry)<br/>RedactorHolder.set(Redactor.NONE)"] prov -->|discovered| load load -->|providers| build build -->|publish| hold end subgraph QUERY["Phase 2 · Query execution (per query: rest '/_cluster/health' ...)"] direction TB gt["OpenSearchStorageEngine.restTable()<br/>registry = RestEndpointRegistryHolder.get()<br/>redactor = RedactorHolder.get()"] src["new RestCatalogSource(registry, spec, client, redactor)<br/>★ endpoint = registry.resolve(spec.endpoint) «allow-list gate»"] scan["CalciteEnumerableCatalogScan<br/>source.createRequest()"] req["RestCatalogSource.createRequest()<br/>→ new RestRequest(client, endpoint, spec, redactor)"] en["OpenSearchCatalogEnumerator<br/>request.search()"] se["RestRequest.search()<br/>★ ctx = RestEndpointContext.of(spec.args, nodeClient)<br/>endpoint.toRows(ctx, redactor)"] tr["Endpoint.toRows(ctx, redactor)<br/>★★ handler.fetch(ctx) «handler + ctx meet»<br/>→ redactor.redact(endpoint, raw) → coerce → rows"] gt --> src src --> scan scan --> req req --> en en --> se se --> tr end hold -.->|"registry / redactor (static bridge: bootstrap → query)"| gt build ==>|"handler travels bundled inside Endpoint"| trTesting
Redactorseam, storage engine, and settings.CalcitePPLRestITandRestCommandSecurityITcover/_cluster/healthplus the allow-list and per-arg gates.Check List
docs/user/ppl/cmd/rest.md).--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.