From 09771df7b9a2b6ec09b6f6e8a8fd5fd6f4abd477 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Wed, 24 Jun 2026 21:26:19 +0800 Subject: [PATCH] Add POST /inspect/values admin API to read foreign metric values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A new admin-only POST /inspect/values reads the VALUES of a metric this OAP does not define locally (persisted by another OAP), by running the real MQE engine over caller-supplied {valueColumn, valueType} — the value-reading companion to the merged GET /inspect/entities foreign path (#13926). A request-scoped InspectQueryContext ThreadLocal makes the foreign metric look registered to every read path (provide-if-absent — the local catalog always wins): ValueColumnMetadata resolves its value column / type / scope, and the storage location registries resolve where it lives — MetadataRegistry synthesizes a BanyanDB measure schema, IndexController resolves the ES metrics-all index, TableHelper probes the JDBC function tables. The storage read paths need no per-DAO special-casing; the response is the native MQE ExpressionResult. Verified end-to-end across banyandb/es/postgresql. --- docs/en/changes/changes.md | 1 + docs/en/setup/backend/admin-api/inspect.md | 85 ++++++++++++- docs/en/setup/backend/admin-api/readme.md | 3 + oap-server/server-admin/inspect/pom.xml | 6 + .../inspect/handler/InspectRestHandler.java | 114 ++++++++++++++++++ .../inspect/request/InspectValuesRequest.java | 69 +++++++++++ .../storage/annotation/ForeignMetricMeta.java | 49 ++++++++ .../annotation/InspectQueryContext.java | 64 ++++++++++ .../annotation/ValueColumnMetadata.java | 29 ++++- .../oap/query/graphql/mqe/rt/MQEExecutor.java | 111 +++++++++++++++++ .../plugin/banyandb/MetadataRegistry.java | 38 +++++- .../measure/BanyanDBMetricsQueryDAO.java | 3 + .../elasticsearch/base/IndexController.java | 15 ++- .../query/MetricsQueryEsDAO.java | 34 ++++-- .../plugin/jdbc/common/TableHelper.java | 13 ++ .../jdbc/common/dao/JDBCMetricsQueryDAO.java | 77 +++++++----- test/e2e-v2/cases/inspect/README.md | 14 ++- .../cases/inspect/inspect-foreign-flow.sh | 14 +++ test/e2e-v2/script/env | 2 +- 19 files changed, 684 insertions(+), 57 deletions(-) create mode 100644 oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/request/InspectValuesRequest.java create mode 100644 oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ForeignMetricMeta.java create mode 100644 oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/InspectQueryContext.java create mode 100644 oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/mqe/rt/MQEExecutor.java diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index fde18400e254..82debb79ee04 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -3,6 +3,7 @@ #### Project * Extend the `GET /inspect/entities` admin API to inspect a metric persisted by **any** OAP, even one this node does not define locally. When the metric is unknown to the local registry, the caller supplies `valueColumn` + `valueType` and the storage backend resolves the physical index/table/group from its own running config (no DB schema/table-metadata read): ES uses the merged `metrics-all` index + `metric_table` discriminator, JDBC probes the node's function tables by the `table_name` discriminator, and BanyanDB synthesizes a read-only measure schema. Scope is no longer required — the `entity_id` is decoded structurally (service / 2nd-level / relations) with a generic `name` leaf. Locally-defined metrics keep the exact field names, scope, and `mqeEntity` as before. +* Add the `POST /inspect/values` admin API — read the value series of a metric persisted by **another** OAP (one this node does not define locally) by supplying its `{valueColumn, valueType}`. The real MQE engine runs over a request-scoped `InspectQueryContext` overlay (provide-if-absent — the local catalog always wins) that makes the foreign metric look registered to every read path: `ValueColumnMetadata` resolves its value column / type / scope, and the storage location registries resolve where it lives (`MetadataRegistry` synthesizes a BanyanDB measure schema, `IndexController` resolves the ES `metrics-all` index, `TableHelper` probes the JDBC function tables), so the read returns the native MQE `ExpressionResult` with no per-DAO special-casing. Admin-only (a forced read this OAP cannot validate); not mirrored onto the public REST / GraphQL surface. See the [Inspect API](../setup/backend/admin-api/inspect.md). * Remove the always-on alarm-to-event conversion (`EventHookCallback`). A triggered alarm is no longer synthesized into the events pipeline as an `Alarm`/`AlarmRecovery` event; events now originate only from real event sources (agents, SkyWalking CLI, Kubernetes Event Exporter). Alarms remain available through the alarm store (`getAlarm`/`queryAlarms`) and the configured alarm hooks. This drops a documented "Known Event" and removes 1-2 synthetic event records per alarm fire. * **New `queryAlarms` GraphQL query — entity / layer / rule filters for alarms.** Adds a comprehensive alarm query API alongside the legacy `getAlarm`. The new diff --git a/docs/en/setup/backend/admin-api/inspect.md b/docs/en/setup/backend/admin-api/inspect.md index c43dea12c958..1fe958ee6680 100644 --- a/docs/en/setup/backend/admin-api/inspect.md +++ b/docs/en/setup/backend/admin-api/inspect.md @@ -1,17 +1,18 @@ # Inspect API -The Inspect API lives on `admin-server` and exposes two browse endpoints that -let operators answer two questions without writing exploratory MQE: +The Inspect API lives on `admin-server` and exposes three endpoints that +let operators answer three questions without writing exploratory MQE: -1. *Which metrics has OAP registered, and at what downsampling?* -2. *For metric `X` in time range `T`, which entities currently hold values?* +1. *Which metrics has OAP registered, and at what downsampling?* — `GET /inspect/metrics` +2. *For metric `X` in time range `T`, which entities currently hold values?* — `GET /inspect/entities` +3. *For metric `X` + entity `E`, what are the values?* — `POST /inspect/values` For a locally-defined metric, the output of (2) carries a ready-to-paste `mqeEntity` payload, so the follow-up MQE call against the public GraphQL `execExpression` mutation is copy-paste from the inspect response. A metric persisted by **another OAP** that this node does not define can also be -inspected with caller-supplied metadata (see -[Foreign metrics](#foreign-metrics-not-defined-on-this-oap)). +inspected with caller-supplied metadata — both its entities (2) and its values +(3) — see [Foreign metrics](#foreign-metrics-not-defined-on-this-oap). ## Enabling @@ -227,6 +228,73 @@ curl 'http://oap-admin:17128/inspect/entities?metric=meter_custom_x&valueColumn= } ``` +### `POST /inspect/values` + +Reads the **values** of a metric this OAP does not define locally, by running the +real MQE engine over caller-supplied metadata. Where `GET /inspect/entities` +answers *which entities hold values*, this answers *what those values are* — the +native MQE `ExpressionResult` (the same shape the UI renders for a catalog metric), +for a metric that is otherwise foreign to this node. + +Because it trusts caller-supplied metadata and forces a read of a metric this OAP +cannot validate, it is **admin-only** (it never mirrors onto the public REST / +GraphQL surface) and takes a request **body**: an MQE expression plus one metadata +entry per foreign metric the expression references. + +Request body (`application/json`): + +| Field | Required | Description | +|-------|----------|-------------| +| `expression` | yes | The MQE expression to evaluate — a single foreign metric name, or an expression combining foreign and/or catalog metrics. | +| `entity` | yes | The MQE query entity; its `scope` binds every foreign metric. e.g. `{ "scope": "Service", "serviceName": "X", "normal": true }` (use `serviceInstanceName` / `endpointName` for the deeper scopes). | +| `start` / `end` | yes | Time range, same format as [`/inspect/entities`](#get-inspectentities). | +| `step` | yes | One of `MINUTE` / `HOUR` / `DAY`. | +| `foreignMetrics` | yes | One entry per metric in `expression` that this OAP does not define: `{ "name": "...", "valueColumn": "value", "valueType": "LONG" }`. `valueColumn` is the post-override physical column; `valueType` is one of `LONG` / `INT` / `DOUBLE` / `LABELED`. A locally-defined metric must **not** be listed here (query it via the public GraphQL `execExpression`). | + +The metadata is overlaid **provide-if-absent** (the local catalog always wins) onto +the same registries the engine already consults, so a foreign metric looks registered +for the duration of the request: `ValueColumnMetadata` resolves its value column / type +/ scope, and the storage location registries resolve its index / table / measure exactly +as described in [Foreign metrics](#foreign-metrics-not-defined-on-this-oap). The overlay +is request-scoped to the calling thread and removed when the read completes; the public +query path never sets it. + +Only scalar (`LONG` / `INT` / `DOUBLE`) and labeled (best-effort) value series are +supported. An expression that resolves to `top_n` / records / heatmaps needs a local +model and surfaces as an error. Under ES `logicSharding=true` a foreign value read is +unsupported (the physical index derives from the metric's stream class), returning `500`. + +Example — read the value series of `meter_custom_x`, defined on another OAP: + +```bash +curl -X POST 'http://oap-admin:17128/inspect/values' \ + -H 'Content-Type: application/json' \ + -d '{ + "expression": "meter_custom_x", + "entity": { "scope": "Service", "serviceName": "payment", "normal": true }, + "start": "2026-05-10 1230", "end": "2026-05-10 1240", "step": "MINUTE", + "foreignMetrics": [ + { "name": "meter_custom_x", "valueColumn": "value", "valueType": "LONG" } + ] + }' +``` + +```json +{ + "type": "TIME_SERIES_VALUES", + "results": [ + { + "metric": { "labels": [] }, + "values": [ + { "id": "1778416200000", "value": "42" }, + { "id": "1778416260000", "value": "42" } + ] + } + ], + "error": null +} +``` + ## Discovering the OAP REST URL for the MQE follow-up To keep the surface minimal, the inspect API does not introduce a separate @@ -248,6 +316,11 @@ session start is enough. | 400 | `{"error":"metric type SAMPLED_RECORD is out of scope for /inspect/entities"}` | Metric is `SAMPLED_RECORD`. | | 400 | `{"error":"process scope is out of scope"}` | Scope is `Process` / `ProcessRelation`. | | 400 | `{"error":"limit must be between 1 and 300"}` | `limit` out of range. | +| 400 | `{"error":"foreignMetrics is required; a locally-defined metric should be queried via the public GraphQL execExpression"}` | `POST /inspect/values` body had no `foreignMetrics`. | +| 400 | `{"error":"metric foo is defined locally; query it via the GraphQL execExpression and drop it from foreignMetrics"}` | A `foreignMetrics` entry names a metric this OAP already defines. | +| 400 | `{"error":"valueColumn is invalid: …"}` | A `foreignMetrics` `valueColumn` is not a bare identifier. | +| 400 | `{"error":""}` | `POST /inspect/values` expression resolved to an unsupported shape (e.g. `top_n` / record / heatmap) for a foreign metric. | +| 500 | `{"error":""}` | A wrong `valueColumn` / `valueType`, or ES `logicSharding=true`, surfaced at the storage layer during a value read. | ## Limits diff --git a/docs/en/setup/backend/admin-api/readme.md b/docs/en/setup/backend/admin-api/readme.md index de50384bd5db..c75dad0d6dc3 100644 --- a/docs/en/setup/backend/admin-api/readme.md +++ b/docs/en/setup/backend/admin-api/readme.md @@ -86,6 +86,9 @@ Common operations: - `GET /inspect/metrics` — metric catalog with type / scope / supported downsamplings. - `GET /inspect/entities?metric=&start=&end=&step=` — capped (≤300) list of entities holding values, decoded into MQE-ready form. +- `POST /inspect/values` — read the value series of a metric this OAP does not + define locally (foreign metric), by supplying its `{valueColumn, valueType}`; + returns the native MQE result. Operator reference: [Inspect API](inspect.md). diff --git a/oap-server/server-admin/inspect/pom.xml b/oap-server/server-admin/inspect/pom.xml index 05da1ae65434..a51f99245d91 100644 --- a/oap-server/server-admin/inspect/pom.xml +++ b/oap-server/server-admin/inspect/pom.xml @@ -39,6 +39,12 @@ admin-server ${project.version} + + + org.apache.skywalking + query-graphql-plugin + ${project.version} + org.junit.jupiter junit-jupiter diff --git a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java index 1d358542ad31..7b8f5abab30f 100644 --- a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java +++ b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java @@ -18,11 +18,15 @@ package org.apache.skywalking.oap.server.admin.inspect.handler; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.linecorp.armeria.common.HttpData; import com.linecorp.armeria.common.HttpResponse; import com.linecorp.armeria.common.HttpStatus; import com.linecorp.armeria.common.MediaType; +import com.linecorp.armeria.server.annotation.Blocking; import com.linecorp.armeria.server.annotation.Get; import com.linecorp.armeria.server.annotation.Param; +import com.linecorp.armeria.server.annotation.Post; import java.io.IOException; import java.util.ArrayList; import java.util.Collections; @@ -36,7 +40,9 @@ import java.util.regex.PatternSyntaxException; import java.util.stream.Collectors; import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.query.graphql.mqe.rt.MQEExecutor; import org.apache.skywalking.oap.server.admin.inspect.decoder.EntityDecoder; +import org.apache.skywalking.oap.server.admin.inspect.request.InspectValuesRequest; import org.apache.skywalking.oap.server.admin.inspect.response.EntitiesResponse; import org.apache.skywalking.oap.server.admin.inspect.response.EntityRow; import org.apache.skywalking.oap.server.admin.inspect.response.ErrorResponse; @@ -51,10 +57,12 @@ import org.apache.skywalking.oap.server.core.query.enumeration.Scope; import org.apache.skywalking.oap.server.core.query.enumeration.Step; import org.apache.skywalking.oap.server.core.query.input.Duration; +import org.apache.skywalking.oap.server.core.query.mqe.ExpressionResult; import org.apache.skywalking.oap.server.core.query.type.Service; import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine; import org.apache.skywalking.oap.server.core.storage.StorageModule; import org.apache.skywalking.oap.server.core.storage.annotation.Column; +import org.apache.skywalking.oap.server.core.storage.annotation.ForeignMetricMeta; import org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnMetadata; import org.apache.skywalking.oap.server.core.storage.model.IModelManager; import org.apache.skywalking.oap.server.core.storage.model.Model; @@ -77,6 +85,9 @@ public class InspectRestHandler { /** Value types a caller may declare for a foreign (locally-undefined) metric. */ private static final Set ACCEPTED_FOREIGN_VALUE_TYPES = Set.of("LONG", "INT", "DOUBLE", "LABELED"); + /** A value column is interpolated into JDBC SQL on the read path; restrict it to a bare identifier. */ + private static final Pattern VALUE_COLUMN_PATTERN = Pattern.compile("^[A-Za-z_][A-Za-z0-9_]*$"); + private static final ObjectMapper VALUES_MAPPER = new ObjectMapper(); private final ModuleManager moduleManager; @@ -398,6 +409,109 @@ private HttpResponse listForeignEntities(final String metric, return HttpResponse.ofJson(MediaType.JSON_UTF_8, body); } + /** + * Read the VALUES of metric(s) persisted by another OAP that this node does not define. The body + * carries an MQE expression plus, in {@code foreignMetrics}, the metadata for each foreign metric + * it references (value column + type). The same MQE engine the public GraphQL surface uses is run + * synchronously with that metadata overlaid PROVIDE-IF-ABSENT (the catalog always wins), returning + * the native {@code ExpressionResult}. Marked {@code @Blocking}: the eval + storage read are + * synchronous and must not run on the event loop. Only scalar (LONG/INT/DOUBLE) and labeled + * (best-effort) value series are supported; {@code top_n} and record/heatmap shapes need a local + * model and surface as an error. + */ + @Blocking + @Post("/inspect/values") + public HttpResponse listValues(final HttpData requestBody) { + final InspectValuesRequest req; + try { + req = VALUES_MAPPER.readValue(requestBody.toStringUtf8(), InspectValuesRequest.class); + } catch (Exception e) { + return error(HttpStatus.BAD_REQUEST, "invalid request body: " + e.getMessage()); + } + if (req.getExpression() == null || req.getExpression().isBlank()) { + return error(HttpStatus.BAD_REQUEST, "expression is required"); + } + if (req.getEntity() == null || req.getEntity().getScope() == null) { + return error(HttpStatus.BAD_REQUEST, "entity (with a scope) is required"); + } + // The scope alone is not enough: the entity must carry the name fields its scope needs + // (e.g. serviceName + normal for Service), or buildId() yields a bogus id that the read + // silently misses — surface that as a 400 instead of an empty 200. + if (!req.getEntity().isValid()) { + return error(HttpStatus.BAD_REQUEST, + "entity is missing required fields for scope " + req.getEntity().getScope() + + " (Service needs serviceName + normal; ServiceInstance/Endpoint also need " + + "serviceInstanceName / endpointName)"); + } + if (req.getForeignMetrics() == null || req.getForeignMetrics().isEmpty()) { + return error(HttpStatus.BAD_REQUEST, + "foreignMetrics is required; a locally-defined metric should be queried via the public " + + "GraphQL execExpression"); + } + + final Step step; + try { + step = Step.valueOf(String.valueOf(req.getStep()).toUpperCase()); + } catch (Exception e) { + return error(HttpStatus.BAD_REQUEST, + "step must be one of MINUTE / HOUR / DAY (got " + req.getStep() + ")"); + } + if (step == Step.SECOND) { + return error(HttpStatus.BAD_REQUEST, "step must be one of MINUTE / HOUR / DAY (got SECOND)"); + } + + final int scopeId = req.getEntity().getScope().getScopeId(); + final List foreign = new ArrayList<>(); + for (final InspectValuesRequest.ForeignMetricInput fm : req.getForeignMetrics()) { + if (fm.getName() == null || fm.getName().isBlank()) { + return error(HttpStatus.BAD_REQUEST, "each foreignMetrics entry needs a name"); + } + if (fm.getValueColumn() == null || !VALUE_COLUMN_PATTERN.matcher(fm.getValueColumn()).matches()) { + return error(HttpStatus.BAD_REQUEST, "valueColumn is invalid: " + fm.getValueColumn()); + } + final String type = fm.getValueType() == null ? "" : fm.getValueType().toUpperCase(); + if (!ACCEPTED_FOREIGN_VALUE_TYPES.contains(type)) { + return error(HttpStatus.BAD_REQUEST, + "valueType must be one of LONG / INT / DOUBLE / LABELED (got " + fm.getValueType() + ")"); + } + if (ValueColumnMetadata.INSTANCE.readValueColumnDefinition(fm.getName()).isPresent()) { + return error(HttpStatus.BAD_REQUEST, + "metric " + fm.getName() + " is defined locally; query it via the GraphQL " + + "execExpression and drop it from foreignMetrics"); + } + foreign.add(new ForeignMetricMeta(fm.getName(), fm.getValueColumn(), type, scopeId, 0)); + } + + final Duration duration = new Duration(); + duration.setStart(req.getStart()); + duration.setEnd(req.getEnd()); + duration.setStep(step); + try { + duration.getStartTimeBucket(); + duration.getEndTimeBucket(); + } catch (IllegalArgumentException | UnexpectedException e) { + return error(HttpStatus.BAD_REQUEST, + "start / end must follow the step's date format (DAY: yyyy-MM-dd, HOUR: yyyy-MM-dd HH, " + + "MINUTE: yyyy-MM-dd HHmm): " + e.getMessage()); + } + + final ExpressionResult result; + try { + result = new MQEExecutor(moduleManager) + .execute(req.getExpression(), req.getEntity(), duration, foreign); + } catch (Exception e) { + // Optimistic read: a foreign top_n / record shape, or a wrong valueColumn/valueType, + // surfaces here rather than as garbage. + log.warn("inspect values execute failed for expression={}", req.getExpression(), e); + return error(HttpStatus.INTERNAL_SERVER_ERROR, e.getMessage()); + } + if (result.getError() != null) { + // e.g. an unsupported shape resolved to UNKNOWN — never put that on the wire as a 200. + return error(HttpStatus.BAD_REQUEST, result.getError()); + } + return HttpResponse.ofJson(MediaType.JSON_UTF_8, result); + } + /** * Mirror of the {@code /inspect/entities} type acceptance set. Kept in one place so * the {@code mqeQueryable=true} filter on {@code /inspect/metrics} and the actual diff --git a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/request/InspectValuesRequest.java b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/request/InspectValuesRequest.java new file mode 100644 index 000000000000..4d9e075c049e --- /dev/null +++ b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/request/InspectValuesRequest.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.admin.inspect.request; + +import java.util.List; +import lombok.Getter; +import lombok.Setter; +import org.apache.skywalking.oap.server.core.query.input.Entity; + +/** + * Body of {@code POST /inspect/values}: an MQE expression evaluated over one or more metrics this OAP + * does not define locally, whose metadata the caller supplies in {@link #foreignMetrics}. + */ +@Getter +@Setter +public class InspectValuesRequest { + /** + * The MQE expression to evaluate (e.g. a single foreign metric name, or an expression combining + * foreign and/or catalog metrics). + */ + private String expression; + /** + * The query entity; its {@code scope} is used to bind every foreign metric. + */ + private Entity entity; + private String start; + private String end; + /** + * One of {@code MINUTE} / {@code HOUR} / {@code DAY}. + */ + private String step; + /** + * Metadata for the foreign metrics referenced by {@link #expression}. + */ + private List foreignMetrics; + + /** + * Caller-supplied metadata for a single foreign metric. + */ + @Getter + @Setter + public static class ForeignMetricInput { + private String name; + /** + * Physical value column (post reserved-word override, e.g. {@code value_} on MySQL/PostgreSQL). + */ + private String valueColumn; + /** + * One of {@code LONG} / {@code INT} / {@code DOUBLE} (scalar) or {@code LABELED}. + */ + private String valueType; + } +} diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ForeignMetricMeta.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ForeignMetricMeta.java new file mode 100644 index 000000000000..914743767bfc --- /dev/null +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ForeignMetricMeta.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.core.storage.annotation; + +import lombok.Getter; +import lombok.RequiredArgsConstructor; + +/** + * Caller-supplied metadata for a metric this OAP does not define locally ("foreign"), used only by + * the admin inspect value path. It carries exactly what the read path cannot recover from the metric + * name: the value column, its data type, the scope, and the empty-bucket default. The scope is the + * metric's, not the queried entity's; it is derived from the query Entity by the caller side. + * + *

It is supplied per-request via {@link InspectQueryContext} and consulted PROVIDE-IF-ABSENT by + * {@link ValueColumnMetadata} — it can only fill gaps for unregistered metrics, never shadow a + * registered one. + */ +@Getter +@RequiredArgsConstructor +public class ForeignMetricMeta { + private final String metricName; + /** + * Physical value column (post reserved-word override, e.g. {@code value_} on MySQL/PostgreSQL). + */ + private final String valueColumn; + /** + * One of {@code LONG} / {@code INT} / {@code DOUBLE} (scalar) or {@code LABELED} (DataTable). + * Distinguishes both the MQE branch (COMMON vs LABELED) and the storage decode (long vs double). + */ + private final String valueType; + private final int scopeId; + private final int defaultValue; +} diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/InspectQueryContext.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/InspectQueryContext.java new file mode 100644 index 000000000000..746db4dfcdd3 --- /dev/null +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/InspectQueryContext.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.core.storage.annotation; + +import java.util.Map; + +/** + * Request-scoped overlay of {@link ForeignMetricMeta}, holding metadata for the foreign metrics + * referenced by ONE inspect value query, keyed by metric name. + * + *

This is NOT a behaviour side-channel: the metadata enters the system as an explicit parameter + * to the synchronous MQE execute entry, which republishes it here and removes it in a {@code finally} + * on the SAME thread that runs the (entirely synchronous) MQE eval + storage read. It is consulted by + * {@link ValueColumnMetadata} (so the MQE resolution sees the foreign metric) and by the storage + * value-read paths (which need {@link ForeignMetricMeta#getValueType()} to decode). It mirrors how + * {@code DebuggingTraceContext.TRACE_CONTEXT} already rides the same eval→DAO thread. The public MQE / + * GraphQL path never sets it, so it is {@code null} for every normal query. + */ +public final class InspectQueryContext { + private static final ThreadLocal> CONTEXT = new ThreadLocal<>(); + + private InspectQueryContext() { + } + + /** + * Open the overlay for the current thread. Must be paired with {@link #clear()} in a + * {@code finally} on the same thread. + * + * @param metaByMetric immutable map of metric name to its caller-supplied metadata + */ + public static void set(final Map metaByMetric) { + CONTEXT.set(Map.copyOf(metaByMetric)); + } + + public static void clear() { + CONTEXT.remove(); + } + + /** + * @param metricName the metric being resolved + * @return the caller-supplied metadata, or {@code null} when no overlay is active or the metric + * is not in it (i.e. it is a locally-registered metric) + */ + public static ForeignMetricMeta get(final String metricName) { + final Map map = CONTEXT.get(); + return map == null ? null : map.get(metricName); + } +} diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ValueColumnMetadata.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ValueColumnMetadata.java index de62c44738fe..69031feda7a7 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ValueColumnMetadata.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/ValueColumnMetadata.java @@ -85,7 +85,32 @@ public int getDefaultValue(String metricsName) { * @return metric metadata if found */ public Optional readValueColumnDefinition(String metricsName) { - return Optional.ofNullable(mapping.get(metricsName)); + return Optional.ofNullable(resolve(metricsName)); + } + + /** + * Resolve a metric's value column: the locally-registered catalog wins; only when a metric is + * absent locally does the request-scoped {@link InspectQueryContext} fill the gap (the admin + * inspect value path). The overlay is therefore provide-if-absent — it can never shadow or + * override a registered metric. Returns {@code null} when neither has it. + * + * @param metricsName the metric to resolve + * @return its value column, or {@code null} + */ + private ValueColumn resolve(String metricsName) { + final ValueColumn registered = mapping.get(metricsName); + if (registered != null) { + return registered; + } + final ForeignMetricMeta foreign = InspectQueryContext.get(metricsName); + return foreign == null ? null : toForeignValueColumn(foreign); + } + + private ValueColumn toForeignValueColumn(ForeignMetricMeta foreign) { + final Column.ValueDataType dataType = "LABELED".equals(foreign.getValueType()) + ? Column.ValueDataType.LABELED_VALUE : Column.ValueDataType.COMMON_VALUE; + return new ValueColumn( + foreign.getValueColumn(), dataType, foreign.getDefaultValue(), foreign.getScopeId(), false); } /** @@ -100,7 +125,7 @@ public Scope getScope(String metricsName) { } private ValueColumn findColumn(String metricsName) { - ValueColumn column = mapping.get(metricsName); + ValueColumn column = resolve(metricsName); if (column == null) { throw new RuntimeException("Metrics:" + metricsName + " doesn't have value column definition"); } diff --git a/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/mqe/rt/MQEExecutor.java b/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/mqe/rt/MQEExecutor.java new file mode 100644 index 000000000000..38277f08a74b --- /dev/null +++ b/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/mqe/rt/MQEExecutor.java @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.query.graphql.mqe.rt; + +import java.text.DecimalFormat; +import java.util.List; +import java.util.function.Function; +import java.util.stream.Collectors; +import org.antlr.v4.runtime.CharStreams; +import org.antlr.v4.runtime.CommonTokenStream; +import org.antlr.v4.runtime.misc.ParseCancellationException; +import org.antlr.v4.runtime.tree.ParseTree; +import org.apache.skywalking.mqe.rt.exception.ParseErrorListener; +import org.apache.skywalking.mqe.rt.grammar.MQELexer; +import org.apache.skywalking.mqe.rt.grammar.MQEParser; +import org.apache.skywalking.oap.server.core.query.input.Duration; +import org.apache.skywalking.oap.server.core.query.input.Entity; +import org.apache.skywalking.oap.server.core.query.mqe.ExpressionResult; +import org.apache.skywalking.oap.server.core.query.mqe.ExpressionResultType; +import org.apache.skywalking.oap.server.core.query.type.debugging.DebuggingTraceContext; +import org.apache.skywalking.oap.server.core.storage.annotation.InspectQueryContext; +import org.apache.skywalking.oap.server.core.storage.annotation.ForeignMetricMeta; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +import static org.apache.skywalking.oap.server.core.query.type.debugging.DebuggingTraceContext.TRACE_CONTEXT; + +/** + * Synchronous MQE evaluation entry, factored out of {@code MetricsExpressionQuery} so the admin + * inspect value path can run the SAME engine without the resolver's {@code queryAsync} ForkJoinPool + * hop. Optionally overlays caller-supplied metadata for foreign metrics (metrics this OAP does not + * define) via {@link InspectQueryContext} for the duration of the synchronous eval; the overlay is + * provide-if-absent and removed in a {@code finally} on the same thread that runs the eval and the + * storage read (the eval is fully synchronous, mirroring how {@code TRACE_CONTEXT} already rides that + * thread into the DAO). + */ +public class MQEExecutor { + private final ModuleManager moduleManager; + private final DecimalFormat valueFormat = new DecimalFormat(); + + public MQEExecutor(final ModuleManager moduleManager) { + this.moduleManager = moduleManager; + this.valueFormat.setGroupingUsed(false); + } + + /** + * Evaluate an MQE expression synchronously. + * + * @param expression the MQE expression + * @param entity the query entity; its scope is used to bind any foreign metric + * @param duration the query time range + * @param foreign metadata for foreign metrics referenced by the expression; {@code null} or + * empty for a purely catalog query (the public GraphQL path passes null) + * @return the native expression result + */ + public ExpressionResult execute(final String expression, final Entity entity, final Duration duration, + final List foreign) { + final boolean hasForeign = foreign != null && !foreign.isEmpty(); + if (hasForeign) { + InspectQueryContext.set(foreign.stream().collect( + Collectors.toMap(ForeignMetricMeta::getMetricName, Function.identity(), (a, b) -> a))); + } + final DebuggingTraceContext traceContext = new DebuggingTraceContext( + "Inspect MQE: " + expression + ", Entity: " + entity + ", Duration: " + duration, false, false); + TRACE_CONTEXT.set(traceContext); + try { + final MQEVisitor visitor = new MQEVisitor(moduleManager, entity, duration); + final MQELexer lexer = new MQELexer(CharStreams.fromString(expression)); + lexer.addErrorListener(new ParseErrorListener()); + final MQEParser parser = new MQEParser(new CommonTokenStream(lexer)); + parser.addErrorListener(new ParseErrorListener()); + final ParseTree tree; + try { + tree = parser.expression(); + } catch (ParseCancellationException e) { + final ExpressionResult errorResult = new ExpressionResult(); + errorResult.setType(ExpressionResultType.UNKNOWN); + errorResult.setError(e.getMessage()); + return errorResult; + } + final ExpressionResult result = visitor.visit(tree); + result.getResults().forEach(mqeValues -> mqeValues.getValues().forEach(mqeValue -> { + if (!mqeValue.isEmptyValue()) { + mqeValue.setValue(valueFormat.format(mqeValue.getDoubleValue())); + } + })); + return result; + } finally { + traceContext.stopTrace(); + TRACE_CONTEXT.remove(); + if (hasForeign) { + InspectQueryContext.clear(); + } + } + } +} diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java index 415e183a88ce..d14fb4055d27 100644 --- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java +++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java @@ -62,6 +62,8 @@ import org.apache.skywalking.oap.server.core.storage.StorageException; import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB; import org.apache.skywalking.oap.server.core.storage.annotation.Column; +import org.apache.skywalking.oap.server.core.storage.annotation.ForeignMetricMeta; +import org.apache.skywalking.oap.server.core.storage.annotation.InspectQueryContext; import org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnMetadata; import org.apache.skywalking.oap.server.core.storage.model.Model; import org.apache.skywalking.oap.server.core.storage.model.ModelColumn; @@ -422,7 +424,41 @@ public Schema findMetricMetadata(final String modelName, Step step) { * Find metadata with down-sampling */ public Schema findMetricMetadata(final String modelName, DownSampling downSampling) { - return this.registry.get(SchemaMetadata.formatName(modelName, downSampling)); + final Schema schema = this.registry.get(SchemaMetadata.formatName(modelName, downSampling)); + if (schema != null) { + return schema; + } + // Provide-if-absent: a locally-registered metric always wins (above). Only when this OAP has no + // schema for the metric AND the inspect overlay is active on THIS THREAD do we synthesize a + // read-only foreign schema. The overlay is a ThreadLocal set only on the admin inspect request + // thread (never a write thread), so writes — even those that reach here via findMetadata() — + // never observe it. + final ForeignMetricMeta foreign = InspectQueryContext.get(modelName); + if (foreign != null) { + return synthesizeForeignMetricSchema( + modelName, stepFromDownSampling(downSampling), foreign.getValueColumn(), foreign.getValueType()); + } + return null; + } + + /** + * Inverse of {@link #deriveFromStep(Step)}: map a {@link DownSampling} back to the {@link Step} + * that {@link #synthesizeForeignMetricSchema} expects. + * + * @param downSampling the down-sampling to map back + * @return the corresponding query step + */ + private Step stepFromDownSampling(DownSampling downSampling) { + switch (downSampling) { + case Day: + return Step.DAY; + case Hour: + return Step.HOUR; + case Second: + return Step.SECOND; + default: + return Step.MINUTE; + } } /** diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java index 413914177438..ac3d95c767f3 100644 --- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java +++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java @@ -59,6 +59,9 @@ public BanyanDBMetricsQueryDAO(BanyanDBStorageClient client) { @Override public MetricsValues readMetricsValues(MetricsCondition condition, String valueColumnName, Duration duration) throws IOException { String modelName = condition.getName(); + // findMetricMetadata is overlay-aware: for a foreign metric (the admin inspect value path) it + // synthesizes a read-only schema from InspectQueryContext; a local metric returns the registered + // one. So this read path no longer special-cases foreign metrics. MetadataRegistry.Schema schema = MetadataRegistry.INSTANCE.findMetricMetadata(modelName, duration.getStep()); if (schema == null) { throw new IOException("schema is not registered"); diff --git a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java index d41ebf755443..3dfd46a13058 100644 --- a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java +++ b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java @@ -24,6 +24,8 @@ import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.server.core.Const; import org.apache.skywalking.oap.server.core.analysis.FunctionCategory; +import org.apache.skywalking.oap.server.core.storage.annotation.ForeignMetricMeta; +import org.apache.skywalking.oap.server.core.storage.annotation.InspectQueryContext; import org.apache.skywalking.oap.server.core.storage.model.Model; import org.apache.skywalking.oap.server.core.storage.model.ModelColumn; import org.apache.skywalking.oap.server.library.util.CollectionUtils; @@ -33,7 +35,6 @@ import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.Optional; /** * The metrics data, that generated by OAL or MAL, would be partitioned to storage by the functions of the OAL or MAL. @@ -159,7 +160,17 @@ public static class LogicIndicesRegister { public static final String MANAGEMENT_TABLE_NAME = "management_table"; public static String getPhysicalTableName(String logicName) { - return Optional.ofNullable(LOGIC_INDICES_CATALOG.get(logicName)).orElse(logicName); + final String catalogEntry = LOGIC_INDICES_CATALOG.get(logicName); + if (catalogEntry != null) { + return catalogEntry; + } + // Provide-if-absent: the catalog wins (above). A foreign metric (inspect overlay active on + // this thread, no local registry entry) lives in the single merged metrics index instead of + // a per-metric index that does not exist; isMergedTable() derives true from this. The same + // overlay gates the foreign handling in MetricsQueryEsDAO. logicSharding is rejected upstream + // in the DAO, where the request context is understood. + final ForeignMetricMeta foreign = InspectQueryContext.get(logicName); + return foreign != null ? METRICS_LOGIC_TABLE_NAME : logicName; } public static void registerRelation(Model model, String physicalName) { diff --git a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java index b4c44e0be2e9..4d3cc2f68ac6 100644 --- a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java +++ b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java @@ -47,6 +47,7 @@ import org.apache.skywalking.oap.server.core.query.type.KVInt; import org.apache.skywalking.oap.server.core.query.type.KeyValue; import org.apache.skywalking.oap.server.core.query.type.MetricsValues; +import org.apache.skywalking.oap.server.core.storage.annotation.InspectQueryContext; import org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnMetadata; import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO; import org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient; @@ -64,16 +65,26 @@ public MetricsQueryEsDAO(ElasticSearchClient client) { @Override public MetricsValues readMetricsValues(final MetricsCondition condition, final String valueColumnName, - final Duration duration) { + final Duration duration) throws IOException { + // getPhysicalTableName / isMergedTable are overlay-aware: a foreign metric (admin inspect value + // path) resolves to the merged METRICS_LOGIC_TABLE_NAME index and reports mergedTable=true with + // no explicit branch here. The one case the overlay cannot cover is logicSharding=true, where the + // physical index derives from the metric's (absent) stream class — reject that up front. + if (InspectQueryContext.get(condition.getName()) != null && IndexController.INSTANCE.isLogicSharding()) { + throw new IOException( + "inspecting a foreign metric is unsupported under ES logicSharding=true: the physical " + + "index is derived from the metric's stream class, which this OAP does not have for " + + condition.getName()); + } final String realValueColumn = IndexController.LogicIndicesRegister.getPhysicalColumnName(condition.getName(), valueColumnName); - String tableName = - IndexController.LogicIndicesRegister.getPhysicalTableName(condition.getName()); + final boolean mergedTable = IndexController.LogicIndicesRegister.isMergedTable(condition.getName()); + final String tableName = IndexController.LogicIndicesRegister.getPhysicalTableName(condition.getName()); final List pointOfTimes = duration.assembleDurationPoints(); Map> indexIdsGroup = new HashMap<>(); final List ids = pointOfTimes.stream().map(pointOfTime -> { String id = pointOfTime.id(condition.getEntity().buildId()); - if (IndexController.LogicIndicesRegister.isMergedTable(condition.getName())) { + if (mergedTable) { id = IndexController.INSTANCE.generateDocId(condition.getName(), id); } String indexName = TimeSeriesUtils.queryIndexName( @@ -116,11 +127,20 @@ public MetricsValues readMetricsValues(final MetricsCondition condition, public List readLabeledMetricsValues(final MetricsCondition condition, final String valueColumnName, final List labels, - final Duration duration) { + final Duration duration) throws IOException { + // getPhysicalTableName is overlay-aware: a foreign metric (admin inspect value path) resolves + // to the merged METRICS_LOGIC_TABLE_NAME index, so aggregationMode derives true and the doc-id + // is metric-name-prefixed without an explicit foreign branch. logicSharding cannot be resolved + // for a foreign metric (index derives from the absent stream class) — reject it up front. + if (InspectQueryContext.get(condition.getName()) != null && IndexController.INSTANCE.isLogicSharding()) { + throw new IOException( + "inspecting a foreign metric is unsupported under ES logicSharding=true: the physical " + + "index is derived from the metric's stream class, which this OAP does not have for " + + condition.getName()); + } final String realValueColumn = IndexController.LogicIndicesRegister.getPhysicalColumnName(condition.getName(), valueColumnName); final List pointOfTimes = duration.assembleDurationPoints(); - String tableName = - IndexController.LogicIndicesRegister.getPhysicalTableName(condition.getName()); + final String tableName = IndexController.LogicIndicesRegister.getPhysicalTableName(condition.getName()); Map> indexIdsGroup = new HashMap<>(); boolean aggregationMode = !tableName.equals(condition.getName()); diff --git a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java index 7d02b4afceaa..e770ef6d340f 100644 --- a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java +++ b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java @@ -32,6 +32,7 @@ import org.apache.skywalking.oap.server.core.analysis.FunctionCategory; import org.apache.skywalking.oap.server.core.analysis.TimeBucket; import org.apache.skywalking.oap.server.core.config.ConfigService; +import org.apache.skywalking.oap.server.core.storage.annotation.InspectQueryContext; import org.apache.skywalking.oap.server.core.storage.model.Model; import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCClient; import org.apache.skywalking.oap.server.library.module.ModuleManager; @@ -39,6 +40,7 @@ import org.apache.skywalking.oap.server.storage.plugin.jdbc.TableMetaInfo; import java.time.Duration; +import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.concurrent.TimeUnit; @@ -107,6 +109,17 @@ public static String getTable(String rawTableName, long timeBucket) { public List getTablesForRead(String modelName, long timeBucketStart, long timeBucketEnd) { final var model = TableMetaInfo.get(modelName); + if (model == null && InspectQueryContext.get(modelName) != null) { + // A foreign metric (admin inspect value path: InspectQueryContext active on this thread) + // has no local model, so its physical function table is unknown. Probe every metric + // function table; the metric-prefixed row ids (generateId) keep only this metric's rows. + // A non-overlay miss is a genuinely unknown metric — fall through and let it surface. + final List tables = new ArrayList<>(); + for (final var rawTable : getMetricRawTables()) { + tables.addAll(getExistingDayTables(rawTable, timeBucketStart, timeBucketEnd)); + } + return tables; + } final var rawTableName = getTableName(model); if (!model.isTimeSeries()) { diff --git a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java index 5aa18fc9e2a9..9277422c7d81 100644 --- a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java +++ b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java @@ -30,6 +30,7 @@ import org.apache.skywalking.oap.server.core.query.type.KVInt; import org.apache.skywalking.oap.server.core.query.type.KeyValue; import org.apache.skywalking.oap.server.core.query.type.MetricsValues; +import org.apache.skywalking.oap.server.core.storage.annotation.InspectQueryContext; import org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnMetadata; import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO; import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCClient; @@ -59,11 +60,9 @@ public MetricsValues readMetricsValues(final MetricsCondition condition, // Label is null, because in readMetricsValues, no label parameter. final var intValues = metricsValues.getValues(); + final var foreign = InspectQueryContext.get(condition.getName()) != null; final var tables = tableHelper.getTablesForRead( - condition.getName(), - duration.getStartTimeBucket(), - duration.getEndTimeBucket() - ); + condition.getName(), duration.getStartTimeBucket(), duration.getEndTimeBucket()); final var pointOfTimes = duration.assembleDurationPoints(); final var entityId = condition.getEntity().buildId(); @@ -82,18 +81,26 @@ public MetricsValues readMetricsValues(final MetricsCondition condition, .collect(Collectors.joining(", ", "(", ")")) ); - jdbcClient.executeQuery( - sql.toString(), - resultSet -> { - while (resultSet.next()) { - final var kv = new KVInt(); - kv.setId(resultSet.getString("id")); - kv.setValue(resultSet.getLong(valueColumnName)); - intValues.addKVInt(kv); - } - return null; - }, - ids.toArray(new Object[0])); + try { + jdbcClient.executeQuery( + sql.toString(), + resultSet -> { + while (resultSet.next()) { + final var kv = new KVInt(); + kv.setId(resultSet.getString("id")); + kv.setValue(resultSet.getLong(valueColumnName)); + intValues.addKVInt(kv); + } + return null; + }, + ids.toArray(new Object[0])); + } catch (Exception e) { + if (!foreign) { + throw e; + } + // Foreign probe spans every metric function table; this one does not carry the + // caller's value column. The metric-prefixed ids match only its real table, so skip. + } } metricsValues.setValues( @@ -109,11 +116,9 @@ public List readLabeledMetricsValues(final MetricsCondition condi final List labels, final Duration duration) { final var idMap = new HashMap(); + final var foreign = InspectQueryContext.get(condition.getName()) != null; final var tables = tableHelper.getTablesForRead( - condition.getName(), - duration.getStartTimeBucket(), - duration.getEndTimeBucket() - ); + condition.getName(), duration.getStartTimeBucket(), duration.getEndTimeBucket()); final var pointOfTimes = duration.assembleDurationPoints(); final var entityId = condition.getEntity().buildId(); @@ -131,20 +136,28 @@ public List readLabeledMetricsValues(final MetricsCondition condi .collect(Collectors.joining(", ", "(", ")")) ); - jdbcClient.executeQuery( - sql.toString(), - resultSet -> { - while (resultSet.next()) { - String id = resultSet.getString("id"); + try { + jdbcClient.executeQuery( + sql.toString(), + resultSet -> { + while (resultSet.next()) { + String id = resultSet.getString("id"); - DataTable multipleValues = new DataTable(5); - multipleValues.toObject(resultSet.getString(valueColumnName)); + DataTable multipleValues = new DataTable(5); + multipleValues.toObject(resultSet.getString(valueColumnName)); - idMap.put(id, multipleValues); - } - return null; - }, - ids.toArray(new Object[0])); + idMap.put(id, multipleValues); + } + return null; + }, + ids.toArray(new Object[0])); + } catch (Exception e) { + if (!foreign) { + throw e; + } + // Foreign probe spans every metric function table; this one does not carry the + // caller's value column. The metric-prefixed ids match only its real table, so skip. + } } return Util.sortValues( Util.composeLabelValue(condition.getName(), labels, ids, idMap), diff --git a/test/e2e-v2/cases/inspect/README.md b/test/e2e-v2/cases/inspect/README.md index 32ff07a7404a..f498087a3421 100644 --- a/test/e2e-v2/cases/inspect/README.md +++ b/test/e2e-v2/cases/inspect/README.md @@ -14,17 +14,19 @@ Two independent OAPs share one storage backend (no cluster): | aware (existing) | oap-a | `/inspect/metrics` lists it; `/inspect/entities` returns `inspect-e2e-svc` with an `mqeEntity` | | — | oap-b | `/inspect/metrics` excludes it | | aware, no metadata | oap-b | `/inspect/entities` → `400 metric unknown locally …` | -| **foreign (new)** | oap-b | `/inspect/entities --value-column --value-type` returns the same entity, `scope:null`, no `mqeEntity` | +| **foreign entity (new)** | oap-b | `/inspect/entities --value-column --value-type` returns the same entity, `scope:null`, no `mqeEntity` | +| **foreign value (new)** | oap-b | `POST /inspect/values` with `foreignMetrics` returns the metric's value series (`42`) | Covered storages: `banyandb/`, `elasticsearch/`, `postgresql/`. ## CI wiring (gated on skywalking-cli) -The foreign assertion calls `swctl admin inspect entities --value-column / --value-type`, -flags added in [skywalking-cli #230](https://github.com/apache/skywalking-cli/pull/230). -The e2e builds swctl from `SW_CTL_COMMIT` (`test/e2e-v2/script/env`), which is pinned to a -cli commit that includes those flags, and the three storage variants are wired into the -`e2e` matrix in `.github/workflows/skywalking.yaml`. +The foreign assertions call `swctl admin inspect entities --value-column / --value-type` +(flags from [skywalking-cli #230](https://github.com/apache/skywalking-cli/pull/230)) and +`swctl admin inspect values --foreign-metric` (command from +[skywalking-cli #232](https://github.com/apache/skywalking-cli/pull/232)). The e2e builds swctl +from `SW_CTL_COMMIT` (`test/e2e-v2/script/env`), pinned to a cli commit that includes both, and the +three storage variants are wired into the `e2e` matrix in `.github/workflows/skywalking.yaml`. To validate locally, build swctl from that commit (or newer) and run any variant's `e2e.yaml` with `skywalking-infra-e2e`; all three storages pass. diff --git a/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh b/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh index f86a130f2093..67ee561b3c6d 100755 --- a/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh +++ b/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh @@ -29,6 +29,8 @@ # 3. OAP-B /inspect/entities WITHOUT valueColumn/valueType → "unknown locally". # 4. OAP-B /inspect/entities WITH valueColumn/valueType (FOREIGN/new path) returns # the SAME entity, scope=null, no mqeEntity. +# 5. OAP-B POST /inspect/values WITH foreignMetrics (FOREIGN VALUE/new path) returns +# the metric's value series (42). set -euo pipefail A_REST="${A_REST:-http://127.0.0.1:17128}" @@ -103,4 +105,16 @@ echo "${b_rows}" | jq -e '.rows[] | select(.decoded.serviceName=="'"${SVC}"'") | || fail "OAP-B foreign row should carry no mqeEntity: ${b_rows}" log " ✓ OAP-B FOREIGN /inspect/entities returns ${SVC}, scope=null, no mqeEntity (new path)" +# --- 5. OAP-B foreign VALUE read (admin inspect values) returns the metric value --- +# Uses the cli `admin inspect values` command (skywalking-cli #232, pinned via SW_CTL_COMMIT) → +# POST /inspect/values. MINUTE step: the per-minute value of meter_inspect_e2e_pool (= +# e2e_rr_pool_size summed over one service) is a constant 42; the DAY downsampling is not persisted +# this early in the run. Relative -30m/0m window so no host date-arithmetic is needed. +b_values="$(b_inspect values --expression "${METRIC}" --service-name "${SVC}" \ + --foreign-metric "${METRIC},${VC},LONG" --start "-30m" --end "0m" --step MINUTE)" \ + || fail "OAP-B admin inspect values errored" +echo "${b_values}" | jq -e '[.results[]?.values[]? | select(.value=="42")] | length > 0' >/dev/null \ + || fail "OAP-B admin inspect values returned no '42' value series: ${b_values}" +log " ✓ OAP-B FOREIGN admin inspect values returns the metric value 42 (new path)" + log "=== inspect-foreign-flow.sh PASSED ===" diff --git a/test/e2e-v2/script/env b/test/e2e-v2/script/env index 373df11394fc..80df25305174 100644 --- a/test/e2e-v2/script/env +++ b/test/e2e-v2/script/env @@ -27,7 +27,7 @@ SW_BANYANDB_COMMIT=c2d925e4eae4d77edda94e1fd438243483960150 SW_AGENT_PHP_COMMIT=d1114e7be5d89881eec76e5b56e69ff844691e35 SW_PREDICTOR_COMMIT=54a0197654a3781a6f73ce35146c712af297c994 -SW_CTL_COMMIT=90365c4bc59de3704ff81b4cefe55d09f706d00d +SW_CTL_COMMIT=85e5afdb3d55c6e5af66a472c3fe8ac024d11690 # Third-party image versions used by e2e infrastructure (not skywalking # components). Pinned here so the matrix is reproducible.