Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions docs-mintlify/api-reference/core-data.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ paths:
description: >-
Run a SQL query against the Cube [SQL API](/reference/core-data-apis/sql-api) and stream the
results. The response is newline-delimited JSON: the first line carries the `schema` (column
names and types) and optionally `lastRefreshTime`; each subsequent line carries a `data`
chunk with one or more result rows.
names and types), optionally `lastRefreshTime`, and optionally `usedPreAggregations` naming
the pre-aggregations the result was served from; each subsequent line carries a `data` chunk
with one or more result rows.
requestBody:
required: true
content:
Expand Down Expand Up @@ -127,6 +128,10 @@ paths:
lastRefreshTime:
type: string
format: date-time
usedPreAggregations:
type: object
additionalProperties:
type: object
data:
type: array
items:
Expand All @@ -140,6 +145,12 @@ paths:
- name: value
column_type: Int64
lastRefreshTime: '2025-01-13T12:00:00.000Z'
usedPreAggregations:
schema.orders_main20240101:
preAggregationId: Orders.main
targetTableName: schema.orders_main20240101_abc_def_1712
lastUpdatedAt: 1712000000000
type: rollup
dataLine:
summary: Data chunk
value:
Expand Down Expand Up @@ -672,6 +683,27 @@ components:
type: object
lastRefreshTime:
type: string
external:
type: boolean
description: '`true` when the result was served from an external (CubeStore) pre-aggregation.'
usedPreAggregations:
type: object
description: >-
Pre-aggregations this result was served from, keyed by pre-aggregation table name. Each
entry carries `preAggregationId`, `targetTableName`, `lastUpdatedAt` and `type`. Absent
when the query hit no pre-aggregation. In dev mode and for the Playground the entries
also carry `refreshKeyValues`.
additionalProperties:
type: object
properties:
preAggregationId:
type: string
targetTableName:
type: string
lastUpdatedAt:
type: integer
type:
type: string
V1LoadResultAnnotation:
type: object
required:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ object with query metadata:
`annotation` field of the [REST API][ref-rest-api] response.
- `lastRefreshTime` — an ISO 8601 timestamp indicating when the data was last
refreshed.
- `usedPreAggregations` — the pre-aggregations the result was served from, the
same as the `usedPreAggregations` field of the [REST API][ref-rest-api]
response. Absent when the query hit no pre-aggregation.

## `CubeQueryArgs`

Expand Down
11 changes: 9 additions & 2 deletions docs-mintlify/reference/core-data-apis/rest-api/reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ Response
- `lastRefreshTime` - An ISO 8601 timestamp indicating when the data was last refreshed.
- `external` - A boolean indicating whether the query was served from a
pre-aggregation in Cube Store. Present only when the query hit a pre-aggregation.
- `usedPreAggregations` - The pre-aggregations this result was served from, keyed
by pre-aggregation table name. Each entry carries `preAggregationId` (the name in

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two doc notes while this section is being touched:

  1. For a rollupJoin / rollupLambda pre-aggregation, preAggregationDescriptionsFor in the schema compiler expands to the referenced rollups, so the entries a client sees are the underlying rollups (their ids and type: "rollup"), never the lambda/join pre-aggregation itself. Worth one sentence — otherwise a user who defined Orders.lambda and looks for it in usedPreAggregations will conclude it's a bug.

  2. The external bullet two lines above claims "Present only when the query hit a pre-aggregation", but /load always emits it (the test in this PR asserts external === false for a non-pre-agg query). Only the SQL API JSONL header omits it when false. Pre-existing inaccuracy, but adjacent and cheap to fix in the same pass.

the data model, e.g. `Orders.main`, stable across rebuilds), `targetTableName`
(the table of one specific build), `lastUpdatedAt` and `type`. Absent when the
query hit no pre-aggregation.
- `total` - The total number of rows returned for the query. Useful for
paginating results.

Expand Down Expand Up @@ -420,8 +425,10 @@ This endpoint is part of the [SQL API][ref-sql-api].
| `x-request-id` | Custom request identifier. When provided, this ID is used to track the query through the system and can be used to [cancel the query](#base_path/v1/running-query/requestid). If not provided, a unique ID is generated automatically. | ❌ No |

Response: a stream of newline-delimited JSON objects. The first object contains
the `schema` property with column names and types, and optionally
`lastRefreshTime` indicating when the data was last refreshed.
the `schema` property with column names and types, optionally
`lastRefreshTime` indicating when the data was last refreshed, and optionally
`usedPreAggregations` naming the pre-aggregations the result was served from
(same shape as in the [JSON query](#base_path/v1/load) response).
The following objects contain chunks of the result set under the `data` property.
Each chunk includes one or more rows of the result set; the maximum number of rows
per chunk is set by
Expand Down
13 changes: 12 additions & 1 deletion docs-mintlify/scripts/extract-core-data.js
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ const EXTRA_PATHS = {
description:
'Run a SQL query against the Cube [SQL API](/reference/core-data-apis/sql-api) ' +
'and stream the results. The response is newline-delimited JSON: the first line ' +
'carries the `schema` (column names and types) and optionally `lastRefreshTime`; ' +
'carries the `schema` (column names and types), optionally `lastRefreshTime`, and ' +
'optionally `usedPreAggregations` naming the pre-aggregations the result was served ' +
'from; ' +
'each subsequent line carries a `data` chunk with one or more result rows.',
requestBody: {
required: true,
Expand Down Expand Up @@ -112,6 +114,7 @@ const EXTRA_PATHS = {
},
},
lastRefreshTime: { type: 'string', format: 'date-time' },
usedPreAggregations: { type: 'object', additionalProperties: { type: 'object' } },
data: { type: 'array', items: { type: 'array', items: {} } },
},
},
Expand All @@ -121,6 +124,14 @@ const EXTRA_PATHS = {
value: {
schema: [{ name: 'value', column_type: 'Int64' }],
lastRefreshTime: '2025-01-13T12:00:00.000Z',
usedPreAggregations: {
'schema.orders_main20240101': {
preAggregationId: 'Orders.main',
targetTableName: 'schema.orders_main20240101_abc_def_1712',
lastUpdatedAt: 1712000000000,
type: 'rollup',
},
},
},
},
dataLine: { summary: 'Data chunk', value: { data: [['123']] } },
Expand Down
17 changes: 17 additions & 0 deletions packages/cubejs-api-gateway/openspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -462,6 +462,23 @@ components:
type: "object"
lastRefreshTime:
type: "string"
external:
type: "boolean"
description: "`true` when the result was served from an external (CubeStore) pre-aggregation."
usedPreAggregations:
type: "object"
description: "Pre-aggregations this result was served from, keyed by pre-aggregation table name. Each entry carries `preAggregationId`, `targetTableName`, `lastUpdatedAt` and `type`. Absent when the query hit no pre-aggregation. In dev mode and for the Playground the entries also carry `refreshKeyValues`."
additionalProperties:
type: "object"
properties:
preAggregationId:
type: "string"
targetTableName:
type: "string"
lastUpdatedAt:
type: "integer"
type:
type: "string"
V1Error:
type: "object"
required:
Expand Down
47 changes: 47 additions & 0 deletions packages/cubejs-api-gateway/src/gateway.ts
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,44 @@ function systemAsyncHandler(handler: (req: Request & { context: ExtendedRequestC

const DEV_TOKEN_SCOPE = 'dev-token';

/**
* Fields of `usedPreAggregations` that are safe to report to any client: the
* identity of the pre-aggregation a result was served from, so the client can
* match the result to a build it is watching.
*
* `refreshKeyValues` is deliberately left out. Those are raw rows of the
* refresh key queries - typically aggregates such as `MAX(updated_at)` or
* `COUNT(*)` - and a `refreshKey.sql` is often written without the security
* context filtering that the cube itself applies, so the values can describe
* data the caller cannot otherwise reach. The full object, including them, is
* still returned in dev mode and to the Playground.
*/
function publicUsedPreAggregations(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design/security posture — worth an explicit decision from reviewers.

The docblock explains well why refreshKeyValues stays out, but the flip side isn't argued: targetTableName is now returned to every consumer, and it embeds the pre-aggregation schema name plus the content/structure version hashes of a specific build (e.g. stb_pre_aggs.foo_foo_main20240101_abc_def_1712). For a public embedded-analytics deployment that's internal storage layout leaking to untrusted browsers, and it's exactly what the test this PR rewrote was guarding.

Two things that would de-risk the reversal:

  1. preAggregationId + lastUpdatedAt alone already satisfy the stated use case ("match a result to the build behind it" / "join a chart to the build it is watching"). targetTableName is only needed if the client wants the physical table, which a data-API consumer can't use anyway. Consider dropping it from the public whitelist and keeping it dev/playground-only.
  2. If it must stay, consider gating the whole public projection behind an env flag (defaulting to on or off, your call) so operators who relied on the previous behavior can restore it without pinning a version.

Either way, the security-relevant part of this change (what ordinary consumers can now see) deserves a line in the docs, not just the usedPreAggregations field description.

usedPreAggregations: Record<string, any> | undefined
): Record<string, any> | undefined {
// A query that hit no pre-aggregation reports nothing rather than an empty
// object, so the key is simply absent from the response.
if (!usedPreAggregations || Object.keys(usedPreAggregations).length === 0) {
return undefined;
}

const publicFields = ['preAggregationId', 'targetTableName', 'lastUpdatedAt', 'type'];

return Object.fromEntries(
Object.entries(usedPreAggregations).map(([tableName, usage]) => [
tableName,
// Undefined fields are dropped rather than kept: the native result
// pipeline deserializes a JS `undefined` into a JSON `null`, so leaving
// them in would put `"preAggregationId": null` on the wire.
Object.fromEntries(
publicFields
.filter((field) => usage?.[field] !== undefined)
.map((field) => [field, usage[field]])
),
])
);
}

function hasDevTokenScope(securityContext: unknown): boolean {
if (typeof securityContext !== 'object' || securityContext === null) {
return false;
Expand Down Expand Up @@ -1945,6 +1983,10 @@ class ApiGateway {
const resObj = {
query: normalizedQuery,
lastRefreshTime: response.lastRefreshTime?.toISOString(),
// Identity of the pre-aggregations behind this result, so a client can
// join it to the build it is waiting on. The dev-mode block below
// replaces it with the unredacted object.
usedPreAggregations: publicUsedPreAggregations(response.usedPreAggregations),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor consistency gap: publicUsedPreAggregations deliberately collapses {} to undefined ("a query that hit no pre-aggregation reports nothing rather than an empty object"), but the dev/playground spread just below re-assigns response.usedPreAggregations verbatim. Since the orchestrator builds that object with R.fromPairs over the used-tables list, it's {} for every query that hit no pre-aggregation — so in dev mode the response carries usedPreAggregations: {} while in production the key is absent.

Not a regression (dev mode did this before too), but now that the key is part of the public contract, a client written against 'usedPreAggregations' in res behaves differently in dev vs prod. Cheap to align:

usedPreAggregations: Object.keys(response.usedPreAggregations || {}).length
  ? response.usedPreAggregations
  : undefined,

The Rust side already has the same normalization (is_empty_used_pre_aggregations), so this would make all three layers agree.

...(
getEnv('devMode') ||
context.signedWithPlaygroundAuthSecret
Expand Down Expand Up @@ -2242,6 +2284,11 @@ class ApiGateway {
// otherwise the SQL API reports "unknown" for every query cubesql
// hands over as pre-generated SQL.
lastRefreshTime: response.lastRefreshTime?.toISOString(),
// Same reason as `lastRefreshTime` above: the pre-aggregation
// identity has to travel with the pushed-down result too, or a
// cubesql query that goes through pre-generated SQL can never tell
// the client which pre-aggregation it read.
usedPreAggregations: publicUsedPreAggregations(response.usedPreAggregations),
// Always false: this branch builds its sqlQuery with
// `disableExternalPreAggregations` set (above), which makes
// `externalPreAggregationQuery()` return false, and the
Expand Down
1 change: 1 addition & 0 deletions packages/cubejs-api-gateway/src/graphql.ts
Original file line number Diff line number Diff line change
Expand Up @@ -675,6 +675,7 @@ export function makeSchema(metaConfig: any): GraphQLSchema {
res.extensions = {
annotation: results.annotation,
lastRefreshTime: results.lastRefreshTime,
usedPreAggregations: results.usedPreAggregations,
};

return results.data.map(entry => R.toPairs(entry)
Expand Down
29 changes: 22 additions & 7 deletions packages/cubejs-api-gateway/test/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1508,7 +1508,7 @@ describe('API Gateway', () => {
describe('external pre-aggregation indicator', () => {
// Helper mock that lets a test pretend the query orchestrator served
// the result from an external (CubeStore) pre-aggregation, optionally
// with the dev-only `usedPreAggregations` object as well.
// with the `usedPreAggregations` object as well.
class AdapterApiMockWithFlags extends AdapterApiMock {
public constructor(
private readonly external: boolean | undefined,
Expand Down Expand Up @@ -1536,16 +1536,23 @@ describe('API Gateway', () => {
.expect(200);

expect(res.body.external).toBe(false);
// Full pre-agg object stays dev/playground-only.
// No pre-aggregation was used, so there is nothing to name.
expect(res.body.usedPreAggregations).toBeUndefined();
});

test('external=true when query was served from an external pre-aggregation (no leak of names)', async () => {
// Pre-aggregation identity is reported to every API consumer so a client
// can match a result to the build it is waiting on. Refresh key values are
// not: they are rows of the refresh key queries, which are often written
// without the security context filtering the cube itself applies.
test('external=true exposes pre-aggregation identity without refresh key values', async () => {
const { app } = await createApiGateway(
new AdapterApiMockWithFlags(true, {
'Foo.fooMain': {
preAggregationId: 'Foo.fooMain',
targetTableName: 'stb_pre_aggs.foo_foo_main',
lastUpdatedAt: 1712000000000,
type: 'rollup',
refreshKeyValues: [[{ max_updated_at: '2024-01-01T00:00:00.000Z' }]],
},
}),
);
Expand All @@ -1556,16 +1563,24 @@ describe('API Gateway', () => {
.expect(200);

expect(res.body.external).toBe(true);
// Pre-aggregation names / table names must NOT be exposed to ordinary
// API consumers — only the boolean flag is safe.
expect(res.body.usedPreAggregations).toBeUndefined();
expect(res.body.usedPreAggregations).toEqual({
'Foo.fooMain': {
preAggregationId: 'Foo.fooMain',
targetTableName: 'stb_pre_aggs.foo_foo_main',
lastUpdatedAt: 1712000000000,
type: 'rollup',
},
});
});

test('usedPreAggregations is exposed under playground auth alongside external', async () => {
test('refresh key values are exposed under playground auth alongside external', async () => {
const usedPreAggregations = {
'Foo.fooMain': {
preAggregationId: 'Foo.fooMain',
targetTableName: 'stb_pre_aggs.foo_foo_main',
lastUpdatedAt: 1712000000000,
type: 'rollup',
refreshKeyValues: [[{ max_updated_at: '2024-01-01T00:00:00.000Z' }]],
},
};
const { app } = await createApiGateway(
Expand Down
75 changes: 74 additions & 1 deletion packages/cubejs-api-gateway/test/sql-api-load.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,27 @@ const logger = (type: any, message: any) => console.log({ type, ...message });

const LAST_REFRESH_TIME = new Date('2024-01-01T00:00:00.000Z');

// Shape the orchestrator reports: identity plus the refresh key values, which
// must not reach a regular client.
const USED_PRE_AGGREGATIONS = {
'schema.orders_main20240101': {
preAggregationId: 'Orders.main',
targetTableName: 'schema.orders_main20240101_abc_def_1712',
lastUpdatedAt: 1712000000000,
type: 'rollup',
refreshKeyValues: [[{ max_updated_at: '2024-01-01T00:00:00.000Z' }]],
},
};

class FreshnessAdapterApiMock extends AdapterApiMock {
public lastRefreshTime: Date | undefined;

public constructor(lastRefreshTime?: Date) {
public usedPreAggregations: Record<string, any> | undefined;

public constructor(lastRefreshTime?: Date, usedPreAggregations?: Record<string, any>) {
super();
this.lastRefreshTime = lastRefreshTime;
this.usedPreAggregations = usedPreAggregations;
}

public async executeQuery(_query: any) {
Expand All @@ -20,6 +35,7 @@ class FreshnessAdapterApiMock extends AdapterApiMock {
// Always falsy for a pushdown query — see the note in
// `sqlApiLoad`. Mirrors what the orchestrator actually echoes back.
external: false,
usedPreAggregations: this.usedPreAggregations,
};
}
}
Expand Down Expand Up @@ -88,4 +104,61 @@ describe('sqlApiLoad freshness metadata', () => {
expect(response.results).toHaveLength(1);
expect(response.results[0].lastRefreshTime).toBeUndefined();
});

// Pre-aggregation identity lets a client join a chart's result to the build
// it is waiting on. It has to travel with the pushed-down result too, since
// that is the branch every cubesql query takes.
test('pushed-down sqlQuery result carries usedPreAggregations', async () => {
const response = await sqlApiLoad(
new FreshnessAdapterApiMock(LAST_REFRESH_TIME, USED_PRE_AGGREGATIONS),
['SELECT * FROM test', []]
);

expect(response.results[0].usedPreAggregations).toEqual({
'schema.orders_main20240101': {
preAggregationId: 'Orders.main',
targetTableName: 'schema.orders_main20240101_abc_def_1712',
lastUpdatedAt: 1712000000000,
type: 'rollup',
},
});
});

// Refresh key values are rows of the refresh key queries, and a
// `refreshKey.sql` is often not filtered by the security context the cube
// itself applies. They stay dev-mode only.
test('usedPreAggregations omits refreshKeyValues', async () => {
const response = await sqlApiLoad(
new FreshnessAdapterApiMock(LAST_REFRESH_TIME, USED_PRE_AGGREGATIONS),
['SELECT * FROM test', []]
);

const usage = response.results[0].usedPreAggregations['schema.orders_main20240101'];
expect(usage.refreshKeyValues).toBeUndefined();
});

// A query that hit no pre-aggregation reports nothing rather than `{}`.
test('usedPreAggregations is absent when no pre-aggregation was used', async () => {
const response = await sqlApiLoad(
new FreshnessAdapterApiMock(LAST_REFRESH_TIME, {}),
['SELECT * FROM test', []]
);

expect(response.results[0].usedPreAggregations).toBeUndefined();
});

// The non-pushdown branch goes through `prepareResultTransformData`, which is
// also what the REST `/load` response is built from.
test('regular query result carries usedPreAggregations', async () => {
const response = await sqlApiLoad(
new FreshnessAdapterApiMock(LAST_REFRESH_TIME, USED_PRE_AGGREGATIONS)
);

const result = response.getResults()[0].getRootResultObject()[0];

expect(result.usedPreAggregations['schema.orders_main20240101'].preAggregationId)
.toBe('Orders.main');
expect(result.usedPreAggregations['schema.orders_main20240101'].refreshKeyValues)
.toBeUndefined();
});
});
Loading
Loading