Subscription: support table-model column-filter for topics#17936
Open
VGalaxies wants to merge 3 commits into
Open
Subscription: support table-model column-filter for topics#17936VGalaxies wants to merge 3 commits into
VGalaxies wants to merge 3 commits into
Conversation
Independent ColumnFilter.g4 grammar; parser/validator/evaluator/binder/matcher/TabletColumnPruner; node-commons ColumnMetadata; column pruning in consensus WAL + pipe tablet + TsFile reseal paths; legacy 'column' regex removed; timeSelected flag wired end-to-end (per-table); CREATE/ALTER TOPIC validation + DataNode refreshColumnFilter cache hot-refresh; UT + IT coverage. Implements V2.0 subscription column-filter design. NOT YET VERIFIED: end-to-end datanode/confignode compile and full IT run.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #17936 +/- ##
============================================
+ Coverage 41.07% 41.19% +0.12%
Complexity 318 318
============================================
Files 5257 5265 +8
Lines 365010 365873 +863
Branches 47180 47337 +157
============================================
+ Hits 149918 150718 +800
- Misses 215092 215155 +63 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add a
column-filtertopic attribute for table-model subscriptions so a topic canrestrict which columns consumers receive, based on column metadata, applied uniformly
across historical, real-time pipe, and IoTConsensus WAL paths.
What
column-filter(table-model only); unset ==true.Restricted boolean predicate over column-metadata fields aligned with
information_schema.columns:database,table_name,column_name,datatype,category. Supports=/!=,IN,LIKE,REGEXP,IS [NOT] NULL,AND/OR/NOT.ColumnFilter.g4grammar (decoupled from the query grammar) producing thereused relational AST nodes; a whitelist validator rejects unsupported syntax/fields.
current schema, cached per topic on the DataNode and used for O(1) runtime pruning.
TAG columns are retained; ATTRIBUTE columns are not transmittable.
(prune at tablet granularity; raw TsFile passthrough kept only when the filter is trivial).
timeSelectedpropagated as an optional, backward-compatible per-table flag on the pollresponse (defaults to exposing time for old consumers).
columnregex attribute.Tests
realtime consistency, ALTER rebind, empty-result progress, snapshot semantics, custom time
column, altered data type, tree-view source-column binding, attribute drop, 1C3D cluster
ALTER-after-owner-transfer, DataNode restart re-snapshot, invalid-expression rejection, permission.
Notes
from the current schema on CREATE/ALTER/restart/owner-transfer; not frozen for the topic lifetime.